@prefix vivo: . @prefix edm: . @prefix ns0: . @prefix dcterms: . @prefix skos: . vivo:departmentOrSchool "Graduate Studies, College of (Okanagan)"@en ; edm:dataProvider "DSpace"@en ; ns0:degreeCampus "UBCO"@en ; dcterms:creator "Cornelissen, Evelyn Constance"@en ; dcterms:issued "2012-10-15T16:56:02Z"@en, "2012"@en ; vivo:relatedDegree "Doctor of Philosophy - PhD"@en ; ns0:degreeGrantor "University of British Columbia"@en ; dcterms:description """Background: Priority setting approaches assist decision makers in choosing between various resource demands. One approach, Program Budgeting and Marginal Analysis (PBMA), supports decision makers to explicitly assess how resources can be used to maximize overall benefit from diverse service delivery options. Previous PBMA work establishes its efficacy (i.e., it can work) and also indicates that contextual factors complicate priority setting which can hamper PBMA effectiveness (i.e., whether it does actually work) in some settings. Methods: Using action research, researchers supported decision makers with implementing PBMA in a community care portfolio. Data were collected through semi-structured participant interviews (twenty pre-PBMA; twelve post year-1; nine post year-2), a pre-PBMA focus group (n=4), meeting attendance over three years, and document review. The interviews and focus group were transcribed. Data were analyzed using a constant comparison technique to explore PBMA effectiveness and implementation. Results: Fit emerged as a key theme in determining PBMA adoption and effectiveness. Here, fit refers to being of suitable quality and form to meet the end-users’ intended purposes and needs, and includes desirability, acceptability, and usability dimensions. Results confirm decision maker desire for rational approaches like PBMA. However, for several contextual reasons, most participants indicated that the timing and form in which PBMA was applied were not well-suited for this study. Their degree of acceptance of and buy-in to PBMA changed during the study: a leadership change, limited organizational commitment, and concerns with organizational capacity were key barriers to PBMA adoption. We found that adoption depended on contextual readiness and capacity, and that initial PBMA goals should include ensuring high-level commitment and moving toward more rational and evidence-informed decision making in general. Conclusions: These findings suggest that adding a contextual readiness/capacity assessment stage to PBMA, recognizing organizational complexity, and considering incremental PBMA adoption may help to improve PBMA’s effectiveness in some contexts. Based on these findings, tactics are suggested to more closely align PBMA with real-world priority setting practice. These suggestions may facilitate greater adoption, especially in contexts experiencing difficulty implementing PBMA. These insights may help in better understanding and working with priority setting conditions to advance evidence-informed decision making."""@en ; edm:aggregatedCHO "https://circle.library.ubc.ca/rest/handle/2429/43394?expand=metadata"@en ; skos:note "IMPLEMENTING A RATIONAL PRIORITY SETTING APPROACH IN COMMUNITY CARE IN THE INTERIOR HEALTH AUTHORITY: ASSESSING FIT, EVALUATING IMPLEMENTATION, AND DETERMINING IMPACT by Evelyn Cornelissen A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE COLLEGE OF GRADUATE STUDIES (Interdisciplinary Graduate Studies) THE UNIVERSITY OF BRITISH COLUMBIA (Okanagan) October 2012 © Evelyn Cornelissen, 2012 ii Abstract Background: Priority setting approaches assist decision makers in choosing between various resource demands. One approach, Program Budgeting and Marginal Analysis (PBMA), supports decision makers to explicitly assess how resources can be used to maximize overall benefit from diverse service delivery options. Previous PBMA work establishes its efficacy (i.e., it can work) and also indicates that contextual factors complicate priority setting which can hamper PBMA effectiveness (i.e., whether it does actually work) in some settings. Methods: Using action research, researchers supported decision makers with implementing PBMA in a community care portfolio. Data were collected through semi-structured participant interviews (twenty pre-PBMA; twelve post year-1; nine post year-2), a pre- PBMA focus group (n=4), meeting attendance over three years, and document review. The interviews and focus group were transcribed. Data were analyzed using a constant comparison technique to explore PBMA effectiveness and implementation. Results: Fit emerged as a key theme in determining PBMA adoption and effectiveness. Here, fit refers to being of suitable quality and form to meet the end-users’ intended purposes and needs, and includes desirability, acceptability, and usability dimensions. Results confirm decision maker desire for rational approaches like PBMA. However, for several contextual reasons, most participants indicated that the timing and form in which PBMA was applied were not well-suited for this study. Their degree of acceptance of and buy-in to PBMA changed during the study: a leadership change, limited organizational commitment, and concerns with organizational capacity were key barriers to PBMA adoption. We found that adoption depended on contextual readiness and capacity, and that initial PBMA goals should include ensuring high-level commitment and moving toward more rational and evidence- informed decision making in general. Conclusions: These findings suggest that adding a contextual readiness/capacity assessment stage to PBMA, recognizing organizational complexity, and considering incremental PBMA adoption may help to improve PBMA’s effectiveness in some contexts. Based on these iii findings, tactics are suggested to more closely align PBMA with real-world priority setting practice. These suggestions may facilitate greater adoption, especially in contexts experiencing difficulty implementing PBMA. These insights may help in better understanding and working with priority setting conditions to advance evidence-informed decision making. iv Preface This doctoral thesis summarizes research that I conducted alongside a CIHR-funded research project -- Priority setting, health care utilization and outcomes evaluation in community care in Interior Health – Partnership for Health System Improvement -- led by principal investigator Dr. Craig Mitton. Co-investigators included Dr. Stuart Peacock from the BC Cancer Research Centre, Drs. Alan Davidson and Colin Reid from UBC Okanagan, Dr. Cam Donaldson from Glasgow Caledonian University, Dr. Anne-Marie Broemeling from Alberta Health Services, Dr. Paul Hasselback from Vancouver Island Health Authority, and Mr. Tom Fulton from Interior Health Authority. The topics upon which my doctoral studies were based and that make up the three results chapters of this thesis evolved from the broader CIHR study. I was responsible for the data collection and analysis and for writing up these findings. My supervisor and research team members provided me with guidance, advice and feedback throughout the study; and my supervisor and committee provided me with the same throughout the development of this thesis. I received the following certificates of approval for conducting the research: 1. Priority setting, health care utilization and outcomes evaluation in community care in Interior Health – Partnership for Health System Improvement. Behavioural Research Ethics Board, the University of British Columbia (H06-03721). 2. Priority setting, health care utilization and outcomes evaluation in community care in Interior Health – Partnership for Health System Improvement. Interior Health Authority Research Ethics Board (IH Research File Identifier 2006-046). v Table of Contents Abstract .................................................................................................................................... ii Preface ..................................................................................................................................... iv Table of Contents .................................................................................................................... v List of Tables .......................................................................................................................... ix List of Figures .......................................................................................................................... x Glossary .................................................................................................................................. xi Acknowledgements ............................................................................................................... xii Dedication ............................................................................................................................. xiii 1 Chapter: Introduction ...................................................................................................... 1 1.1 Brief background................................................................................................................... 1 1.2 Thesis outline ........................................................................................................................ 2 1.3 Problem statement ................................................................................................................. 7 1.4 Research objectives ............................................................................................................... 8 1.5 Review of the literature ......................................................................................................... 9 1.5.1 Overview .......................................................................................................................... 9 1.5.2 Healthcare decision making ............................................................................................ 10 1.5.2.1 Concerns with decision making ............................................................................. 12 1.5.2.2 Various disciplines inform decision making ......................................................... 13 1.5.2.2.1 Evidence-based practice .................................................................................... 14 1.5.2.2.2 Economics ......................................................................................................... 14 1.5.2.2.3 Ethics ................................................................................................................. 15 1.5.3 Healthcare priority setting .............................................................................................. 16 1.5.3.1 Challenges with priority setting............................................................................. 18 1.5.3.2 Economics and priority setting .............................................................................. 19 1.5.3.3 Ethics and priority setting ...................................................................................... 23 1.5.3.3.1 Accountability for Reasonableness (A4R) ........................................................ 26 1.5.3.4 Successful priority setting ..................................................................................... 27 1.5.4 Program Budgeting and Marginal Analysis (PBMA) ..................................................... 27 vi 1.5.4.1 PBMA and economics, ethics and evidence-based practice .................................. 30 1.5.4.2 Successful and unsuccessful PBMA exercises ...................................................... 31 1.5.4.2.1 Lessons from unsuccessful exercises ................................................................ 31 1.5.4.3 Barriers and facilitators to using PBMA ............................................................... 32 1.5.4.4 Unresolved issues in the literature this research is filling ..................................... 34 1.5.4.5 Summary of PBMA relevant to the objectives of this thesis ................................. 36 1.5.5 Summary of literature review ......................................................................................... 37 2 Chapter: Methodology and Methods ............................................................................ 39 2.1 Overview of research chapters ............................................................................................ 39 2.2 Methodology ....................................................................................................................... 39 2.2.1 Social constructivist perspective ..................................................................................... 39 2.2.2 Positionality .................................................................................................................... 41 2.2.3 Action Research .............................................................................................................. 43 2.2.4 Validity ........................................................................................................................... 46 2.2.5 Context ............................................................................................................................ 48 2.3 Defining key constructs – fit, desirability, acceptability, usability ..................................... 50 2.4 Research questions .............................................................................................................. 50 2.5 Methods ............................................................................................................................... 52 2.5.1 Overview and study design ............................................................................................. 52 2.5.1.1 Participants and recruitment .................................................................................. 54 2.5.1.2 Unit of analysis ...................................................................................................... 54 2.5.2 PBMA implementation ................................................................................................... 54 2.5.3 Data collection ................................................................................................................ 56 2.5.3.1 Pre-PBMA ............................................................................................................. 57 2.5.3.2 PBMA implementations ........................................................................................ 57 2.5.3.3 Post PBMA implementations ................................................................................ 58 2.5.4 Data analysis ................................................................................................................... 58 2.5.4.1 Pre- and post-PBMA interviews ............................................................................ 58 2.5.4.2 Other qualitative data ............................................................................................ 59 2.5.4.3 PBMA implementation evaluation ........................................................................ 60 2.5.5 Research ethics ............................................................................................................... 61 2.6 Summary ............................................................................................................................. 61 vii 3 Chapter: Fit – Results and Discussion .......................................................................... 62 3.1 Synopsis of chapter ............................................................................................................. 62 3.2 Fit results ............................................................................................................................. 63 3.2.1 Pre-PBMA implementation – Desirability ..................................................................... 64 3.2.2 Post PBMA implementation – Acceptability .................................................................. 69 3.2.3 Post PBMA implementation – Usability ........................................................................ 71 3.3 Fit discussion ...................................................................................................................... 75 3.3.1 Pre-PBMA: Baseline versus desired drivers of priority setting ...................................... 75 3.3.2 Post PBMA implementation: Did PBMA deliver? ......................................................... 76 3.3.3 Restrictions to adopting a rational approach ................................................................... 77 3.4 Summary of fit .................................................................................................................... 80 4 Chapter: Implementation – Results and Discussion .................................................... 81 4.1 Synopsis of chapter ............................................................................................................. 81 4.2 Implementation results ........................................................................................................ 82 4.2.1 PBMA implementation evaluation findings, by PBMA stage ........................................ 82 4.2.2 Qualitative themes .......................................................................................................... 87 4.3 Implementation discussion .................................................................................................. 92 4.4 Summary of implementation ............................................................................................... 98 5 Chapter: Impact – Results and Discussion ................................................................... 99 5.1 Synopsis of chapter ............................................................................................................. 99 5.2 Impact results .................................................................................................................... 100 5.2.1 System-level impact...................................................................................................... 101 5.2.2 Individual and group level impact ................................................................................ 103 5.3 Impact discussion .............................................................................................................. 111 5.4 Summary of impact ........................................................................................................... 119 6 Chapter: Conclusion ..................................................................................................... 120 6.1 Overall analysis and integration of fit, implementation, and impact chapters with priority setting research ................................................................................................................. 120 6.2 Overall significance/contribution and potential applications of this research .................. 126 6.2.1 Contextual readiness and capacity assessment stage .................................................... 127 6.2.2 Incremental adoption of PBMA’s rational approach .................................................... 128 6.2.3 Develop each PBMA stage to be independently useful ................................................ 130 6.3 Conclusions based on the research objectives .................................................................. 131 viii 6.4 Limitations and generalizability (transferability) .............................................................. 134 6.4.1 Leadership change ........................................................................................................ 135 6.4.2 Education before pre-PBMA data collection ................................................................ 135 6.4.3 Transferability .............................................................................................................. 135 6.5 Potential future research directions emerging from the thesis research ............................ 136 6.5.1 Develop and test PBMA adaptations recommended in this thesis ............................... 139 6.6 Summary ........................................................................................................................... 141 References ............................................................................................................................ 142 Appendices ........................................................................................................................... 152 Appendix A Visual representation of problem statement .............................................................. 152 Appendix B Conceptual model for the diffusion of innovations in healthcare organizations ....... 153 Appendix C Study design .............................................................................................................. 155 Appendix D Weighted criteria and score-sheet.............................................................................. 156 Appendix E Investment and disinvestment proposals .................................................................... 158 Appendix F Pre-PBMA focus group and one-on-one qualitative interview question guide .......... 159 Appendix G Post year-one PBMA qualitative interview question guide ....................................... 160 Appendix H Post year-two PBMA qualitative interview question guide ...................................... 162 ix List of Tables Table 1.1 How economics and ethics can inform healthcare decision making and priority setting ....................................................................................................... 13 Table 1.2 Reasons for limited impact of health economics on priority setting ..................... 22 Table 1.3 Factors that impact distributive and procedural justice ......................................... 25 Table 1.4 Accountability for Reasonableness ........................................................................ 26 Table 1.5 Stages in a PBMA priority setting exercise ........................................................... 30 Table 1.6 Barriers and facilitators to explicit and rational priority setting ............................ 34 Table 2.1 Some principles of Action Research ...................................................................... 44 Table 2.2 Research questions, data collection methods, link to research objectives ............. 53 Table 2.3 PBMA implementation .......................................................................................... 55 Table 5.1 Primary priority setting drivers pre-PBMA and criteria chosen as drivers during PBMA process ......................................................................................... 108 Table 6.1 Summary of future research areas ....................................................................... 137 x List of Figures Figure 1.1 Thesis overview diagram ........................................................................................ 4 Figure 3.1 Spectrum between baseline (original) and desired drivers of priority setting ...... 64 Figure 5.1 Impact: continuum, how it relates to readiness & capacity assessment stage .... 114 Figure A.1 Visual representation of problem statement ...................................................... 152 Figure B.1 Conceptual model for the diffusion of innovations in healthcare organizations ..................................................................................................... 154 Figure C.1 Study design ....................................................................................................... 155 Figure D.1 Weighted criteria ............................................................................................... 156 Figure D.2 Score-sheet......................................................................................................... 157 Figure E.1 List of investment and disinvestment proposals ................................................ 158 Figure E.2 Ranked list of investment and disinvestment options ........................................ 158 xi Glossary Abbreviation Definition A4R Accountability for Reasonableness AR Action Research BC British Columbia EBP Evidence Based Practice FG Focus Group IH Interior Health Authority LHA Local Health Area PAR Participatory Action Research PBMA Program Budgeting and Marginal Analysis xii Acknowledgements As the saying goes, ‘it takes a village to raise a child’. Based on my experience, so too, can it be said, that it takes a village to develop a graduate student. Thank you to my supervisor and committee – Drs. Craig Mitton, Colin Reid, Alan Davidson, Rachelle Hole, and Anne-Marie Broemeling – for your collective wisdom, patience, and generosity. Key words of advice and well-timed questions and comments stimulated my thinking and maintained my motivation throughout this learning experience, and helped me to achieve more than I ever anticipated at the start of this journey. I am ever grateful to Dr. Craig Mitton for his excellent supervision, mentorship and guidance throughout my doctoral studies. He provided fantastic research and learning opportunities and an inclusive environment to learn and grow in my academic pursuits. His support far exceeded my expectations. I hope I have the opportunity to ‘pay it forward’. I also thank fellow researcher, Neale Smith, for thought-provoking conversations and for answering (and asking) so many questions! Thank you to Dr. Sam Sheps and the Western Regional Training Centre for Health Services Research (WRTC), for broadening and contextualizing my graduate studies. And thank you to the WRTC for providing financial support and a Tutorship opportunity. I offer my gratitude to the faculty, staff and my fellow students at UBC and in the WRTC, and my colleagues in the Knowledge Translation Trainee Collaborative (http://ktclearinghouse.ca/kttc/), who contributed to my experience in so many ways. A heartfelt thank you to my parents, Augusta and John, for ongoing love, support and genuine interest in my academic pursuits! Finally, I am so thankful to my husband, Glenn, and son, Coen, for more than I could ever write here. xiii Dedication To Glenn and Coen, for your inspiration and for sharing in this journey with me… And in memory of Hendrik (Henk) Jonker, who generously and openly shared his journey, and by doing so, continues to enrich mine. 1 1 Chapter: Introduction 1.1 Brief background Allocating scarce healthcare resources to meet growing population needs in an evolving healthcare context is a challenging task. Competing and increasing demands for service, shifting care models, and demographic change complicate efforts to decide how best to meet population needs with limited resources. In most Canadian provinces, responsibility for making healthcare resource allocation choices falls to regional health authorities (Lomas, Woods, & Veenstra, 1997) where decision makers are often constrained by institutional practices and legislated requirements. These constraints may interfere with setting priorities based on local population needs or maximizing benefit from services provided. Decision makers are further challenged by the inevitable role that values – those of the public, clients, clinicians, managers and their own – play in the process of setting priorities (Coulter & Ham, 2000). In some instances decision makers lack sufficient knowledge, skills and awareness of available tools that could assist with priority setting (Mitton & Donaldson, 2002). In such situations, services are often funded based on historical patterns (Mitton & Donaldson, 2004c), meaning that funding for a given year is determined largely by what was funded in previous years. Even if this historical allocation is adjusted to accommodate current organizational and population demands, it is not necessarily designed to maximize benefits from limited resources – something to which decision makers think a priority setting process should aspire (Teng, Mitton, & MacKenzie, 2007). A number of priority setting tools are available to help decision makers choose between options. One is an evidence-informed and systematic process known as Program Budgeting and Marginal Analysis (PBMA). Multiple surveys have shown that decision makers prefer PBMA over historical and political approaches to priority setting (Dionne, Mitton, Smith, & Donaldson, 2008; Mitton & Donaldson, 2002; Teng et al., 2007). However, research has also revealed that contextual factors complicate priority setting and thereby PBMA implementation in some settings (Mitton & Donaldson, 2003b). As a result, PBMA implementation has evolved with repeated use in different contexts. The seven-step implementation approach currently described in the literature (Mitton & Donaldson, 2004a; 2 Peacock et al., 2006) is designed to contextualize PBMA and the economic principles upon which it is based. As with any innovation, how effectively PBMA supports decision makers’ priority setting practice depends, in part, on the manner in which PBMA is implemented – implementation will affect PBMA acceptance and use by decision maker end-users. This thesis is about healthcare decision making, specifically how priorities are set to inform resource allocation decisions at the policy level. The focus is on the processes decision makers use to make resource allocation decisions and on how decision makers implement a rational priority setting approach (PBMA) to inform their decisions. To address these interests, in collaboration with a group of decision makers and experienced researchers, action research was used to explore PBMA effectiveness and impact, including examination of the role of PBMA implementation in changing decision maker priority setting practice. Although this research focused on PBMA as the priority setting approach under investigation, the lessons learned should be relevant to decision makers and others who are implementing other formal priority setting and evidence-informed practices. This generalizability or transferability is discussed in the concluding chapter. To set the stage for this research, the literature review below explores healthcare decision making in general, then narrows down to priority setting. It then reviews key ethical and economic considerations, given that these two fields are intrinsic to resource allocation. Economics is also reviewed, in part, because PBMA is founded on economic principles; and ethics is also reviewed because it has influenced the development of PBMA in healthcare. The literature review concludes with a discussion of PBMA, including its evolution in the healthcare field, important barriers and facilitators to its use, and unresolved issues in the literature that this thesis is addressing. 1.2 Thesis outline This thesis includes six chapters, which are outlined below. Figure 1.1 provides an overview of the thesis, including the objectives, methodology, main results and key findings. This figure demonstrates how the research project evolved and highlights the key themes that emerged from this work and that are subsequently explored in this thesis. At the center of the 3 figure is the PBMA implementation, as this research revolved around implementing PBMA over two sequential fiscal years in one community care portfolio in the Interior Health Authority of British Columbia: 2007-2008 and 2008-2009. 4 Figure 1.1 Thesis overview diagram Overarching research objectives: explore PBMA effectiveness and impact, and describe / evaluate the implementation process Action research: PBMA Implementation (7 stages) Outcomes / Key findings: Effectiveness depends on adoption which depends on contextual readiness and capacity Independently useful PBMA stages can facilitate adoption by increasing PBMA’s adaptability PBMA can be conceptualized as both a priority setting approach and as a tool to develop knowledge and practice Results Chapter 2: Role of implementation in priority setting practice change Adapt implementation: o Focus on participant clarity o Independently useful stages o Contextual readiness and capacity assessment Results Chapter 1: PBMA – Fit for purpose? Dimensions of fit: Desirability, Acceptability, Usability To improve adoption, effectiveness: o Add a contextual readiness and capacity assessment stage o Recognize organizational complexity o Consider incremental adoption Results Chapter 3: Determining and broadening the definition of impact from PBMA Positive impact found on a continuum: o Individual-level - knowledge, attitudes, practice o Group-level - knowledge, practice o System-level - priority setting, resource reallocation 5 Chapter 1, the current chapter, lays the foundation for the research project. It starts by setting up the research topic, which is about introducing a rational priority setting approach (i.e., PBMA) to healthcare decision makers to support their resource allocation decisions. The term ‘rational’ here refers to priority setting and decision making that formally incorporates a systematic and analytical approach (Klein, 1999). As discussed in more detail at the end of the literature review, PBMA supports decision makers in determining relevant criteria to inform healthcare priority setting and then in systematically analyzing various service delivery options based on these criteria. This chapter includes a description of the problem addressed in this thesis, which is two-fold: originally, the problem was about how to support decision makers in making decisions to allocate scarce healthcare resources; as the study progressed, the problem became more about how decision makers can implement a rational priority setting approach to effectively support their resource allocation decisions. It is not unexpected in action research for this type of evolution to occur. This chapter then introduces the objectives addressed in this thesis – which focus on exploring PBMA effectiveness, implementation and impact. It then reviews the relevant healthcare decision making, priority setting, economics, ethics, and PBMA literature. Chapter 2 describes the research methodology (i.e., action research) including the guiding philosophical perspective (i.e., social constructivism), and details the methods used and research questions addressed in this study. This chapter also includes definitions of key constructs that emerged in this work, including fit and its dimensions of desirability, acceptability and usability. The results are presented and discussed in three overarching and interrelated themes that emerged during concurrent data collection and analysis: PBMA fit, implementation, and impact. These are discussed in the next three chapters – chapters 3, 4 and 5. To address the research objectives, the discussion of each of these key themes draws on the broader literature on decision making, priority setting, implementation science, organizational behaviour and adult education. 6 Chapter 3 details the results of, and then discusses, the key theme ‘fit’. Early during data collection and analysis, which started during PBMA implementation, fit emerged as a key theme in exploring PBMA effectiveness. Fit refers here to being of suitable quality and form to meet the intended purposes and needs of the decision maker end-users. As the study and data analysis progressed, fit was further divided into the dimensions of desirability, acceptability and usability to probe more deeply into PBMA effectiveness and to highlight potential PBMA adaptations that may improve PBMA effectiveness in this context. Chapter 4 details the results of, and then discusses, the key theme ‘implementation’. Implementation became a focus in this study because how PBMA is implemented directly impacts how effective PBMA can be in a context. Also, given that previous research has found that change in healthcare is difficult to achieve (Graham & Tetroe, 2007; Grol, Bosch, Hulscher, Eccles, & Wensing, 2007), it could be realistically assumed that changing priority setting practice would also be difficult to achieve. And, indeed, this is what we found in this study. This chapter describes and evaluates the PBMA implementation process used in this study, and describes the experience and lessons learned implementing PBMA in this community care context. As such, this chapter also closely examines the role of PBMA implementation in changing decision maker priority setting practice. Chapter 5 details the results of, and then discusses, the key theme ‘impact’. Along with exploring PBMA effectiveness, determining PBMA impact was one of the a priori research objectives. Determining PBMA impact was important because the way in which impact is defined in the priority setting literature varies (Mitton & Donaldson, 2001; Tsourapas & Frew, 2011), with success often defined in terms of priorities set and/or resources reallocated (Peacock et al., 2010) although other potential impacts also exist. Chapter 6, the final chapter, includes an overall analysis and integration of the findings related to fit, implementation and impact (chapters 3, 4 and 5) and ties these findings into the key conclusions made as a result of this work. The key conclusions made in this thesis can be summarized as follows: 7 1. PBMA effectiveness depends on adoption of PBMA’s various components and rational approach, which depends on contextual readiness and capacity; 2. Although PBMA is currently described as a comprehensive seven-stage approach, developing these stages to be independently useful can facilitate adoption by increasing PBMA’s adaptability; and, 3. PBMA can be conceptualized as both a priority setting approach and a tool to develop knowledge and practice. This final chapter also includes conclusions based on the research objectives, a discussion of the overall contribution and significance of this work to the field of priority setting, the strengths and limitations of this research, and the transferability (generalizability) of these findings to other contexts. The thesis closes with a discussion of potential future directions that evolved from this research. 1.3 Problem statement Allocating scarce healthcare resources to meet growing population needs in an evolving healthcare context is challenging. PBMA is one priority setting approach available to assist decision makers in choosing between various resource demands. It supports decision makers in explicitly assessing how resources can be used to maximize overall benefit from a spectrum of service delivery options. And research has demonstrated that decision makers desire such a rational priority setting process to guide resource allocation decisions (Mitton & Patten, 2004). Previous PBMA work establishes its efficacy (i.e., it can work) and indicates that it is effective (i.e., it does actually work) in some settings (Mitton & Donaldson, 2001). However, prior to this study, there was limited focus on evaluation of PBMA in a community care context. Previous research also indicated that contextual factors complicate priority setting which can hamper PBMA effectiveness (Mitton & Donaldson, 2003b). These two points – limited evaluation of PBMA in a community care context and reference to contextual factors that complicate priority setting – gave rise to the two problems addressed in this study. See Appendix A for a visual representation of the problem statement. 8 At the start of this study, the problem to be addressed was about how to support decision makers in a community care context in making decisions to allocate scarce healthcare resources. As such, after determining what the participants desired in a priority setting practice and deeming PBMA to be a suitable approach, PBMA was introduced as a tool to support their priority setting practice. As the study progressed, however, it focused increasingly on the contextual factors that complicate priority setting, and thus a second related question arose, that of how decision makers can implement a rational priority setting approach like PBMA to effectively support their resource allocation decisions. This thesis addresses both of these problems. 1.4 Research objectives Following from the two problems outlined above, the objectives of this research were to implement PBMA in a Canadian regional health authority community care portfolio to: 1. Explore PBMA effectiveness (i.e., whether it does actually work) in a community care context, and then determine if and how PBMA can be adapted to make it more effective in this context. 2. Describe and evaluate the PBMA implementation process, and describe the experience and lessons learned, to gain insight into the role of implementation in changing priority setting practice. 3. Study the impact on the participants and the context of implementing a formal priority setting approach, using PBMA as an example. Impact is interpreted in light of the estimated implementation cost, i.e., what did these decision makers get for their time spent participating in PBMA? To address these objectives, using action research, a multi-disciplinary group of community care decision makers was supported in implementing PBMA to inform resource allocation decisions for two consecutive budget years. 9 1.5 Review of the literature 1.5.1 Overview Healthcare decision making encompasses a range of decisions, from patient-specific decisions made between a clinician and patient, to operational decisions made regarding service delivery, to those made in the boardroom regarding resource allocation. Many of the same concerns and complexities – competing issues, scarce resources, difficult decisions when dealing with people’s lives, values, needs and cultures – are prevalent in both clinical and policy decisions. This thesis focuses on the latter, primarily resource allocation decisions made at the policy level. Rationing and priority setting are terms used to describe the process of ranking various demands or needs for services to allocate limited resources (Ham & Coulter, 2000b). This is a difficult task. It has been described as “an unavoidably messy, conflict ridden, ultimately tragic social process” (Calabresi & Bobbitt, 1978, cited in Sabin, 1998, p. 1004)1. And once priorities are set, the work continues: allocation decisions must be made and operationalized; i.e., services distributed accordingly, whether that be as an individual clinician decides about patient care or a health authority decides about service provision at a community level. This literature review explores healthcare priority setting and decision making. It outlines how the disciplines of evidence-based practice, ethics and economics inform decision making, particularly resource allocation decisions. It also highlights the roles of rationality, intuition and politics in the process. The literature review then turns to priority setting, discussing what this entails, and what successful priority setting practice looks like. This segues into a review of PBMA: what it is, successful and unsuccessful exercises, and barriers and facilitators to its use with a special focus on how contextual factors complicate priority setting and therefore PBMA effectiveness in some settings. It closes with a summary of PBMA, in particular the gaps in the PBMA and priority setting literature that this research aims to fill and the importance of this work to society. 1 The original 1978 citation was maintained in this 1998 reference to demonstrate the enduring nature of the concerns identified with priority setting. 10 1.5.2 Healthcare decision making Healthcare decision making is important to focus on for two key reasons (Muir Gray, 2001): 1. To provide healthcare, a great number of decisions are made by clinicians about patients and by managers about groups of people/patients – approximately 40-50 million decisions/million population in the UK. 2. Decision making, whether by clinicians or executives, and whether multiple small decisions or fewer large ones, directly influences the cost of delivering care. Decisions are made at every level within the healthcare system, ranging from clinical to programmatic or operational, to financial and executive. Thus, decision makers are found at every hierarchical level also, with decisions ranging in magnitude and severity of outcome. Healthcare decision making – i.e., the purposeful selection from amongst a set of available options to meet a given objective – is complicated by several factors. These include competing goals, diverse needs, multiple interests and values, time constraints, external influences like political mandates or public input, and limited data and knowledge to inform decisions. The meaning of the terms ‘decision making’ and ‘decision maker’, although seemingly obvious, has been questioned. Some practices decision makers engage in are designed to postpone or bypass difficult decisions, making the term decision maker oxymoronic (Williams & Bryan, 2007a). This also complicates how decision making is defined as it makes it more difficult to pinpoint exactly when a decision is actually made; that is, ready to act on. Just how decisions are actually made is also a source of confusion. Decision making processes range from rational to judgmental 2 to political to various combinations of these, and incorporate a variety of approaches such as evidence-informed, comparative, intuitive or those based on negotiation or opinions. Part of the reason for the varied processes and approaches is that there are many drivers of healthcare decision making. The major drivers include evidence, values and resources – if only the latter two drive the decisions, then the 2 In judgment (or intuition) “formal analysis is replaced by tacit knowledge based on experience” (Baker, Ginsburg, & Langley, 2004, p. 88). This tacit knowledge, which comes from ‘learning by doing’, enables decision makers to judge situations based on experience; they may also tap into their creativity, which is an important but often neglected asset in decision making (Cooley, 2007). 11 process can be described as opinion-based decision making (Muir Gray, 2001). When value- driven, the process has been called ‘substantive rationality’ to differentiate it from rule-based or formal rationality (McDonald, 2002). Both opinion-based and substantive processes, to some, would be the contrast to evidence-based decision making where evidence is central to the decision making process. Also critical to evidence-based decision making is a structured, comprehensive approach (Muir Gray, 2001). However, a purely ‘rational comprehensive’ model of decision making – based on universally applied rules – assumes complete knowledge is available to inform the decision and that the goal is always to maximize value based on the information (McDonald, 2002). This does not allow for inevitable change, the reality of messy decision processes, the likelihood of value-laden information being used, and the limits of rationality; therefore, although technically perfect, this model may not prove of much practical use in real-world contexts. However, with increasing pressure on resources, decisions will have to be made more explicitly and linked to evidence when possible (Muir Gray, 2001). Thus, it appears that the role of rationality is increasingly important. Rational models are more conducive to comparative decision making. However, most decisions are not simply a choice between two or more options since options often have different consequences (Muir Gray, 2001). A further difficulty with comparative decision making approaches is that they inevitably result in winners and losers leading to tension with the process and outcomes (Martin, Pater, & Singer, 2001). This is in contrast to non- comparative approaches, where factors for each option are considered independently. These approaches may not result in such tension; however, may lead to a different kind of tension due to a lack of clarity regarding how the decision was actually made. These tensions – what role rationality should play and how explicit the process should be – result in different approaches being used to make decisions. Different disciplinary perspectives also produce different decision making approaches relevant to healthcare. These include evidence-based models (e.g., evidence-based medicine) that focus on using evidence to inform decisions; economic techniques (e.g., economic evaluation) that focus on applying analytical methods to define costs and consequences of options to facilitate explicit 12 decisions, especially regarding resource allocation (Jefferson, Demicheli, & Mugford, 2000); and ethical approaches (e.g., Accountability for Reasonableness) that focus on the fairness and legitimacy of the decision making process. 1.5.2.1 Concerns with decision making As helpful as these various decision making approaches are, their full benefit is only achievable for relatively straightforward decisions that align with their processes and tools. For complex, conflict-ridden decisions, they do not provide the full solution nor can they be relied on exclusively. Research into decision making in a variety of contexts has revealed several issues and questions that require further investigation. These include the following (Cooley, 2007; Klein, 1999; McDonald, 2002): How to balance the societal impression that a calculation is precise, analytical, scientific, apolitical and objective with the reality that the decisions that influence how the calculation is formulated are sometimes none of these. Determining how intuition based on ‘deep situational involvement and recognition of similarity’ becomes expertise, as these intuitive skills enable the expert to cope with uncertainties and unforeseen events and gives the expert the ability to override or disagree with calculated solutions as needed. How to manage ambiguity, which, unlike uncertainty, cannot be reduced through data collection. Ambiguous situations highlight that the world cannot be partitioned into mutually exclusive and exhaustive segments. What role power should play, e.g., the medical model of health results in medical dominance in healthcare delivery decisions. How to build in mental simulation, found in ‘expert performance’. How to balance rationality with individual perspectives, needs and values. 13 Lessons from the fields of economics, ethics and evidence-based medicine 3 are relevant to help address these issues and questions, and, therefore, are contributing to the evolution of healthcare decision making. See Table 1.1 for details. Table 1.1 How economics and ethics can inform healthcare decision making and priority setting 4 Problems that arise in decision making and priority setting How economics might help How ethics might help Conflicting opinions and demands, diverse needs, multiple interests and values – often dependant on individual perspectives, i.e., worldviews Complicating internal and external factors, e.g., budget restrictions introduced, political mandates, interference from outsiders, ‘specialist knowledge’ influence Competing goals; power relations Time constraints Lack of clarity regarding how decision rationales are assembled; complexity Lack of information Adds scientific characteristics, i.e., predictability, repeatability, quantifiability Provides methodology to highlight options, and to consider clusters of relevant factors, rather than simple trade-offs Clarifies and informs options to ease comparisons, adds explicitness Provides analytic tools to support the process, e.g., decision analysis Adds economic focus to contribute costing information and methods to measure and compare treatment and/or service effectiveness, e.g., QALYs 5 Contributes logic, explicitness, consistency Contributes to accountability and defensibility Builds fairness and transparency into the process Helps determine how opinions are formed and decisions made, rather than simply reporting what the opinions and decisions are Contributes to accountability and legitimacy Contributes comprehensive and consistent ethical principles to guide actions of people who give or shape healthcare 1.5.2.2 Various disciplines inform decision making Decision making approaches, particularly at the policy-level, continue to evolve in response to changing, growing and competing demands. Various disciplines inform this evolution, 3 In this thesis, the term ‘evidence-based medicine’ is often replaced with the more generic terms ‘evidence- based practice’ and ‘evidence-based decision making’. Also, in recognition of the inevitable role played by other factors in healthcare decision making, such as ‘expert’ opinion and politics, the term ‘evidence-based’ is increasingly being supplanted by ‘evidence-informed’ in the research literature. Both of these terms are also used in this thesis. 4 (Cooley, 2007; Jefferson et al., 2000; Klein, 1999; Martin et al., 2001; Smith, Hiatt, & Berwick, 1999; Tolley & Whynes, 1995; Williams & Bryan, 2007a) 5 QALY = Quality Adjusted Life Year; an outcome measure used in cost-utility analysis which incorporates both length of life and subjective levels of quality of life or wellbeing (Jefferson et al., 2000). 14 including economics, evidence-based practice, and ethics, among others. These different disciplinary perspectives have different aims. Generally, the aim of economics is efficiency (best use of available resources to optimize population-level health and non-health benefits); the aim of evidence-based practice is population-specific effective care (use clinical evidence to support the right care for the right person at the right time); and the aim of ethics is justice (fair population-level resource allocation to meet health needs by treating similar cases alike) (Gibson, Martin, & Singer, 2005). Each of these is briefly discussed below, along with some potential problems inherent in each approach. Due to their link to the evolution of the priority setting approach, PBMA, used in this study, ethics and economics are discussed in more depth in the priority setting section. 1.5.2.2.1 Evidence-based practice To address the ‘messy, conflict-ridden’ nature of decision making, evidence-based practice (EBP) is a good place to turn. Its focus on applying logic and using a rational 6 approach to analyze problems and find evidence and solutions brings an objective, empirical perspective to decision making and the tools it advocates. However, the implementation of evidence- based decision making in healthcare has not been easy. Its formal rationality requires clear objectives and clear means for producing results, both of which can be problematic in healthcare (Baker et al., 2004). Shifting goals – due to political influence – and difficulty determining what constitutes and then obtaining relevant evidence to inform decisions are both prominent issues. So, too, are issues of just how to address societal needs and values, and the fairness and transparency of the decision making processes used. Despite these challenges, many still desire increased rationality and evidence use. 1.5.2.2.2 Economics Economics may be an obvious discipline to turn to when seeking to enhance evidence-based decision making. Economics, the ‘science of choice’, is based on principles and techniques that facilitate logical and explicit decision making to inform choices regarding how best to use resources. The rational approach that underpins economic analyses lends itself well to 6 Rational decision making incorporates a systematic and analytical approach. It involves decomposing and decontextualizing information to enable logical calculations, deductive reasoning and explicit descriptions (Klein, 1999). 15 EBP. The economic principles of opportunity cost 7 and the margin aid rational thinking regarding resource allocation by highlighting relevant economic considerations. In this way, economics can be seen to contribute to the objective and empirical approach and desired outcomes advocated by supporters of evidence-informed decision making. Questions exist, however, regarding how to reconcile rational economic approaches with real-world decision making (McDonald, 2002). 1.5.2.2.3 Ethics When it comes to the fairness of the decision making processes, and how to address societal needs and values, the discipline of ethics provides guidance. Ethics does not eschew objectivity, but rather focuses more on the fairness and transparency of the process than on the specific desired outcomes. It contributes comprehensive and consistent principles to guide the actions of people who give or shape healthcare, which improves the accountability and legitimacy of the decisions made. It also helps in determining how opinions are formed rather than simply reporting what the opinions are. The answer to ‘how’ opinions are formed is valuable when exploring those decisions that are made in a deliberative fashion and that are based on values and beliefs. However, ensuring an ethical process at the expense of determining acceptable solutions does not solve the decision making concerns identified above. In summary, evidence-based practice, economics and ethics each illuminate different factors that should be considered when setting priorities and making decisions, and they contribute different mechanisms and tools for these practices. Some focus primarily on the outcomes, whereas others focus more on the process. Decision making mechanisms or approaches range in how rational and negotiable they are, ranging from political mandates that direct decisions, to comparative approaches that explicitly weigh options, to collaborative or negotiated decision making that enables multiple perspectives to be heard. Priority setting mechanisms and approaches also vary in how rational and negotiable they are. The principles that guide priority setting will depend on prevailing societal perspectives and norms. How the 7 Opportunity cost can be defined as the lost benefit from the next best use of resources. The margin refers to the benefit gained (or lost) from adding (or subtracting) the next unit of resources to a program (Mitton & Donaldson, 2004c). 16 disciplines of ethics and economics contribute to priority setting, and how priority setting approaches encompass ethical and economic principles, is discussed in more detail below. 1.5.3 Healthcare priority setting Healthcare priority setting is about deciding what services, programs, technologies, treatments, and care to cover and how to allocate resources accordingly. It entails setting priorities amongst all the competing claims for resources and then using these priorities to inform decisions regarding resource allocation. Historically, when priority setting was referred to more commonly as rationing or resource allocation (Singer, 2000), a more technical or rational approach was in favour; this was considered the first phase of priority setting (Holm, 1998). Holm (1998) contended that the second phase of priority setting in healthcare had to move away from devising a simple set of priority setting rules as this would not help produce legitimate decisions. He proposed focusing on the process and its transparency instead with the hope that priority setting would be legitimized by creating rules to govern the process rather than the decisions. Given that information is often incomplete and people involved have differing views, the process should be designed to help reconcile these different views and help redefine the issues so they are capable of being solved (Tolley & Whynes, 1995). This confirms Klein’s (1993) reflection that what really matters is how the debate is structured to facilitate reasoned, informed and transparent dialogue, and to include a variety of perspectives and interests. Research has identified that what is important in guiding priority setting practice is establishing a process that is fair, clear, consistent, explicit and evidence-informed (Dionne et al., 2008; Mitton & Prout, 2004; Teng et al., 2007). Regardless of the focus – the outcome (actual decisions) or process or both – healthcare priority setting is a challenge because demand exceeds available resources, resulting in inevitable winners and losers. This is further exacerbated by the roles that the legal and political systems, and media, play regarding priority setting decisions (Ham & Coulter, 2000a). When priorities are set that discriminate against certain people or populations, the potential for legal action increases. The media may report priority setting examples in sensational ways or present an unbalanced perspective of the situation. These factors reinforce the need to ensure that the priority setting process is both fair and transparent. 17 One of the underlying fundamental questions in healthcare priority setting is ‘what should be covered’ (Kitzhaber, 1993)? This is often defined with terms such as ‘comprehensive’, ‘basic’, ‘core services’, and ‘medically necessary’; however, these terms are subject to varying interpretations depending on the perspective taken. If societies are unable to resource everything, then limits must be set. This requires examining the relationship between healthcare provision and health, determining the relative effectiveness and appropriateness of various services, and exploring issues of costs, liabilities, societal and individual expectations, and responsibilities (Kitzhaber, 1993). Setting limits is a politically and socially challenging endeavor that those tasked with priority setting have been grappling with for years. Thus a process is required; preferably one that “involves the public, is linked to the reality of fiscal limits, and has clear lines of accountability” (Kitzhaber, 1993, p. 374). Priority setting practice has evolved along with healthcare reforms, such as regionalization in Canada and various funding schemes and private sector delivery innovations in the US. Economics, ethics, and evidence-based practice have all contributed to this evolution. Several examples of priority setting exercises and approaches are referred to in the literature. Approaches such as PBMA, categorized as an economic approach, and Accountability for Reasonableness (A4R), categorized as an ethical approach, are widely discussed in the healthcare priority setting literature. Both are described below. Several priority setting exercises or experiments described in the literature highlight the difficulties inherent in setting priorities to inform healthcare resource allocation decisions. For example, the educational program known as ‘The Priority Setting Exercise’ in the UK (Tolley & Whynes, 1995) highlights the issues involved in identifying priorities. It uncovered various priority setting criteria and processes used by different people and found that most people have a difficult time denying care. Another is the often-mentioned explicit priority setting exercise conducted in Oregon in the late 1980’s to 1990’s (Bodenheimer, 1997; Dixon & Welch, 1991; Drummond, Sculpher, Torrance, O'Brien, & Stodart, 2005; Kitzhaber, 1993). The Oregon experiment found that, due to the ongoing evolution of healthcare, the prioritization process had to be dynamic and ongoing. What ended up being largely a consensus building 18 exercise was supported by many citizens and groups 8 . Although applauded by some, the rational and explicit approach was deemed disconcerting by others, including some politicians, clinicians, ethicists, and some members of the public. Others, e.g., Senator Gore (1990), criticized Oregon for failing to examine healthcare waste, not obtaining more money through increased taxes or shifting from other programs. At the end of the first phase, the Oregon Health Plan included an increase in coverage to approximately 100,000 previously underinsured people. 1.5.3.1 Challenges with priority setting Although the Oregon experiment and The Priority Setting Exercise noted above were both rational and explicit exercises, these approaches do not supply the full solution or teach all the lessons for priority setting in healthcare. There are several issues that research into vertical 9 and horizontal priority setting activities has found that remain unresolved. These include the following (Dixon & Welch, 1991; Gibson, 2008; Kitzhaber, 1993; Tolley & Whynes, 1995): How to address the (perceived) absence of available and relevant information; and how to obtain suitable information to determine medical necessity, cost, clinical effectiveness, and to determine how to finance and deliver healthcare most effectively. A question of who, in reality, should determine priorities; noting that consensus becomes progressively more difficult with more people contributing opinions. How to manage specialist knowledge that some participants have which may enable them to play a greater role in priority setting than others. How to manage the politics. How to incorporate quality of life and public values into the process; concerns exist regarding how to obtain data on these inputs and how to weight them appropriately. 8 It was supported by the Oregon Medical Association, the Association of Oregon Hospitals, consumer groups, organized labour and the business community. It was also supported by both political parties and passed both houses of the Oregon legislature. 9 Horizontal priority setting is about “setting priorities between areas, e.g., care of the mentally ill vs treatments for cancer” (Tolley & Whynes, 1995, p. 391). This can be compared to vertical priority setting, which entails setting “priorities within a specific health care area, e.g., care of the mentally ill” (Tolley & Whynes, 1995, p. 391). 19 Lack of understanding regarding the critical success factors of health system readiness to participate in effective priority setting, including the change management strategies required to facilitate priority setting. Limited strategies and awareness of how to effectively engage clinicians and the public in the process. Several of these issues are similar to those encountered and described above for decision making; therefore, as outlined above, incorporating lessons and principles from economics, ethics and EBP is relevant to help address these challenges and questions 10 . Ongoing research into priority setting and decision making, including this thesis project, is shedding light on and contributing solutions to these issues. 1.5.3.2 Economics and priority setting Economics contributes both a way of thinking and a set of techniques to assist decision making. Regarding the former, economic principles – such as equitable distribution of resources, resource scarcity, and explicit attention to opportunity cost and the margin – contribute a way of thinking about the various topics that enter healthcare discourse. Regarding the latter, economic techniques – such as economic evaluation and decision analysis – contribute analytical methods to define costs and consequences of various interventions to aid explicit resource allocation decision making (Drummond et al., 2005; Greenhalgh, 2006; Jefferson et al., 2000). These economic techniques can capture multidisciplinary input which is often important in healthcare given the complex, messy nature of decision making. For example, economic evaluation in healthcare combines insights from health professionals (regarding clinical effectiveness), managers and accountants (regarding current and projected costs), and social scientists (regarding value and demand estimates, including from the patient perspective) (Jefferson et al., 2000). Decision analysis provides a method to clarify problems and to decrease the likelihood of making mistakes during data collection and interpretation to inform the decision (Jefferson et al., 10 See Table 1.1 (p. 13) for details and the sections below to delve more deeply into the link between priority setting and both economics and ethics. 20 2000). The consistent, explicit methods advocated by health economists enable outsiders to ask if the methods used to make decisions are up to the task, strengthening their validity (Freemantle & Mason, 1999). Early work in health economics focused on developing techniques to valuate life and limb, to determine the relationship between economic growth and workers’ health, and to valuate productivity loss by calculating a person’s monetary value (future earnings – maintenance) (Jefferson et al., 2000). Health economics evolved from this work when, in the 1950’s, US economists Arrow and Friedman started to apply classic economic theory to healthcare, specifically to aid resource allocation decisions and social reform (Jefferson et al., 2000). Over the years, it has grown to include increasingly more complex techniques with broader applicability in healthcare (Drummond et al., 2005). Health economics deals with resource allocation decisions, and how resources are used, at several levels of the healthcare system. Thus it has relevance to participants at each level, ranging from patients to providers to administrators to government authorities. Economics attempts to illuminate the ‘black box’ of resource allocation decision making through its principles and techniques, so that participants have a consistent language for discussion. There appears to be growing support for, and acceptance of, economic input into decision making. Surveys of decision makers indicate that they appreciate the potential value in using economic evaluation to inform decisions (Drummond, Cooke, & Walley, 1997; Hoffmann, 2000; Ross, 1995). Also, some governments require economic evaluation, along with clinically-related evidence, for public reimbursement (Drummond, 1992). The rational, evidence-based approach facilitated by economic evaluation is seen by both decision makers and health economists as an improvement over ad hoc and historical approaches to resource allocation decisions (Teng et al., 2007). Through its techniques and principles health economics assists decision makers in reaching conclusions to set priorities to allocate resources so that the most people will benefit (Jefferson et al., 2000), assuming this is at least one of the objectives shared by decision makers. Its reliance on logic and specific techniques also enables the priority setting process 21 and resulting decisions to be made explicit (see Table 1.1, p. 13 above). Consequently, health economics assists decision makers in meeting two commonly articulated goals of priority setting: maximizing benefit and making explicit the priority setting and decision making processes. This perhaps explains the increasing popularity of economic evaluations found in healthcare literature (Elixhauser, Halpern, Schmier, & Luce, 1998) and the generally positive attitude decision makers have towards economic evaluation (Hoffmann, 2000). Other work, however, has found limited impact of economic evaluation on priority setting (Drummond et al., 1997; Ham, 1993; Hoffmann, 2000; Ross, 1995; Walley, Barton, Cooke, & Drummond, 1997; Williams & Bryan, 2007b). Some reasons for this limited impact are outlined in Table 1.2 below, with the majority attributed to organizational factors rather than intrinsic properties of economic evaluation itself (Hoffmann, 2000). Hoffmann (2000) found that decision makers wanted a better explanation of the practical relevance of economic studies and indicated a need for more economic training. Ross (1995) concluded that those doing economic evaluations should be more responsive to the needs of decision makers, and that impact will increase when economic evaluation is linked directly to the decision making process; i.e., more focus on contextualizing priority setting is required. Only then will the objective of health economics – to enable decision makers, including clinicians and patients, to make informed decisions (Freemantle & Mason, 1999) – be realizable. 22 Table 1.2 Reasons for limited impact of health economics on priority setting 11 Reasons for limited impact attributable to the decision makers and/or the organization Reasons for limited impact attributable to health economics Lack of expertise and knowledge regarding economic evaluation Perceived lack of credibility of techniques or appropriate use; i.e., use of economic evaluation to delay a decision Perceived lack of validity of economic studies Communication issues; i.e., economist ‘jargon’ Idea that academic health economists are more concerned with methodological rigour than practical applicability, and that health economists require a better understanding of budgetary and contracting processes Time and timeline barriers Difficulty establishing health outcome measures Lack of acceptance by clinicians of non-clinicians evaluating clinical outcomes Perception that economic evaluation precludes consideration of other relevant factors; e.g., equity, political mandates Difficulty transferring budgets, freeing up resources Lack of data to inform the economic evaluation; e.g., costs, benefits Assumptions inherent in economic evaluation not made explicit; e.g., assumptions made regarding clinical effectiveness that are based on incomplete or lacking data Selective use of evidence; e.g., lack of use of unpublished data can be an issue for economic evaluation of new drugs; inappropriate transfer of results from one setting to another Lack of sufficient representation to inform analysis; e.g., from relevant health professionals Complexity Health economics contributes part of the solution to help address the challenges to priority setting outlined above. However, the logical and analytical tools of health economics should be combined with other inputs that enter into priority setting, such as politics and ethics. Reliance on logic and explicitness does not override the powerful influence of politics and ethics: …the preoccupation of health economists with methodological refinements, designed to increase the use of ‘rational’ methods is misguided. Where values conflict, such conflict cannot be resolved by an appeal to scientific enquiry or the formulation of calculable rules. In other words, the barriers to the use of health economics in practice are not amenable to speedy resolution by resort to ‘rational’ methods as they concern relationships of power, legitimacy and puzzlement at the heart of…decision making (McDonald, 2002, p. 169). 11 (Freemantle & Mason, 1999; Hoffmann, 2000; Ross, 1995; Williams & Bryan, 2007b) 23 Thus, perhaps, economics can best be seen as contributing perspectives and techniques that enable estimates of costs and benefits to support priority setting, and that other mechanisms must be built into priority setting to address the other relevant factors. 1.5.3.3 Ethics and priority setting In everyday usage, ethics is associated with morality and dealing with matters of right and wrong (Babbie & Benaquisto, 2002). The dictionary definition of ethics refines this to a system of moral principles or rules of conduct and embeds it in the branch of philosophy that deals with values regarding right and wrong actions, and/or good or bad motives (Webster's, 1989). This definition has broad applicability to healthcare, including the domains of research, care provision and administration of services. It is also applicable at the individual, population and system levels; perhaps most significantly when shedding light on the conflicts inherent in making decisions at these different levels. When considering the definition of ethics as it relates to healthcare decision making in general, a search for moral absolutes may be difficult, if not impossible, as what is considered moral or ethical depends on individual perspectives (Babbie & Benaquisto, 2002). Different individuals and cultures bring different perspectives to the table regarding needs, values, and what constitutes fair concerning outcomes and the process by which the outcomes are achieved. Thus, decision making requires negotiation and discussion about both desired outcomes and the process(es) used to achieve them. The discipline of ethics is useful for this because it sheds light on relevant issues to consider when developing principles and strategies to guide decisions and actions. Two goals of ethics pertinent to decision making and priority setting include the following: Legitimacy; i.e., addressing the conditions under which morally controversial decisions are viewed as valid and reasonable by those who disagree (Martin & Singer, 2000). Fairness of both process and outcomes; e.g., access to and use of resources (Gibson et al., 2005). Several problems that contribute to ethical dilemmas are inherent in healthcare systems today, especially in developed countries. These include the following (Smith et al., 1999): 24 Financial pressures due to illness coverage (especially catastrophic illness) exceeding what most individuals can afford, and resulting in private/public insurance financing. Limited resources, requiring choices be made about who receives care and to what extent. Complex and costly delivery systems which can result in tension between what is good for society versus what is good for an individual patient. Flaws in the system can result in bad outcomes, which can lead to people manipulating the system to benefit specific patients/populations rather than working to improve the system for all. This manipulation often results in more flaws and a downward spiral. Although these problems affect decision making at all levels of the system, from client- clinician discussions to ministerial resource distribution decisions, a broader societal impact is felt as one moves up the hierarchy of decision making. At all levels, both the rationale for making specific decisions and the process whereby decisions are made should be clear to those affected by the decisions. In the discipline of ethics, these two concerns are captured in the concepts of ‘distributive or substantive justice’ (what priorities should be set; the focus here is on outcome) and ‘procedural justice’ (how should priorities be set; the focus here is on process) (Gibson et al., 2005; Williams & Yeo, 2000). Several complicating and competing healthcare goals impact both of these forms of justice (see Table 1.3). As with economics, the discipline of ethics contributes both a way of thinking and a set of tools. In the case of ethics, these contributions help to address ethical goals of legitimacy and fairness in priority setting – in particular as these relate to justice. 25 Table 1.3 Factors that impact distributive and procedural justice 12 Distributive justice is impacted by questions such as: Procedural justice is impacted by questions such as: Is the goal the best outcome, the greatest good for the most people, social equity 13 , or helping the sickest or worst off, or addressing the most urgent needs? How much priority should be given to each group? How should ability to pay and the ‘rule of rescue’14 be factored in? Should priority be set based on a first come, first serve basis or on a ballot system? Should there be inclusion/exclusion criteria, and should the process be explicit or implicit? How do we know the right people are being prioritized? How do we resolve conflicts? Priority setting inherently involves conflict and controversy when decisions or policies are made that serve only some demands or needs at the expense of others. These decisions or policies often involve conflicting beliefs and values, especially regarding the values that must be compromised when all values in a particular situation cannot be honoured and maintained (Roy, Williams, & Dickens, 1994). The treatment versus prevention or acute care versus health promotion arguments are good examples of this debate. The decisions become even more difficult when they involve life-and-death consequences; e.g., the ‘tragic case of child B’ in the UK, where a father fought for funding coverage for an expensive investigational treatment for his daughter’s life-threatening cancer. Key lessons learned from the Child B case clearly highlight some of the concerns outlined above. Lessons from the Child B case include the following (Ham, 1999): Funding experimental and costly treatments raises ethical and practical dilemmas Concern to use resources for the benefit of the population must be weighed against the urge to respond to individual needs The priority setting and decision making processes must be rigorous and fair Decision makers should explain the reasons behind decisions, show their relevance, give opportunities for appeal, and ensure the process is regulated. 12 (Gibson et al., 2005; Martin & Singer, 2000; Williams & Yeo, 2000) 13 Equity is about fair distribution of resources and/or benefits; as such the underlying premise is about people in similar situations being treated similarly (Griffiths, Reynolds, & Hope, 2000; Jefferson et al., 2000). 14 Jonsen coined the term ‘Rule of Rescue’ (RR) to describe the imperative people feel to rescue identifiable individuals facing avoidable death (McKie & Richardson, 2003). 26 These lessons, and the conflict and controversy inherent in priority setting that are outlined above, support the use of ethical priority setting approaches like Accountability for Reasonableness (see below). Approaches of this type highlight and address ethical concerns to help decision makers as they strive to achieve fair outcomes. 1.5.3.3.1 Accountability for Reasonableness (A4R) Accountability for reasonableness is an ethical priority setting approach designed to offer a practical framework for fair priority setting (Gibson et al., 2005). Developed by Daniels and Sabin and based on justice theories of democratic deliberation, it is designed to provide practical guidance to improve the fairness of, and enhance public accountability for, priority setting (Gibson, Martin, & Singer, 2004). Accountability for reasonableness identifies five conditions of a fair priority setting process: relevance, publicity, revision, empowerment, and enforcement (see Table 1.4 for details). It supports decision makers with managing conflicting values that cannot be “resolved by an appeal to science and where the search for some formula or set of principles designed to provide decision-making rules will always prove elusive” (Klein & Williams, 2000, pp. 20-21). Table 1.4 Accountability for Reasonableness 15 Relevance Decisions based on reasons that are relevant under the circumstances Publicity Decisions and the reasons for them are transparent and publicly accessible Revision Mechanisms available to revisit/revise decisions and to resolve disputes Empowerment Power differences minimized and effective participation optimized Enforcement Regulation and mechanisms to ensure these above conditions are met Insofar as A4R addresses the process of debate and deliberation required in priority setting – i.e., procedural justice – simply getting the process right is not enough to ensure that the answers are right, i.e., the right priorities are set for the given situation. Efforts are also required to determine the effectiveness of the various options available 16 and the associated costs. This information should also be built into the priority setting process to attend to the 15 (Daniels & Sabin, 2002; Gibson et al., 2005) 16 Daniels refers to this as market accountability, which distinguishes it from Accountability for Reasonableness (Daniels, 2000). 27 distributive justice component of ethics. Thus, it appears that ethics alone cannot adequately inform priority setting. 1.5.3.4 Successful priority setting Although ethics and economics have been and continue to inform priority setting practice, contextual factors inherent in healthcare culture complicate priority setting and, therefore, influence real-world priority setting and resource allocation decisions. Due to their different aims, economics and ethics illuminate different factors to consider when setting priorities and making decisions, and thus contribute different approaches and tools. Some approaches focus more on outcomes; others more on process. And, in the priority setting literature to date, no one approach “stands out as the gold standard” (Mitton, Patten, Waldner, & Donaldson, 2003, p. 1653). Research into what constitutes successful priority setting has found several key elements, including the following: stakeholder understanding and engagement, resulting resource reallocation, decision making quality, stakeholder acceptance and satisfaction, positive externalities, an explicit process that considers values and context, information management, and inclusion of a mechanism to appeal or revise decisions (Sibbald, Singer, Upshur, & Martin, 2009). Different priority setting approaches provide the structure to fulfill these elements to varying degrees. For example, the ethical framework Accountability for Reasonableness (Daniels & Sabin, 1998) (described above) focuses more on process explicitness and inclusion of an appeal mechanism than on other elements noted above. The economic framework Program Budgeting and Marginal Analysis (described below) focuses more on resource reallocation and stakeholder engagement, although PBMA has evolved to incorporate more of the other success elements in recent years (Peacock et al., 2010). 1.5.4 Program Budgeting and Marginal Analysis (PBMA) PBMA is a framework designed to assist decision makers in making choices around limited resources. It does so through operationalizing the economic principles of opportunity cost and the margin. PBMA has been used in the healthcare field since the 1970s and is currently 28 being used in health authorities in several Canadian provinces. PBMA’s seven stages are outlined in Table 1.5 below. PBMA is often categorized as an economic approach to priority setting. It merges two distinct but often linked economic activities: program budgeting (PB) and marginal analysis (MA). Program budgeting, sometimes viewed as a planning framework, describes the distribution of resources across different programs; it does so by providing a map of expenditures and sometimes activities for the various programs (Jefferson et al., 2000; Mitton & Donaldson, 2004b). Marginal analysis is the examination of the costs and benefits of small changes in the existing pattern of expenditure in a particular setting or portfolio (Jefferson et al., 2000; Mitton & Donaldson, 2004b). Both PB and MA are techniques that can be and are used on their own. When merged together in the PBMA framework, questions exist regarding whether or not program budgeting is truly necessary (Mitton & Donaldson, 2004b); however, it has evolved as a combined activity in this priority setting approach. A brief literature review of PBMA implementation reveals an evolution in the process used to implement PBMA from the mid 1990’s to the present. It must be noted however that there is no ‘one way’ of implementing PBMA (Mitton & Donaldson, 2004a); it requires adaptation to each local context. An example of PBMA conducted in 1994-5 in Scotland (Ratcliffe, Donaldson, & Macphee, 1996) reveals a 5-stage process (stages 1, 2, 5, 6, 7 from Table 1.5 below), which emphasizes the PB and MA stages of the activity. Bohmer et al. (2001) in their review of a PBMA exercise in New Zealand in the late 1990’s, also emphasized the PB and MA stages; however, they also mention additional steps added regarding the advisory panel and criteria (stages 3 & 4 in Table 1.5). These two stages (3 & 4) put emphasis on contextualizing the economic and rational approach facilitated by PBMA. Current PBMA literature (Mitton & Donaldson, 2004a; Peacock et al., 2006) includes all seven stages outlined in Table 1.5 as integral to the PBMA process. As PBMA has evolved, other activities have been found to facilitate its implementation and adoption for future use. For example, support with the priority setting process and education 29 on relevant economic principles and PBMA, provided to decision makers by researchers, have been found to be beneficial. Peacock (1998a) determined education to be a valuable precursor to transforming decision making practices. Also, organizational buy-in is often facilitated with support from an internal champion with previous PBMA experience and assistance from an internal project coordinator (Mitton & Donaldson, 2003a). Since decision making does not end following completion of a PBMA process, each subsequent cycle within an organization should be designed to build on the last, enabling the process to be refined by incorporating lessons learned from earlier experience. The intention is that during subsequent iterations of PBMA, the role of external supporters (such as researchers or health economists) will diminish and greater internal leadership will take shape thereby fostering acceptance and long term process sustainability (Mitton et al., 2003). These activities serve to further contextualize PBMA and are thus being used to facilitate PBMA implementation today, although they are not explicitly articulated in the PBMA implementation framework described in the literature. 30 Table 1.5 Stages in a PBMA priority setting exercise 17 1. Determine the aim and scope of the priority setting exercise Determine whether PBMA will be used to examine changes in services within a given program (micro/within program study design) or between programs (macro/between program study design). 2. Compile a ‘program budget’ The resources and costs of programs may need to be identified and quantified, which, when combined with activity information, is the program budget. 3. Form a ‘marginal analysis’ advisory panel The panel is made up of key stakeholders (managers, clinicians, consumers, etc.) in the priority setting process. 4. Determine locally relevant decision-making criteria To be elicited from the advisory panel, with reference to national, regional and local objectives, and specified objectives of the health system and the community (e.g., maximizing benefits, improving access and equity, reducing waiting times, etc.). 5. Identify options for (a) service growth, (b) resource release from gains in operational efficiency, and (c) resource release from scaling back or ceasing some services The program budget, along with information on decision-making objectives, evidence on benefits from service, changes in local healthcare needs, and policy guidance, are used to highlight options for investment and disinvestment. 6. Evaluate investments & disinvestments Evaluate, in terms of costs and benefits; make recommendations for funding growth areas with new resources and/or moving resources from 5 (b) and 5 (c) to 5 (a) above. 7. Validate results & reallocate resources Re-examine and validate evidence and judgments used in the process and reallocate resources according to cost-benefit ratios and other decision-making criteria. 1.5.4.1 PBMA and economics, ethics and evidence-based practice Various studies into PBMA’s use have determined that it effectively incorporates both economic (Mitton & Donaldson, 2004b) and ethical principles (Gibson, Mitton, Martin, Donaldson, & Singer, 2006). The PBMA process addresses many of the economic and ethical concepts and principles that assist priority setting and decision making as outlined in Table 1.1 (p. 13 above) and described in the economics and ethics sections above. Compared 17 (Peacock et al., 2006) 31 to other economic approaches to priority setting, PBMA provides the practical relevance found to be lacking with these other approaches (Hoffmann, 2000) and has been found to be more responsive to the needs of decision makers (Ross, 1995). And when it comes to ethics, PBMA attends to both distributive and procedural justice, and facilitates a priority setting process that addresses the key ethical goals of legitimacy and fairness (Gibson et al., 2005; Martin & Singer, 2000). As for evidence-based practice, PBMA is as evidence-informed as time, data and decision maker expertise and capacity for evidence-informed decision making allows. PBMA supports a broad definition of evidence by incorporating several sources of evidence into the process including the following: cost and benefit data for the various services under consideration; output or activity information for the various services; expert opinion; evidence from the literature; and policy guidance. PBMA also directly links priority setting to the organizational values and vision so that these can directly inform the process. 1.5.4.2 Successful and unsuccessful PBMA exercises PBMA has been successfully used in a variety of settings (e.g., Bohmer et al., 2001; Mitton & Donaldson, 2003a; Mitton et al., 2003; Scott, Currie, & Donaldson, 1998); however, it has not been universally successful (e.g., Halma, Mitton, Donaldson, & West, 2004; Miller & Vale, 2001). Unresolved issues exist regarding how best to implement PBMA. Issues identified in the literature include difficulty obtaining adequate information for the program budget (Craig, Parkin, & Gerard, 1995; Twaddle & Walker, 1995), difficulty with disinvestment (Ruta, Mitton, Bate, & Donaldson, 2005), limited monetary impact (Urquhart, Mitton, & Peacock, 2008), difficult-to-measure outcomes (Mitton & Donaldson, 2003a), organizational barriers to adoption (Mitton & Donaldson, 2003a), and questions of long term sustainability (Mitton & Donaldson, 2001). 1.5.4.2.1 Lessons from unsuccessful exercises Although a great deal has been learned about PBMA implementation from the many successful exercises, much can also be gained from exploring those experiences that did not go as anticipated. As identified above, there are several cases where PBMA experiences highlighted areas for improvement. Miller & Vale (2001), researching why PBMA was not adopted by the UK NHS (National Health Service), found barriers to PBMA implementation described by respondents as `institutional inertia’. These included the reactive versus 32 proactive environment, the fear of `unnecessary’ explicitness, and the demand for concrete evaluation evidence. They conclude that implementation strategies and future research on such commissioning innovations needs to focus on institutions and the interventions. Halma et al (2004), researching PBMA implementation in rural Alberta, found that redesign efforts were hampered by the groups’ inability to come up with areas for resource release, lack of experience, belief that all programs were already operating with ‘bare bones’ funding, and perception that they had minimal control over the budget. In both of these examples, the barriers experienced were related to the organization adopting PBMA rather than specifically to the PBMA tools. These barriers were the limiting factors in these organizations achieving resource reallocation as a result of implementing PBMA or in these organizations adopting PBMA as their priority setting approach. These barriers are important to dissect and report. It is also important to explore these barriers in detail when developing specific strategies to overcome or manage them. This enables other organizations implementing PBMA to learn from past PBMA implementation experiences and strategies used to overcome or manage barriers. 1.5.4.3 Barriers and facilitators to using PBMA Several barriers and facilitators to explicit and rational priority setting (as supported by PBMA) have been highlighted in the literature and are listed in Table 1.6 below. Some of these facilitators, such as inclusion of a champion and use of incentives, have been incorporated into the PBMA process. Other facilitators, such as strong leadership and a learning culture, are more difficult to address. At best, potential PBMA users can assess the organizational context for these facilitators; however, if these facilitators are found to be lacking, they may be difficult to incorporate. This is an area that requires further study. The barriers listed in Table 1.6 below, and other practical challenges healthcare decision makers face when trying to make resource allocation decisions, negatively impact the success of structured and rational approaches to priority setting (Peacock et al., 2006). Thus, PBMA and its implementation process would be well-served by potential PBMA users paying particular attention to these barriers and developing strategies to overcome them. One 33 method to address barriers is to use an implementation framework (e.g., Greenhalgh and colleagues (2004) conceptual model for the diffusion of innovations in healthcare organizations 18 ) to guide both the assessment of barriers and the development of strategies to manage them. Implementation frameworks can also enable implementers to take varying perspectives of an innovation like PBMA and its implementation in order to address concerns from these varying outlooks. For example, the potential users of PBMA are decision makers. Therefore, it is prudent to examine their perspectives of both the problem that PBMA is intended to help solve, and their experience managing the problem and implementing PBMA (if and when they have experience with it). This can help decision makers to adapt PBMA and the implementation process to best suit their context and meet their needs, which will have direct bearing on PBMA usability and adoption, and its sustained use in the organization. Exploring priority setting and PBMA implementation from the decision makers’ perspectives was a focus in this thesis research. So, too, was ensuring that the implementation was evaluated on the basis of both the intervention (e.g., PBMA) and the context (e.g., institutional factors) (Miller & Vale, 2001). 18 See Appendix B for details of this model. 34 Table 1.6 Barriers and facilitators to explicit and rational priority setting 19 Barriers Facilitators Lack of genuine buy-in Competing demands, priority setting as low priority activity; lack of time Limited incentives for change; i.e., no fiscal pressure to disinvest, misalignment of incentives, organizational inertia Political interference; i.e., politics trumps evidence- based practice and/or evaluation Change in personnel Lack of trust between stakeholders and with the process Fear of ‘unnecessary’ explicitness, desire to avoid political fallout from explicit decisions Physicians not on board No (real or perceived) authority to change; lack of autonomy, incentive, and/or power to innovate independently Lack of allocation experience, health economics knowledge, and clarity Vertical budget silos Lack of sufficient resources; i.e., portfolio ‘too small’ Unrealistic expectations of participants Lack of relevant data/information regarding health outcomes, benefits of services, and/or concrete evaluation evidence Reactive versus proactive environment; tactical behaviours that interfere with rational priority setting Aversion to unilateral innovation High level champion Strong leadership Learning culture Culture conducive to change Integrated budgets Resources earmarked for both the process and to follow up on the proposals Built-in incentives for appropriate and efficient spending 1.5.4.4 Unresolved issues in the literature this research is filling Use of a structured priority setting approach like PBMA is intended to increase the rationality of decision making. It also addresses important concepts in economics and ethics. PBMA has evolved from its format in the first phase of priority setting (i.e., a more rational, technical approach) (Ratcliffe et al., 1996), to a more contextualized approach in the second phase (i.e., more focus on process and contextual relevance) (Mitton & Donaldson, 2004a; Peacock et al., 2006). This coincides with the evolution of priority setting more generally. It also aligns 19 (Miller & Vale, 2001; Mitton & Donaldson, 2004a; Mitton & Donaldson, 2004b) 35 with the US Institute of Medicine’s recommendations for setting priorities for health technology assessment and guideline development: combine analytic, data-intensive approaches with a consensus building process, and use explicit criteria to ensure the process is not based only on implicit judgment (Donaldson & Sox, 1992). As we experience the third phase of priority setting, one that appears to be focusing on fit and usability of priority setting approaches in real world situations (Robinson, 1993), PBMA is in a good position to evolve accordingly. This focus on fit and usability, particularly as they relate to effectiveness of a priority setting approach (using PBMA as the example), is one area that this thesis explores. The second major area of focus in this thesis is the implementation process and its role in changing priority setting practice. Since adoption of PBMA is a key prerequisite for PBMA effectiveness, and since it is the “interaction among the innovation, intended adopter(s) and a particular context that determines the adoption rate” (Greenhalgh et al., 2004, p. 598), this interaction – i.e., implementation – is an important area to examine along with fit and usability. Identifying barriers and facilitators to PBMA implementation and then tailoring implementation strategies to address these appears to be an efficient approach to implementation (Logan & Graham, 1998). Previous PBMA work has identified barriers and facilitators, and the PBMA implementation approach currently used has addressed some barriers and facilitators (as discussed earlier). However, priority setting research should continue to evaluate the implementation process used – to both improve it and to determine if it is contributing to success/failure of priority setting processes like PBMA in various contexts. By routinely evaluating and validating the PBMA implementation process, the feasibility and usefulness of the process can then be tested in a variety of contexts. This will add to the overall understanding and empirical knowledge base for priority setting research and for the complex field of implementation science in general, which is found to be lacking empirical work in many areas (Greenhalgh et al., 2004; Grol et al., 2007). There are several areas that remain un-researched within implementation science in general (Greenhalgh et al., 2004) that are also relevant to implementation of priority setting approaches. For example, since there is “very little direct empirical evidence on how to identify, and systematically harness the energy of, organizational champions” (Greenhalgh et al., 2004, p. 603), this is one area which should be further researched in the priority setting field. Other areas relevant 36 to priority setting include how a tension for change can be created in an organization and how ‘innovation-system fit’ can best be assessed (Greenhalgh et al., 2004). Finally, developing and validating suitable implementation processes for priority setting approaches is a precursor for researching the adoption and sustainability of priority setting approaches like PBMA. The final major area of focus in this thesis is the impact from using a structured and rational priority setting approach. A 2001 systematic review evaluated PBMA impact using the system-level (i.e., impact across a set of interrelated units, like an organization, that share a common goal (Rogers, 2003) criteria of resources reallocated or priorities set (Mitton & Donaldson, 2001). Using these criteria, the authors found that PBMA had a positive impact in 59% of the cases. A more recent literature review evaluated PBMA success using various definitions of success and found that the success rate was highest (65%) when success was defined as ‘implementation of some/all of the advisory panel’s recommendations’ and lowest (22%) when success was defined as ‘adopting the framework for future use’ (Tsourapas & Frew, 2011). The differing definitions of positive impact or success make it difficult to compare studies. Some authors have also reported limited impact of PBMA on specific outcomes – e.g., limited effectiveness (Bohmer et al., 2001), limited monetary impact (Urquhart et al., 2008), limited evaluation and/or difficult-to-measure outcomes (Mitton & Donaldson, 2003a). Yet, even in these examples, participants often indicated that PBMA was “worthwhile and valuable” (Bohmer et al., 2001, p. 47) and that it had a positive impact. If the ultimate goal of using PBMA is to reallocate resources to redesign services to better meet organizational objectives (Peacock et al., 2010), determining success by this outcome makes evaluative sense (Patton, 1997). Resource reallocation is one aim of PBMA; however, there are other aims and thus a broader definition of impact is warranted. Broadening the definition of impact, and defining impact from the decision makers’ perspective, was a goal in this study. 1.5.4.5 Summary of PBMA relevant to the objectives of this thesis PBMA is a framework that facilitates explicit and rational priority setting to support resource allocation decisions. It has been successfully used in many different healthcare contexts; 37 however, there are also several examples in the literature of difficulties with its implementation and contextual factors that influence its effectiveness in some cases. The literature identifies several barriers to explicit and rational priority setting, including lack of buy-in, political interference, lack of trust by key stakeholders, and limited incentives for change. “Theoretically, elegant policies [innovations], such as PBMA, become white elephants if they cannot or will not be implemented. Understanding how organizations work, however, will aid the development of priority-setting innovations and strategies to implement them successfully” (Miller & Vale, 2001, p. 164). These barriers to rational priority setting that have been described in the literature support the focus on PBMA effectiveness, impact, and implementation in this thesis. 1.5.5 Summary of literature review Since healthcare is not synonymous with health and all health services are not of equal value and effectiveness, we (as society) cannot focus on the delivery of healthcare to the exclusion of the pursuit of health (Kitzhaber, 1993). This requires critical assessment of what the healthcare system provides, and has historically been tasked with, regarding the pursuit of health. Difficult decisions are required. The data to support these decisions – e.g., clinical effectiveness, comparative analyses between different options including comparable costing and benefit data, locally-relevant and timely statistics on need – is often incomplete and/or unavailable in order to simplify the decisions. Some worry that effectiveness data (required by formulaic approaches) may always be inadequate (Dixon & Welch, 1991). Thus, priority setting requires robust and effective methods that go beyond formulaic approaches. Focusing on both the process and outcomes of priority setting and decision making is an important step in this regard. This evolution in priority setting practice may help address concerns that “even where information about costs and outcomes is available, the interpretation of this information, which may itself be incomplete, may be disputed” (Ham & Coulter, 2000a, p. 245). It is a simple fact of economic life that healthcare resources will be allocated across the various services available from year to year (Tolley & Whynes, 1995). Therefore, if priority setting does not occur explicitly and with due process, it is possible that resources will be 38 allocated according to any priorities rather than those determined deliberately using economic and ethical principles. Bringing principles and tools from the disciplines of ethics and economics together into one priority setting framework, such as PBMA, can help in this respect. PBMA goes further, however, in also focusing on education of decision makers and the bargaining process itself. These areas have been identified as potentially important aspects of the ‘third phase in the rationing debate’, moving the focus beyond the technical and process orientation of the first and second phases respectively, to “seek a synthesis which reflects the complexities that exist in practice” (Ham & Coulter, 2000a, p. 250). This may serve to keep in the forefront of decision makers’ minds the important fact that at the end of each decision lies an individual and their health (Muir Gray, 2001). 39 2 Chapter: Methodology and Methods 2.1 Overview of research chapters The following research chapters describe the methodology (including guiding philosophical perspective), key constructs, research questions, and methods (these sections are presented in chapter 2), and results with corresponding discussions (chapters 3, 4 and 5). The results are presented in three overarching and interrelated themes that emerged during concurrent data collection and analysis that took place during this three-year study. The three overarching themes are: PBMA fit, implementation and impact. 2.2 Methodology This qualitative study was guided methodologically by action research and epistemologically by social constructivism. As such, action research and social constructivism together informed the research process, the methods used, the decisions made regarding analysis and documentation of the findings, and, ultimately, the knowledge claims made (Carter & Little, 2007; Kaplan, 1964). They also influenced the two-way knowledge exchange between researchers and decision makers required in this research. The goal of this section is to articulate how these philosophical and methodological foundations, and my position in the research setting, affected the quality of this work and also to describe actions taken to achieve rigour, or trustworthiness (Lincoln & Guba, 1985), in this research. 2.2.1 Social constructivist perspective A central criterion of rigour and validity in qualitative research is to situate oneself epistemologically in the work and then to strive for consistency and coherence with the philosophical approach throughout the study (Holloway & Todres, 2003). This can be done by describing the philosophical and methodological approach(es) the researcher takes in conducting the work, and by being both reflective and transparent about how these affect the research activities, including the analysis and documentation of the findings. Qualitative research was suitable for this project due to its interpretive nature – we were interpreting priority setting as a social phenomenon in terms of the meanings participants 40 brought to the phenomenon (Pope & Mays, 2006). Therefore, as researchers, we asked “fundamental and searching questions about the nature of [priority setting]” (Pope & Mays, 2006, p. 4) rather than accepting common knowledge and assumptions often made in the priority setting field. For example, an overarching and emergent finding of our research explored the fit of a rational priority setting approach in the community care context we studied. This means we did not assume that the rational priority setting approach would or could fit and, therefore, move directly into measuring the impact of the rational approach used. This interpretive approach lends itself well to social constructivist philosophical perspective taken when conducting this research. “Constructivism is a theory about knowledge and learning…[it] describes knowledge not as truths to be transmitted or discovered, but as emergent, developmental, non-objective, viable constructed explanations by humans engaged in meaning-making in cultural and social communities” (Twomey Fosnot, 2005, p. ix). This perspective informed the assumptions and conceptualizations made, and shaped how the research was conducted and documented in this thesis. For example, the conceptualizations of the terms ‘fit’ and ‘impact’ described and used in the chapters below are based on the researchers’ and participants’ collaborative interpretation of the situation, with these conceptualizations evolving inductively as the study progressed. As is often the case in qualitative research, “terms as defined by participants are of primary importance” (Creswell, 2007, p. 19) and thus are emphasized in reporting the findings below. In seeking to address PBMA effectiveness using fit and impact from multiple decision maker participant perspectives, including the contextual influence, a deductive, scientific approach did not support the inquiry adequately. A scientific approach is well-suited when the research emphasizes empirical data collection and cause-and-effect relationships (Creswell, 2007). In comparison, a social constructivist perspective supports development of subjective meanings from multiple perspectives (Creswell, 2007), which was suitable for this inquiry. Social constructivism also recognizes social, historical and political contextual influences on constructed meanings (Creswell, 2007), which again was necessary for this inquiry. Furthermore, the iterative nature of action research (see Action Research section below) 41 supported evolving definitions of fit and impact as this work progressed which influenced how these concepts were operationalized in answering the research questions. Researchers and participants work together in action research which shifts responsibilities, like defining key terms, from an individual to a collective activity (Reason & Bradbury, 2008) resulting in evolving definitions as the working relationships develop. In the social constructivist paradigm, knowledge is constructed to satisfy knowers’ social needs and experiences, and therefore has multiple perspectives (Vogt, 2005). Research strategies rooted in constructivism are well-suited for the context-dependant knowledge – in multiple forms, from decision makers’ everyday practice/culture and perspective – which was necessary to inform this project. This theoretical perspective and its philosophical assumptions underpinned the approach used. 2.2.2 Positionality Traditionally, action researchers were seen as outsiders coming into a setting to facilitate change (Herr & Anderson, 2005). However, an action researcher can take various positions within the research setting depending on the research and researcher’s goals and the researcher’s relationship to the setting and role in the research. This positionality has been described as a continuum by Herr and Anderson (2005), ranging from an insider studying ones’ own practice to an outsider studying insiders. Using Herr and Anderson’s continuum, for this study my position and that of the academic research team can be described primarily as ‘outsider(s) in collaboration with insider(s)’. Our ultimate position – ‘reciprocal collaboration (insider-outsider teams)’ – lies at the middle of Herr and Anderson’s continuum. At times, such as when making key decisions regarding PBMA implementation, we achieved this positionality; however, for pragmatic reasons we could not maintain this positionality throughout the study. Researchers occupy several positions in relation to their research sites (Herr & Anderson, 2005). Along with the positionality regarding the researcher relationship with participants, a researcher’s position also stems from their location within/outside the research context (Herr & Anderson, 2005), which in this study was the community care organizational group. My 42 position in this respect was as both a contract clinician and a researcher. My clinician position influenced the research in several ways. My ontological position stems from my past/present clinician training and experience and this influenced the perspectives I brought to my role as researcher in this project. My ‘knowing’ is from this perspective (Reason, 1994). Although my clinical practice prior to and during this study was as a contractor in community care, my clinical role was not part of this study. My prior professional experience did mean that I had relationships with some of the decision maker participants prior to the study; however, these pre-existing relationships were not related to priority setting which was the underlying area of interest in this study. These past relationships likely did influence participant-researcher collaboration and communication. For example, particularities of the context and client-service experiences which were used as examples to clarify points during interviews were sometimes not explored in detail. Clarifying and reflecting on my position was necessary to ensure that potential assumptions were illuminated and issues of bias, validity and research ethics were addressed (Herr & Anderson, 2005). My clinician perspective had bearing on how I experienced the research, analyzed the data, interpreted and articulated the findings, and on the knowledge claims made as a result of this work. This can introduce bias into the study and should be explored with this in mind. However, bringing multiple perspectives can also enable “crafting [of] uniquely complex understandings of the research” (Herr & Anderson, 2005, p. 44). Specifically reflecting on how I “influenced and was influenced by” (Sandelowski, 1986, p. 30) the priority setting subject and the experience of implementing PBMA enhances the internal validity, or creditability (Lincoln & Guba, 1985), of this research. Researcher experiences are influenced by our perspectives, but through regular cycles of action and reflection we can be conscious of these influences (Reason, 1994). For example, reflecting on how my clinical background influenced the research highlighted for me the similarity between dietary/lifestyle change at an individual level and the individual and group-level priority setting behaviour change explored in this study. In comparison to my clinical perspective, a political scientist or health economist participating in this work may have different ontological positions and would likely bring different assumptions to bear during the research activities and on the findings. 43 2.2.3 Action Research This project was guided by action research (AR) methodology. Action research, also referred to as participatory research, collaborative inquiry, and action learning, has its historical roots in the field of education, with psychologist Kurt Lewin generally receiving credit as founder (Kidd & Kral, 2005; Reason & Bradbury, 2008). Paulo Freire is recognized for declaring its potential to raise consciousness amongst the oppressed so that, through research, they can create their own conditions for change and improvement (Hart & Bond, 1995). Action research spans a broad range of approaches, from the experimental approach associated with Lewin, to an empowering approach arising from community development, to a professionalizing approach identified in nursing and education (Hart & Bond, 1995). There is a spectrum in AR, with some forms focusing on action (i.e., the more technical/experimental approaches focused on change) and others more on its collaborative or participatory nature (i.e., the more emancipatory/empowering approaches focused on developing participant skill) (Hampshire, 2000). This has resulted in the many terms used to describe it, with perhaps participatory action research (PAR) and AR being the most commonly used interchangeably. Action research can be seen as a distinct methodology from PAR, however, as it does not necessarily share PAR’s ultimate goal of social justice and emancipatory change (Hagey, 1997). Action research as used in this study and thesis refers to an approach to inquiry that embraces the principles described below in Table 2.1. 44 Table 2.1 Some principles of Action Research 20 An emphasis on action in the world, rather than representation of it A focus on change and/or improvement Educational for those involved; i.e., learning by doing A response to practical, at times pressing, issues; sometimes AR is referred to as having a social purpose Collaborative relationships with open communication; participation Co-researchers identifying themselves as in relation with each other and their contexts/cultures Inclusion of self-study; inclusion of others Embracing multiple ways of knowing and uncertain knowledge Knowledge creation as a collaborative process A value-based orientation A dynamic, open-ended and emergent process, often precluding pre-determination, which develops as co-researchers deepen their understanding of the problem and capacity to solve it Further to the PAR and AR distinction above, the level of participant involvement also varies. Participants can be involved in all aspects of planning, conducting and using the research and results, as in PAR. Alternatively, there can be minimal participant input in planning but strong collaboration on implementation and evaluation of the action component, whether that is education, a change in practice, and/or adoption of a new technique or tool (Stringer & Genat, 2004). This latter description aligns with the level of participant involvement in this study. All but one of the participants who eventually implemented PBMA were not involved in the original planning of the research project; however, they were all critical collaborators when it came to implementing and evaluating PBMA. In AR, participant and researcher roles may also change throughout a project as both the problem(s) and potential solution(s) come into focus. And, indeed, this occurred in our study. Action research is a process of systematic investigation to increase understanding of an issue and to acquire information with practical application to solve a specific problem (Stringer & Genat, 2004). Thus, it embraces the active aspect of knowing. Its purpose is to generate new knowledge and implement change (McNiff & Whitehead, 2006). This combined purpose is facilitated by the cyclical approach of AR. Working together, researchers and participants 20 (Hampshire, 2000; Martin, 2008; McNiff & Whitehead, 2006; O'Brien, 2001; Reason & Bradbury, 2008) 45 form a community of inquiry (referred to as co-researchers below) that engages in iterative cycles of action and reflection. In this study, researchers and decision makers worked together to implement PBMA to inform priority setting practice (implement change) and at the same time to evaluate it (generate new knowledge). The purpose of AR is also facilitated through its “orientation of change with others” (Reason & Bradbury, 2008, p. 1) where co- researchers work together as partners in research and problem solving. This is in contrast to researcher ‘outsiders’ coming into research with a desire to study and, thereby, sometimes (whether purposefully or unintended) to change ‘others’. The cyclical nature of AR is the foundation of its approach to inquiry. It involves the co-researchers engaging in relatively systematic cycles of action – to examine practices and gather evidence – and reflection – to make sense together and plan further actions to resolve problems (Reason & Bradbury, 2008; Stringer & Genat, 2004). In this study, we implemented PBMA to inform priority setting for two successive years. We collaboratively evaluated its fit, and how best to implement PBMA and determine its impact in this setting. Studying how decision makers adopted PBMA and engaged in and used knowledge, such as the economic principles PBMA is based on, provided insight into how PBMA could be implemented to maximize its fit in this context and impact on priority setting. Continual exploration from within a process (e.g., PBMA implementation) and problem (e.g., making difficult resource allocation decisions) enabled us to find what worked in this group’s practice. As a result, external validity (generalizability) or transferability (Lincoln & Guba, 1985) was traded, to some degree, for internal validity (situation-specific knowledge) (Swepson, 1995) or credibility (Lincoln & Guba, 1985). Both the researchers and decision makers in this study were ‘learning by doing’, a key component of action research (O'Brien, 2001). This can be helpful in shedding light on how people behave in practice. One of the difficulties with researching decision making and priority setting is that what people say they do may be different from what they think they do, which may be different again from what they actually do (McDonald, 2002). Thus when researchers participate with decision makers within the priority setting context, they can gain valuable insight into the nature of priority setting practice in real-time. This participation is 46 inherent in AR. When multiple stakeholders, such as decision makers and researchers, collaborate to address a joint problem (e.g., resource scarcity), it ensures that issues are not seen as belonging to one individual or group. This promotes understanding by these stakeholders of the issues, which facilitates solutions that are well-informed, practical, acceptable, and encourage self-reliance (Stringer & Genat, 2004). Coherence with action research methodology was sought by questioning and ensuring congruence between the study purpose, research questions, study design, data collection and analysis, and discussion of the findings. The research process, including research questions and data collection and analysis procedures, was modified as needed based on emerging findings. This methodological coherence is a verification strategy that helps to ensure validity of the resulting knowledge claims (Morse, Barrett, Mayan, Olson, & Spiers, 2002). 2.2.4 Validity Although controversy exists over the use of the terms validity and reliability in qualitative inquiry and in action research, I agree with Morse and colleagues (2002) and Herr and Anderson (2005) that these terms are still relevant to demonstrating rigour in both qualitative and action research. Rather than use the term trustworthiness (Lincoln & Guba, 1985) when addressing the rigour of this research, which meets the objective of coherence within the social constructivist philosophical paradigm, I believe using the terms rigour and validity is more appropriate to the priority setting field that this work is contributing to and to mainstream science in general. “Rigour is the means by which we show integrity and competence” (Tobin & Begley, 2004) and is thereby crucial to demonstrate in this research and throughout this thesis. In the thesis, and in particular here in the methodology section, I have referred to rigour and validity as they apply to qualitative research and have described the strategies used to demonstrate rigour and validity that are relevant to the qualitative action research approach used in this study. To demonstrate reliability and validity and thereby a rigourous research process, verification strategies were continually sought and implemented to shape the research and to identify and address potential problems before corrections were no longer possible. To start, this research 47 spanned three years, thereby facilitating prolonged engagement with participants and persistent observation in the setting, which are strategies to help validate the knowledge claims established through qualitative research (Morse et al., 2002). These strategies also facilitated cycles of plan-act-observe-reflect (Lewin, 1948) that were conducive to the ongoing problematization of the priority setting practice under study (Herr & Anderson, 2005). Use of a research journal and regular peer debriefing (discussed below) supported the regular reflection required to achieve process (appropriate research methodology) and catalytic (education of researchers and participants) validity in this study (Herr & Anderson, 2005). Another strategy was conducting data collection and analysis concurrently in an iterative process to enable a “mutual interaction between what is known and what one needs to know” (Morse et al., 2002, p. 18). Other overarching validation efforts used in this study include data, investigator and method triangulation. In part, triangulation was pre-planned in the study design, and in part, triangulation was sought for validation purposes during methodologically or analytically challenging times. Triangulation in both cases was used not as “a means of confirming existing data, but as a means of enlarging the landscape of [this] inquiry, offering a deeper and more comprehensive picture” (Tobin & Begley, 2004, p. 393). Triangulation was used to seek completeness, i.e., “to allow for recognition of multiple realities” (Tobin & Begley, 2004, p. 393). This study involved data triangulation (e.g., meeting attendance and interviews with participants), investigator triangulation (e.g., more than one researcher attended meetings, analyzed and interpreted the data), and method triangulation (e.g., data were gathered using interviews, observations, ongoing discussions and document analysis) in an effort to accurately represent and validate the findings (Polit, Beck, & Hungler, 2001). Another specific validation strategy included member checks of early conceptualizations of the findings with the participants to contribute to the iterative and concurrent data collection and analysis. Member checks of the final knowledge claims were not sought as member checks were seen as a continuous and integral component of data collection and analysis, rather than as a strategy to verify the overall findings at the end of the study (Guba & Lincoln, 1981). Peer debriefing, throughout the study and during the thesis writing, was 48 used to achieve analytic validity by regularly re-examining emergent themes and eventual knowledge claims. ‘Representation of voice’ (Arminio & Hultgren, 2002), indicating acknowledgment of multicultural researchers and participants, was attended to through both my reflection on my relationship with the participants and the research subject (Tobin & Begley, 2004). Representation of voice and inclusion of multiple voices was also addressed through the use of direct quotes from participants and inclusion of their particular terms for key themes that emerged during analysis. This is important to ensuring coherence with social constructivism and is thereby another strategy to strengthen the rigour of this work. In the results, I also sought to include quotes from many different participants (and indicated this by providing unique participant identifiers) rather than a few particularly eloquent participants. This was done, in part, to avoid ‘elite bias’ (Miles & Huberman, 1984) by establishing the perspectives of all participants in relation to the group rather than focusing on the more articulate or higher-status members of the group. Inclusion of multiple voices also helps to achieve democratic validity in action research by demonstrating how multiple perspectives were considered in the study (Herr & Anderson, 2005). Specific details of when and how some of the above-mentioned validation strategies were employed are discussed below in the data collection and analysis sections to contextualize them within the research process. 2.2.5 Context From 2006-2009, this research was conducted with community care decision makers from the Central Okanagan Local Health Area (LHA) of British Columbia’s Interior Health Authority (IH). This LHA covers a geographical area spanning from Peachland to Oyama, with the majority of clients located between these two communities, in Kelowna, British Columbia. At the time of this study, the Central Okanagan LHA provided service to a population of approximately 176,130 of whom 18.6% were 65 years of age or older (Interior Health, 2007). The Central Okanagan LHA community care portfolio had an annual operating budget of approximately $25.5 million (Canadian; $1CAD $1US) and provided 49 the following services: home support, community nursing, rehabilitation, case-management, adult day programs, some chronic disease management (e.g., Diabetes Clinic) and specialized residential programs, and the community-based portion of those services which cross the continuum from acute to community, such as social work, respiratory, and dietitian care. For these latter cross-continuum programs, only the community portion of these services was included in the PBMA project because that was the only portion financially managed through the community care director’s budget. Although community care provides other services, such as Health Services for Community Living, which provides health services to community-dwelling (primarily in group homes and family care homes) developmentally delayed adults, the director chose not to include these services in the PBMA project. Other programs which link to community care, such as Access and Transition Services (to support people as they transition from acute services to community services), were not included because the budget for these program was not under the control of the director. Despite the included programs being managed through one central site, the majority of these services were provided either at the clients’ residence or at various locations within the Central Okanagan community. Clients receive services either through physician or other health provider referral, or through self or family referral. Some clients are referred to receive ongoing community care services, such as home support for a frail elderly client living independently. Other clients are referred to receive short term service, such as home-based physiotherapy to help them rehabilitate with the expectation that they will no longer require the service after a period of time, e.g., someone referred for physiotherapy after hip replacement surgery. Researchers supported decision makers in implementing PBMA to set priorities to inform resource allocation for two budget cycles, spanning one fiscal year each. A standard approach to PBMA implementation was undertaken as described in the literature and outlined in Table 2.3 below. Organizational buy-in was promoted with support from an internal champion with previous PBMA experience and assistance from an IH project coordinator. Both the original community care director and the new community care director who joined the project part-way through the first PBMA implementation (in September 2007) were actively engaged in the project. Health economics and PBMA education was 50 provided by health economists on the research team to a broad community care stakeholder group before PBMA was implemented. 2.3 Defining key constructs – fit, desirability, acceptability, usability Early during data collection and analysis, fit emerged as a key theme in determining PBMA adoption and consequent effectiveness and impact. Fit refers here to being of suitable quality and form to meet the intended purposes and needs of the end-users. Fit was further divided into the following dimensions to probe more deeply into PBMA effectiveness and to highlight potential PBMA adaptations to improve PBMA adoption (a prerequisite for effectiveness): a. Desirability – did PBMA fulfill the features that participants identified, pre-PBMA, as aligning with their priority setting vision and as sought-after in priority setting practice? b. Acceptability – did participants deem PBMA suitable? Did they buy-in to the approach? c. Usability – did PBMA provide a functional priority setting process (and tools) that supported participants in establishing and meeting their resource allocation goals? 2.4 Research questions Given that PBMA has had mixed success, based on variable definitions of success in Canadian health authorities and elsewhere internationally, the goal of this research was to assess the effectiveness of PBMA in a community care context and to determine its impact. Due to the cyclical and iterative nature of action research, the research questions evolved as the study progressed. The five questions addressed in this thesis include the following: 1. At baseline, how did decision makers set priorities for resource allocation decisions, and what features do they desire in a priority setting practice? 2. Can PBMA help decision makers by providing an effective priority setting process that helps decision makers establish and meet their ultimate resource allocation goals? 3. Does PBMA fit the priority setting context, i.e., did PBMA meet the features decision makers identified as desirable in priority setting practice (research question 1) and is it acceptable and usable? 51 4. What was PBMA’s impact and how can this be determined? 5. How can PBMA be adapted (both the tool and the implementation process) to better fit the priority setting context (i.e., to be more desirable, acceptable and usable, from the decision maker perspective, in supporting resource allocation decisions). For example, what particular tools, information and resources are required by decision makers to facilitate implementation of the approach? Research questions 1, 2 and 4 were original research questions at the start of this study. They were revised slightly after the study began, in part to address participants’ developing needs and in part to address the focus in this action research study as that focus became more refined. Research questions 3 and 5 were added as the study progressed to address concepts (e.g., fit) that emerged as key to exploring PBMA effectiveness. Research questions 1, 3 and 4 add to current literature on PBMA use internationally. Questions 2 and 5 address current gaps in the literature specifically related to contextual factors that complicate priority setting by both identifying barriers and facilitators to PBMA implementation and developing potential solutions to the barriers. This helps to address effective use of PBMA in this and potentially other real-world contexts, which should help users in “realizing the full potential of PBMA” (Mitton, 2001, p. 180). 52 2.5 Methods 2.5.1 Overview and study design Using multi-method design (Morse, 2003), this study consisted of the following objectives and activities: To examine PBMA’s effectiveness in this context based on decision maker perceptions of baseline priority setting practice vis-à-vis desired practice, and their perceptions of PBMA usability and acceptability after using it in practice. Data were collected pre-PBMA to determine the gap between baseline and desired priority setting practice to help identify those features of priority setting that decision makers desire. This helped to inform whether or not PBMA fit the context with respect to meeting desired practice. Data were collected post year-1 and year-2 of PBMA to explore participant perceptions of PBMA usability and acceptability, the other two dimensions of fit that emerged during analysis. To explore the role of implementation in changing priority setting practice by describing and evaluating the process, and describing the experience and lessons learned implementing PBMA in this community care portfolio. Data were collected during and after two PBMA implementations, one for each of two sequential fiscal years, to address this objective. To describe the impact on the participants and the context of implementing a formal priority setting approach, using PBMA as an example, by asking: - Did PBMA affect resource allocation in this study? - Did PBMA change practice, including decision maker use of evidence in decision making? - Did PBMA change decision maker knowledge of and attitudes towards priority setting, including how decision makers learn from and use PBMA? Data were collected during and after the two PBMA implementations to address this objective. See Table 2.2 below and study design diagram (Figure C.1) in Appendix C for details about how the research objectives were met and research questions addressed. 53 Table 2.2 Research questions, data collection methods, link to research objectives Research question (Q) Data collection methods How question addresses research objective(s) 1. At baseline, how did decision makers set priorities for resource allocation decisions, and what features do they desire in practice? Pre-PBMA interviews and focus group to assess PBMA fit against what decision makers say they desire in a priority setting process Addresses desirability 2. Can PBMA help decision makers by providing an effective priority setting process that helps decision makers establish and meet their ultimate resource allocation goal(s)? Year-1 and year-2 interviews, meeting attendance, document review, and research journal – to assess PBMA fit in this context during 2 sequential PBMA implementations Addresses usability; addresses impact via usefulness and changes to priority setting practice 3. Does PBMA fit the priority setting context, i.e., did it meet the features decision makers identified as desirable in priority setting practice (Q1); is it acceptable, usable? Year-1 and year-2 interviews Addresses effectiveness; and fit via desirability, acceptability, and usability 4. What was PBMA’s impact and how can this be determined? Year-1 and year-2 interviews, meeting attendance, document review, research journal Addresses impact in terms of usefulness, effects on priority setting practice, and changes to decision maker attitudes towards and/or knowledge of priority setting 5. How can PBMA be adapted (both the tool and implementation process) to better fit the priority setting context, (i.e., to be more desirable, acceptable and usable, from the decision maker perspective, in supporting resource allocation decisions). For example, what particular tools, information and resources are required by decision makers to facilitate implementation of the approach? Year-1 and year-2 interviews Meeting attendance Document review Research journal Investigate implementation using concepts from organizational behaviour/ education and implementation science Investigate tools as above; consider adult education principles Investigate rational process using decision making and behavioural economics literature Addresses effectiveness and fit by exploring: If PBMA helps decision makers set priorities and use evidence If PBMA is understandable/ usable as designed, and provides a useful structure, or requires adaptation The support and information required to facilitate PBMA use to set priorities 54 2.5.1.1 Participants and recruitment With assistance from the Community Care Director, participants were purposively recruited from community care in the Central Okanagan LHA and adjacent LHAs to ensure inclusion of a broad range of perspectives and decision making experiences, covering various programs and services. Participants included community care decision makers and other stakeholders, e.g., clinical leaders and managers from programs that interact with community care. Prior to implementing PBMA, twenty-four participants were either interviewed or participated in a focus group. For the first of the two PBMA implementations, which occurred over two sequential fiscal years, a 12-member multidisciplinary priority setting advisory panel was formed. It was made up of key stakeholders, including the director, program managers, a community representative, community care business support and a consulting physician. It had representation from all service areas participating in the project scope. These twelve participants (also referred to as decision makers and managers in this thesis) were interviewed after the first PBMA implementation (year-1). Nine participants implemented PBMA in the second year and participated in post year-2 interviews. A change in community care leaders resulted in a smaller group in the second year. 2.5.1.2 Unit of analysis Two units of analyses were used. The first unit of analysis is the individual decision maker, which was used to explore and evaluate individual and group priority setting practice and the outcomes of PBMA from the individual decision makers’ perspectives. Second, the community care group was used as the unit of analysis to evaluate group priority setting practice and outcomes of PBMA via observations of impact on the community care group practice. 2.5.2 PBMA implementation Researchers supported decision makers in implementing PBMA to set priorities to inform resource allocation in the 2007/08 and 2008/09 budget cycles. A standard approach to PBMA implementation was undertaken as described in the literature and outlined in Table 2.3. Organizational buy-in was promoted with support from an internal champion with previous PBMA experience and assistance from an IH project coordinator. 55 Table 2.3 PBMA implementation PBMA Priority Setting Exercise Stages (Peacock et al., 2006) IH PBMA Priority Setting Exercise Stages 1. Determine aim and scope Determine whether PBMA will be used to examine changes in services within a given program (micro design), within a portfolio with several programs (meso- level) or between programs/portfolios (macro design). 1. The aim and scope were established by the research team, IH senior level management and the initial Central Okanagan community care director, and reviewed and confirmed with the new director who joined part-way through year-1. It was decided that this would serve as a pilot project with potential to roll-out organization-wide. It included several distinct programs from one portfolio (i.e., meso-level application). 2. Compile a ‘program budget’ Identify and quantify program resources and costs; combine this with activity information to obtain program budget. 2. Used cost centers as a visual representation of the resources available. 3. Form a ‘marginal analysis’ advisory panel The panel is made up of key stakeholders (managers, clinicians, consumers, etc.) in the priority setting process. 3. A 12-member multidisciplinary priority setting advisory panel was formed. It was made up of key stakeholders, including the director, program managers, a community representative, and community care business support and a consulting physician. It had representation from all participating service areas. 4. Determine locally relevant decision- making criteria To be elicited from the advisory panel with reference to national, regional and local objectives, and specified objectives of the health system and community. Criteria can be weighted for significance. 4. Locally relevant criteria were initially generated through a community care stakeholder brainstorming session. Researchers refined criteria by comparing them to those found in the literature, and the advisory panel refined them twice. Once the advisory panel accepted the criteria, researchers used them to develop a score-sheet to rate proposals (stage #5 below). The panel and other key stakeholders were individually surveyed to weight each criterion’s significance, with weightings then factored into the score-sheet 21 . 21 See Appendix D for the weighted criteria and the score-sheet. 56 PBMA Priority Setting Exercise Stages (Peacock et al., 2006) IH PBMA Priority Setting Exercise Stages 5. Identify options for (a) service growth, (b) resource release from gains in operational efficiency, and (c) resource release from scaling back or ceasing some services Investment and disinvestment options are highlighted using the program budget and information on decision- making objectives, evidence on service benefits, changes in local care needs, and policy guidance. 5. Advisory panel members outlined opportunities for service growth (investment proposals) and resource release (disinvestment proposals) from both increasing operational efficiency and from scaling back or discontinuing some services 22 . 6. Evaluate investments and disinvestments Evaluate all proposals in terms of costs and benefits; make recommendations to fund growth areas with new resources, and/or move resources from 5 (b) and 5 (c) to 5 (a) above. 6. The advisory panel rated proposals using the score-sheet to evaluate total costs (benefit lost from a disinvestment) or total benefit (gain from an investment) for each proposal. They then developed a list 22 of recommendations for funding growth areas by moving resources saved through increasing operational efficiencies and/or service cuts (as described in disinvestment proposals). 7. Validate results and reallocate resources Re-examine and validate evidence and judgments used in the process; reallocate resources according to cost- benefit ratios and other decision-making criteria. 7. The plan was, with support from key stakeholders and researchers, for the community care director to re-examine and validate the results. Then, based on the priorities set, she was to develop a resource reallocation plan. Due to factors described in the results chapters below, this did not happen. 2.5.3 Data collection Data were collected pre-PBMA implementation, throughout implementation and after each of two cycles (one per fiscal year) of implementing PBMA. Data were collected prior to PBMA implementation to explore the features decision makers said they desired in a priority setting process. Data were collected during and after the two sequential PBMA implementations to assess PBMA’s fit in the community care context and impact on priority setting. Data collection is described below according to these temporal stages; however, it 22 See Appendix E for the list of investment and disinvestment options, and for the ranked list developed after rating the proposals using the score-sheet. 57 must be noted that in line with the constant comparative method (see data analysis section below) data collection and analysis were conducted concurrently in an iterative process with emerging analytical findings used to inform the project: the ongoing PBMA implementation and subsequent stages of data collection including the interview questions. Informed consent was obtained prior to the focus group and individual interviews at all stages of the study. 2.5.3.1 Pre-PBMA Prior to PBMA implementation, one focus group (n = 4) and twenty individual semi- structured face-to-face interviews were conducted with participants. Interview and focus group questions (Appendix F), adapted from previous research (Mitton & Prout, 2004), were designed to facilitate both analysis of participants’ perceptions and reflections on baseline priority setting approaches vis-à-vis desired approaches, and the extent to which PBMA aligned with desired practice. The questions were pilot tested with one interviewee first, with minor revisions made prior to use with subsequent participants. The interviews and focus group were digitally audio-recorded and transcribed verbatim, and the transcripts were checked for accuracy (Morse & Richards, 2002). The focus group enabled participants to build on each others’ comments and ideas; the group dynamic provides an opportunity for participants to reflect on organizational priority setting practices. 2.5.3.2 PBMA implementations During PBMA implementation, one or two researchers attended all PBMA advisory panel meetings to both support and observe decision makers in implementing PBMA to assist and develop their priority setting practice. Data included detailed field notes taken during these meetings (participant observation) and after regular conversations with participants and the IH project coordinator. A research journal was used to keep track of the project and to assist with project management, communication, and reflection. Relevant documents including meeting agendas, minutes, and reports created and used during PBMA implementation were also collected and analyzed as needed to assist with the evaluation of PBMA implementation. Gathering data over the duration of the implementation enabled inclusion of data about real- time behaviour and real-time perspectives rather than relying entirely on participant recall (Rogers, 2003). 58 2.5.3.3 Post PBMA implementations After using PBMA to inform priority setting for one budget cycle, individual interviews were conducted as above with all twelve participants 23 who engaged in PBMA. After the second PBMA cycle, individual interviews were conducted with all nine participants 24 . A leadership change in community care resulted in a smaller group in the second year. As above, all interviews were recorded, transcribed, and checked for accuracy. The interview questions were tailored each year to reflect the current state of PBMA implementation and contained questions designed to evaluate implementation from the participants’ perspectives. The year-1 interview questions (Appendix G) were adapted from previous research and also contained questions designed to explore participants’ understanding of and engagement in PBMA, and to assess their desire to continue with the program. The year-2 interview questions (Appendix H) were different in that they also addressed, in part, participant perceptions of the usability of PBMA’s tools and approach. Both sets of interview questions were pilot tested with one interviewee first, with minor revisions made before using with subsequent participants. 2.5.4 Data analysis 2.5.4.1 Pre- and post-PBMA interviews Interview transcripts were first read through to become familiar with the data and to identify potential analytic categories (Pope, Ziebland, & Mays, 2000), then analyzed and coded using NVivo8 qsr qualitative software. The preliminary analytic categories included concepts developed from issues identified during the implementation experiences to date at the time and from concepts that emerged during familiarization with the data. Analysis occurred in several stages, both concurrently with and after data collection was complete, so that emerging concepts could be built into subsequent stages of data collection. The preliminary analytic categories were further developed and refined during coding. Topic coding was used to determine key concepts that arose in response to the interview questions, and open coding 23 Four of these 12 participants were also interviewed pre-PBMA; two of these 12 participated in the pre-PBMA focus group. 24 Eight of these 9 participants were also interviewed post year-1 of PBMA; four of these 9 participants were interviewed (or attended the focus group) both pre-PBMA, post year-1 and post year-2 of PBMA. 59 was used to identify emergent themes (Morse & Richards, 2002). Analytic memos were used to identify potential areas for further exploration in the field and with peers, or explanation via literature review or peer discussion. A constant comparison technique was used to discover patterns between the codes and the preliminary analytic categories, and between the codes with these latter patterns used to develop further categories with identifying attributes (Morse & Richards, 2002). These categories were added to or used to refine the preliminary analytic categories. This constant comparison continued until the categories were internally consistent (internal convergence) yet distinct from one another (external divergence) (Lincoln & Guba, 1985). These analytic categories were reviewed (in NVivo8 qsr so that linked coded text and analytic memos could be examined as needed), further refined and grouped together based on the analytical and theoretical ideas developing from the research and the ongoing literature review (Pope et al., 2000). Key themes were then selected. These themes were further examined and distilled through an iterative process of data and literature review and peer discussion until the key themes to inform the results of the research were determined. These themes were regularly re-examined by the researchers, along with the linked textual data and memos, when writing this thesis to ensure the final themes presented were an accurate representation of the findings. 2.5.4.2 Other qualitative data The field notes taken during meeting observation, and the meeting agendas, minutes and reports were also thematically analyzed – first for emerging concepts and ideas and then for recurring concepts and ideas relevant to PBMA fit, implementation and impact. The concepts and ideas generated from this qualitative data analysis were distilled to key themes during the iterative process of data collection and analysis, and during discussions with at least one additional research team member in peer debriefing that occurred several times during PBMA implementation and after. Specific to the impact theme that emerged during data analysis, interview, observation and document data were all analyzed for concepts related to PBMA impact. Regarding the document data, investment and disinvestment proposals generated during PBMA were analyzed to identify evidence used in their development (as one method of determining 60 impact) and to determine PBMA’s impact via the total dollar value of the proposals. The impact theme was further refined into the varying levels of impact as analysis progressed: individual level, group level and system level. Data were subsequently reanalyzed to determine the specific impacts at these three levels. 2.5.4.3 PBMA implementation evaluation Our evaluation was guided by insights from the utilization-focused evaluation, change management, and implementation science literatures. Utilization-focused evaluation guides information collection to support the intended use of the evaluation by the intended users (Patton, 1997). We evaluated PBMA implementation to determine what worked and what required further development to enhance PBMA use by decision makers in this particular context. Change management encompasses how “to make fundamental changes in how business is conducted in order to help cope with a new, more challenging…environment” (Kotter, 1995, p. 59). This is relevant to PBMA implementation since PBMA requires decision makers to change their individual and group priority setting practice while continuing to attend to their regular healthcare business routines. Implementation science was also relevant with its focus on methods to promote the integration of evidence into healthcare policy and practice (Schackman, 2010) since PBMA focuses, in part, on decision maker use of various sources of evidence in priority setting. Data were analyzed to identify barriers and facilitators to implementation, and then concepts from the above fields were used to develop strategies to address the barriers and facilitators. Strategies developed during the first PBMA implementation were incorporated in the second cycle of PBMA implementation. The goals of the evaluation were first, to adjust the implementation approach according to participants needs at the time, and second, to collect data on barriers/facilitators experienced and effective strategies to address these in this context. As such, our evaluation was formative, focused on adapting and refining PBMA and its implementation with the participants rather than retrospectively assessing its effectiveness. PBMA implementation evaluation was conducted by the researchers through iterative cycles of plan-act-observe-reflect that are germane to action research (Lewin, 1948). Data collection and analysis occurred concurrently, with data collected through participant interviews and 61 researcher attendance of PBMA meetings where detailed field notes about implementation were taken both during and after the meetings. These field notes were discussed in debriefing with at least one other research team member after the meetings. The data collection and analysis methods used to inform the implementation evaluation are described above. 2.5.5 Research ethics This study was approved by the Behavioural Research Ethics Board at the University of British Columbia and the Research Ethics Board of the Interior Health Authority. 2.6 Summary In summary, this action research project, guided epistemologically by social constructivism, explored PBMA implementation in a community care context to assess its effectiveness and to determine its impact in this setting. Data were collected pre, during and post PBMA implementation and the study spanned three years. Three overarching and interrelated themes emerged during the iterative and concurrent process of data collection and analysis: PBMA fit, implementation, and impact. The pre-PBMA data was used to determine the gap between baseline and desired priority setting practice to help identify those features of priority setting that decision makers desire. This helped to inform whether or not PBMA fit the context with respect to meeting desired practice. Data were collected post year-1 and year-2 of PBMA to explore participant perceptions of PBMA usability and acceptability, the other two dimensions of fit that emerged during analysis. Data were collected during and after two PBMA implementations to explore the role of implementation in changing priority setting practice by describing and evaluating the process, and describing the experience and lessons learned implementing PBMA in this community care portfolio. Finally, data were collected during and after the two PBMA implementations to describe the impact on the participants (e.g., priority setting practice, knowledge and attitudes) and the context (e.g., resource allocation) of implementing PBMA. In the next section, the results are discussed according to the three overarching themes: PBMA fit, implementation and impact. 62 3 Chapter: Fit – Results and Discussion 3.1 Synopsis of chapter With the emergence of the evidence-based medicine movement in the 1970s (Muir Gray, 2001) and derivative attempts to establish evidence-based practice in other clinical domains, it is not surprising that evidence-based decision making has appeared as a prescription for healthcare management. However, evidence-based decision making in healthcare has not been easy. As mentioned in the literature review, its formal rationality 25 , which requires clear objectives and clear means for producing results, can be an issue (Baker et al., 2004). Problems inherent in healthcare include shifting goals and difficulty determining what constitutes evidence and then obtaining relevant evidence to inform decisions. Despite these challenges, increased rationality and increased use of evidence in healthcare decision making are often still desired. Mechanisms to facilitate evidence-informed decision making are available. One approach is PBMA. The priority setting literature has established the efficacy of PBMA: i.e., it can work (e.g., Bohmer et al., 2001; Mitton & Donaldson, 2001, 2003a; Mitton et al., 2003; Peacock, 1998b; Ruta, Donaldson, & Gilray, 1996; Scott et al., 1998; Twaddle & Walker, 1995; Viney, Haas, & De Abreu Lourenco, 2000), providing justification for use. However, previous research has also indicated that contextual factors complicate priority setting (Mitton & Donaldson, 2003b) and thereby can hamper PBMA effectiveness (i.e., whether it does actually work) in some settings. This chapter presents the findings from this study that build on previous work (Mitton & Donaldson, 2004c; Mitton et al., 2003) by investigating PBMA effectiveness. It does so through elucidating and studying factors related to its contextual fit 26 , in particular in the community care context. The purpose of this chapter is to explore PBMA’s effectiveness (thereby addressing thesis objective #1, Section 1.4) in this context based on decision maker perceptions of baseline priority setting practice vis-à-vis 25 Rational decision making and evidence-informed practice have scientific evidence and a systematic approach as their foundation, whereas with decision making based on judgment or intuition “formal analysis is replaced by tacit knowledge based on experience” (Baker et al., 2004, p. 88). 26 The definition of fit used in this study is provided below in the results section. 63 desired practice, and their perceptions of PBMA usability and acceptability after using it in practice. Although PBMA was found to be a desirable innovation in this study, several factors contributed to its early discontinuation (prior to the planned year-two conclusion). This highlighted the need to study fields in addition to priority setting that could provide insight into PBMA effectiveness. These fields include complexity science, organizational change, and decision science, the latter of which supported delving more deeply into decision making practices and conditions, and why and how these might affect PBMA acceptability and usability. Insights gained will help us work with these conditions and better understand situational constraints – both of which will help to advance evidence-informed decision making. Ultimately, healthcare decision makers are faced with making difficult decisions to concurrently address growing client care demands and resource limitations. The goal of this work is to support decision makers in this activity. 3.2 Fit results As outlined in the methodology section above (Section 2.3, defining key constructs), fit emerged as a key theme in determining PBMA adoption and consequent effectiveness. Fit, in this study, refers to being of suitable quality and form to meet the intended purposes and needs of the end-users. Fit was further divided into the dimensions of desirability, acceptability, and usability (defined above in the methodology section) to probe more deeply into PBMA effectiveness and to highlight potential PBMA adaptations to improve PBMA adoption (a prerequisite for effectiveness). The results are presented in three parts below, corresponding with these three dimensions of fit. The pre-PBMA results focus on participants’ reflections on baseline priority setting practice vis-à-vis desired approaches, and the extent to which PBMA might match with desired practice. The post year-1 and year-2 results explore participants’ acceptability of PBMA and how usable they determined the process and tools to be after using PBMA to inform priority setting practice. 64 3.2.1 Pre-PBMA implementation – Desirability These results highlight the gap between the features which participants identified as key influences on or drivers of baseline priority setting practice and those they desire. We use the term drivers below to capture more than just an influence on priority setting practice, but also those contextual aspects and other compelling features that effect a change on or directly influence how the priority setting activity proceeds. Desired drivers are those features that participants indicated should receive greater emphasis in setting priorities. Baseline drivers are not necessarily negative or unnecessary; participants identified them as being the primary drivers which they wanted to reconcile with desired drivers. Three main themes (Figure 3.1) regarding drivers emerged. Figure 3.1 Spectrum between baseline (original) and desired drivers of priority setting Drivers are not static. They can change between baseline and desired practice over time and in different circumstances. Reactive to Proactive Priority setting practice pre-PBMA appeared to stem from reactionary responses to external stimuli such as political mandates and public pressure. Mandates and targets are imposed onto IH and trickle down the organization. One mechanism for operationalizing mandates is identification of specific targets, such as a percentage increase in home care services, in the 65 performance agreement IH has with the BC Ministry of Health. Participants stated an obligation to align with mandates: “We receive funding from the government and the government sets their priorities and then we kind of have to stay in alignment” (Pre-PBMA participant #2 27 ). These political mandates, together with limited dollars (especially targeted new funding) took precedence over desired drivers such as need or evidence in resource allocation decisions. Media attention and public expectations often focused on immediate health needs such as those attended to via acute and emergency care rather than the chronic care needs of the clients that community care serves. The media and public attention became drivers for changes in the organization and delivery of community care through demands to adjust community services to facilitate decongestion in overflowing acute and emergency wards. The manner in which community care accomplished this would often require community care to provide more of those services that facilitated discharges from the hospital. This moved them away from establishing and providing those services that best met their clients’ chronic care needs. Everybody knows that upstream chronic disease management is where we should be if we are going to do a decent job of looking after everybody, but the heck with that, deal with discharges out of acute because we’re drowning so all the resources get pointed at that rather than the upstream even though everybody knows that is where they want to go. But how do you get there when you’re all in the front pages of the newspapers and that is all they ever talk about. (Pre-PBMA Participant #4) Decision makers indicated a desire to be more proactive when setting priorities by focusing on local population needs or service delivery based on maximizing health outcomes. We’re relatively good at measuring outputs, that we can provide fairly detailed analysis of what work is being done and what it costs us, but we are totally missing the actual health outcome piece. So automatically if your system is privileging measuring health outputs, there is no incentive in that system to switch to a system that actually measures the effectiveness of those outputs. (Pre-PBMA FG participant) 27 Some participants were the same from pre-PBMA through year-1 and year-2; however, to ensure anonymity the participant numbering in this study is random, i.e., the same number is not assigned to the same person for each year. 66 Need, although a desired priority setting driver, was not straightforward as need appeared to have different meaning to different participants, with some emphasizing client need and others focusing on population need. Participants also questioned who was in the best position to identify need. Compare the perspective, “Major driving forces should be population based…what is truly the need, not what people think the need is. It should be up to us to determine what your need is and how we can best manage it” (Pre-PBMA Participant #17), with the idea, “Talk to our clients…to really see what it is that is needed and what is missing, and then try to make our programs fit their needs as opposed to trying to make their needs fit our programs” (Pre-PBMA Participant #2). Finally, participants indicated numerous problems with defining and using evidence in priority setting. There is a lack of evidence. I think we’d like to move away from opinion-based planning. But, right now, evidence is used in theoretical terms, not practical terms. Evidence is just not used, for various reasons – we don’t have it, don’t collect the right evidence. Outcomes evaluation – it is talked about, but not used. Rather those who scream the loudest; reacting vs. thinking through the decision-making process. That’s the culture in IH. (Pre-PBMA participant #9) Priority setting based on reactive responses to government mandates and media attention was an ingrained practice pre-PBMA. Participants expressed both an understanding of the need to attend to government mandates and a clear desire to also consider proactive drivers; i.e., participants wanted to reconcile the reactive and proactive priority setting drivers. The main proactive drivers described included focusing on community care client needs and health outcomes and adopting a more evidence-informed process when setting priorities for resource allocation decisions. This reactive-to-proactive driver dyad of priority setting is the first of three driver dyads that emerged during pre-PBMA data analysis. Past to Future Historical funding decisions are operationalized through departmental budgets and serve to maintain budget ‘silos’. Each portfolio, and each program area within it, receives funding based largely on the previous year’s budget. Revisions are sometimes made for political mandates or other context and time related factors; however, this is not built into the budgeting process as a formalized step. Decision makers declared this budgeting process unfair since many of the decisions that the budget is based on have not been recently 67 examined (e.g., how program areas and associated cost centers are set up) and the process does not contain a step to ‘level the playing field’. As a result, historical allocations perpetuate any inequities that have been created by previous decisions. Funding decisions have historically been independent of workload realities and rather have been based on the historical model of what envelope is available for this service; that is basically the money we have to work with, whether that means watering the service down to cover more people or whether it means creating a service gap…that has been less of a priority to resolve historically. (Pre-PBMA FG participant) Established targeted funding (i.e., funds that come from the Ministry with the directive of where/how they are to be spent), addressing deficits, or managing resource requirements (e.g., staff union contracts) were perceived to take precedence over issues related to client care. This unsettled many participants and supported their decision to try a more structured and evidence-informed priority setting process like PBMA. Everything boils down to dollars and cents for this organization at the present time. I mean right now we are very concerned about our budget for this fiscal year and we are all being asked to figure out budget strategies to bring our overrun in line. And so it continues…finance drives a lot of what we are doing instead of clinical practice, and best practice standards and client safety and care. (Pre-PBMA Participant #1) Participants argued that any priority setting activity should evolve from a clear organizational vision and that this was lacking: “There’s been a lack of visioning…how can we set these priorities and what we’re going to measure if we don’t even know where we’re going?” (Pre-PBMA Participant #19). The lack of a vision to guide priority setting not only perpetuated historical funding practice and decisions, it also supported the reactive responses discussed above. Participants indicated that a clearly defined organizational vision would enable them to reconcile the past and future drivers by helping them to move away from some of the historical funding decisions that were no longer serving client care effectively. They also stated that priority setting driven by a clear vision would support their desire for a more rational priority setting approach. 68 Ad hoc to Rational The lack of a formal priority setting process resulted in ad hoc decision making pre-PBMA. Sometimes this was based on new political mandates (termed ‘flavour of the day’ by participants) or directives from senior executives and sometimes on presumed urgent situations that arose and required immediate attention. There is just not a lot of planning and everything is ‘OK, we have to make the decision NOW, and it seems to me to be made without any real logical, real organized fashion and as [name] says today’s priority but next week we are on to something else before we have had a chance to really look at this priority. (Pre-PBMA FG Participant) The immediacy of the situation often left little time for any type of structured and evidence- informed process. We do have them [priority setting meetings] regularly, like once or twice a year, but I can’t say that we always do it in January or July, it is not a regular cycle, it is as the energy becomes frenetic around the issue, then you pull together a meeting and try to establish your priorities. (Pre-PBMA FG participant) Participants indicated that, at baseline, priority setting did not appear to be founded on any logic: it was arbitrary, unclear and difficult to navigate. Further, ‘one-time’ funding announcements from the government – where decision makers are given little time to develop proposals for new funding made available to support healthcare innovation – left little opportunity for evidence and standards to inform decisions in their pre-PBMA priority setting process. Our joke phrase for it is…‘fire-ready-aim’. And there is a lot of truth in that. When I look at some of the stuff…and say ‘OK, what are the standards, what is the evidence, how might that inform where we want to go as a program, how am I going to monitor it, etc’. A lot of what I am doing is going back to previous things that were done, and it was ‘Just do it, we’ll fix it later.’ (Pre-PBMA Participant #4) Participants desired an explicit priority setting process that could facilitate evidence use and that was adaptable to changing healthcare needs and context: “What it should entail is almost like a zero-based approach on the health benefit/cost benefit analysis that we can put in and it should be much more inclusive of upstream primary healthcare concerns” (Pre-PBMA FG Participant). 69 To summarize the pre-PBMA findings, participants expressed the following desired features of priority setting practice: a structured process, based on proactive goals, guided by a clear vision, and a rational, evidence-informed approach. Participants wanted support to reconcile their pre-PBMA priority setting practice – which they indicated was based on reactive responses, historical allocation decisions, and an ad hoc process – with their desired features. Based on these findings, PBMA appeared to be a suitable priority setting process to implement in this setting. 3.2.2 Post PBMA implementation – Acceptability Although PBMA aligns well with the above desired priority setting features, data gathered from participants after using PBMA to support priority setting revealed limits to the acceptability (another dimension of PBMA fit) of the approach in this setting. Sub-themes of acceptability included a lack of trust and time, and questionable scope and leadership commitment. Acceptability depends on both the PBMA tools and approach, and on the way in which these are used in the particular context. I think it is hard to evaluate the ranking process fairly given that whole context. I think the categories and everything were pretty tight; the scale was as tight as it could be…so I think, as a tool in itself, there is a lot of strength to it. It was, again, how we were using it and in this case, we kind of used it as a way to bog down the whole process. (Year-1 Participant #11) The way in which the tools and approach are used relates back to how trusting participants are of the process and leadership commitment to it, the time they have available to commit to it, and how accepting participants are of project parameters like scope. Trust, in both the PBMA process and leadership commitment to it, was a key factor affecting PBMA acceptability in this context. “People were really, really careful about what [they] put on the table. You may lose outside of this process. So trust was a huge issue” (Year-1 Participant #6). Participants felt unsafe in the current climate to contribute real and significant disinvestment proposals from their portfolios. There was a lack of trust in leadership (various levels) support of and commitment to PBMA. This was exacerbated by a lack of transparency and clear communication. There was questionable trust between our level, the director/administrator role and the senior team and I think it was lack of communication…So it was one against the 70 other. Yeah, communication was definitely something that needed to be worked on…I had a feeling that there was a lot more that could be put on the table. And part of that was mistrust externally or protection internally. (Year-1 Participant #10) Lack of time was another key contextual factor influencing PBMA acceptability. Participants found that PBMA required a lot of effort to understand and time to use and, in retrospect, they determined that they did not devote sufficient time to it. What maybe limited my true engagement as we moved along was more time issues, so competing demands…the ability to give it the due attention that it likely deserved…I think [adequate time] is so fundamentally key to how this rolls out; [we] need dedicated, established time. (Year-1 Participant #3) The PBMA project scope, determined at the outset by the community care director and other IH senior leaders, was questioned by participants throughout both implementations, in part due to an evolving community care portfolio, in part due to the strong ties between community and other health sectors, and in part due to IH’s complex organizational structure. Some services and their budgets spanned community, acute, and residential sectors. Participants indicated that they would have accepted the PBMA process more readily had they had more input into project scope selection and if the scope was clear and remained consistent throughout the project. “The organization changed so much…we had part of residential care before and then we thought maybe a portion of acute… and so it was confusing in terms of the scope as the organization changed” (Year-1 Participant #2). Participants’ degree of acceptance of and buy-in to PBMA to support priority setting changed throughout the study. This can be traced back to several factors including a mid-project change in leadership, questionable organizational commitment – “the lack of organizational commitment was so evident” (Year-1 Participant #9) – and shifting organizational priorities, all of which led to a derailing of the process in the second year. Because of the big switch in leadership, I think it just made things go off track. So some decisions that were made at the beginning were then overturned with the new leader and I think that was a real shame. So personally, my level (of buy-in) didn’t change. It was very interesting to watch and I could see some people got a bit frustrated. (Year-1 Participant #10) There was a period of time that I believed that it really was going to help us and that it was going to be used in an appropriate way and got past some trust issues and some 71 understanding with the group and so I would say [my buy-in] was medium-high at that time. And then as the process derailed with new players added and new realities coming forward and decision making happening outside of the process that didn’t necessarily fit, then I went back to being probably low or medium-low (Year-1 Participant #5) These concerns identified above – lack of trust and time, difficulty accepting the scope, and perception of limited leadership commitment – continued throughout PBMA implementation; therefore, the reactive approach continued to prevail: “Politics…trumped the process” (Year-1 Participant #7). Results described thus-far indicate that, pre-PBMA, participants desired a priority setting process that was in line with the structured, evidence-informed process supported by PBMA. However, after using PBMA to inform priority setting, contextual factors limited participants’ acceptability of and buy-in to the process. To address this, at the end of the first cycle using PBMA, we re-confirmed the project scope and aim with the new community care director and the commitment was made to continue with PBMA and use it to guide priority setting for a second cycle in community care. However, the director eventually decided to discontinue PBMA at stage five (identify options for service growth and resource release) of the second cycle. Even if attending to the above factors – lack of trust and time, and questionable scope and leadership commitment – improved the degree of PBMA acceptance by these decision makers, using PBMA in practice revealed another concern with its fit – that of its usability in this context. 3.2.3 Post PBMA implementation – Usability This final section of the results focuses on whether PBMA helped participants by providing a priority setting process and tools that supported them in establishing and making resource allocation decisions; i.e., how usable PBMA was in this context. Usability was another dimension of fit (in addition to desirability and acceptability) that emerged during data analysis. Sub-themes of usability were categorized as either challenges with PBMA (participant perception of PBMA as an academic rather than practical tool) or challenges with the context (time pressures including participant lack of time and the timing of PBMA 72 implementation; difficulty using evidence; lack of a clear vision; and lack of contextual readiness and capacity). Overall, participants deemed the PBMA tools and approach usable, at least in a de- contextualized or academic sense; however, challenges with both the context and PBMA limited PBMA usability in this study. Regarding the context: I see culture being our biggest obstacle in making PBMA work here. It is the information culture that we talked about used as a weapon or a currency, the whole decision making culture here, the fiscal culture of money rules – cause that is not what PBMA is saying – money doesn’t rule in PBMA; in IH money rules. (Year-1 Participant #5) And regarding PBMA, participants perceived of PBMA as an academic tool, founded in health economic principles and supported by research, rather than a readily adoptable and practical one: “What it turned out to be was a more academic exercise than a practical exercise in terms of its utility in the operations world” (Year-1 Participant #5); “Although the process makes sense, when it comes to applying it there are roadblocks” (Year-1 Participant #8). Participants indicated several specific organizational challenges to using PBMA in this context, including time pressures, difficulty using evidence, and lack of a clear vision and contextual readiness and capacity. Just as a lack of time was one factor that limited PBMA acceptability in this study, time pressures, including both a lack of time and the timing of the exercise (i.e., the need to meet specific timely budget requirements), limited PBMA usability also. I think what was difficult and challenging, because we were under time pressure too, was the different tools that would be used and the relevance of those tools in terms of the context of what we were facing in terms of the pressures and what was relevant. Some of it felt a little bit artificial, you know, given the other struggles that we were having within the program at that time – mainly it was the budget. So the ranking and the tools also need to fit within the context of what we are doing in the real day-to- day world. (Year-1 Participant #2) Key concerns identified pre-PBMA, such as lack of a clear vision and difficulty using evidence in priority setting, did not abate after the introduction of PBMA. These also affected PBMA usability in this setting. 73 Nobody has a really good understanding of what Community Care is from a visionary point of view where you get the key elements about what Community Care is. It feels at the line-level that they are just scattered all over the place, chasing here and chasing there. (Year-2 Participant #9) We get caught in the conundrum of wanting information and not having it, and having it and not trusting it. And then having it and not using it – what we do have… So that was another hurdle for PBMA that made it hard to ground itself. (Year-1 Participant #5) These results make it clear that simply introducing PBMA does not necessarily change the drivers of priority setting to be more in alignment with the desired drivers expressed by participants. Specific actions and contextual factors are required to evolve priority setting practice to become guided by a clear vision, to be more proactive, structured and evidence- informed. As such, participants indicated that PBMA’s usability would be enhanced by more robustly attending to the interplay between context and PBMA, starting with developing and ensuring contextual readiness and capacity, specifically in relation to vision-guided, evidence-informed, and rational priority setting, before implementing PBMA. Beef up the evidence base, beef up the time, beef up the capacity and permission that people have to engage in debate and challenge, and possibly have a discussion earlier on, on what the outcome of this decision making process is going to be. (Year-1 Participant #7) Ensuring contextual readiness and capacity would include providing context-relevant and timely (e.g., before and during implementation) PBMA education to participants, emphasizing clarity 28 and communication, to address participant and, thereby, organizational capacity. The recent re-orientation [provided at the start of year-2] that you gave to everybody was good. It was like a support group – you get a little shot of ‘ok that’s why we should do it’. It is a better way to do it. It is just like repeat education that locks in again at a different level. (Year-1 Participant #7) It would also include ensuring that local data and evidence were available and that participants were aware of it and found the format of the data was usable for their purposes, and that organizational research support was available to inform the process. 28 The term ‘clarity’ as used in this study is defined in Chapter 4 – Implementation, where it is discussed as a key theme. 74 It seems to me they knew at the very beginning that they had no evidence. Then what they had, they couldn’t seem to access. So if we know that evidence is key to making rational decisions that go along with your intuitive understanding of knowledge, then you have got to have that evidence in place or you’ve got to have an understanding of what you’ve got. So maybe at the time these key problems appear, maybe there needs to be then a process to get that in place before you move ahead with the other. (Year- 2 Participant #9) A contextual readiness and capacity assessment stage would provide an opportunity to assess for factors required to use PBMA, and then take actions to address these to facilitate PBMA implementation. For example, this would provide the opportunity to ensure that staff are sufficiently informed and engaged, adequate time and resources are allocated to the work, and higher level organizational support is strong and clear. These were all limitations to PBMA usability in this study. The required actions that stem from the assessment could be taken by researchers (e.g., staff education) and/or the organization (e.g., develop participant and organizational capacity), depending on the specific actions required to support PBMA adoption. In this study, participants recommended several organization-specific actions: “There has to be some acknowledgement that the process is valuable and time allotted for it. And then the assurance that disinvestments lead to opportunities for re-investment within the service as opposed to the fear of losing it altogether” (Year-2 Participant #1). [There] has to be an ‘above-us’ kind of communication thing and a true commitment in order to be able to influence that and not have mixed messages that “This is very important but you also have to do this, this and this at the same time”. And I think we will act more in response to those expectations if they are clear and linked into other deliverables, other performance management, other objectives. (Year-1 Participant #3) Participants also recommended researcher-specific actions. For example: Figure out the politics before you come in. And I think before there is kind of an approval process, meet with the whole team. So insist that you don’t just meet with who you perceive to be the leader because the leader doesn’t always have the capacity to do what you guys want to do. (Year-1 Participant #4) To summarize, these results – exploring PBMA fit via dimensions of desirability, acceptability and usability – indicate that a rational, evidence-informed approach guided by a clear vision remained elusive even after working with PBMA for two cycles: “The tool didn’t quite deliver on that marginal analysis, so the default was back to the old politics” 75 (Year-1 Participant #11); and, after the second cycle, “we are still heavily making decisions on deficit-based thinking” (Year-2 Participant #7). Participants expressed concerns with operationalizing their desired drivers; i.e., basing priority setting on proactive goals, guided by a clear vision and using a rational and evidence-informed process. The baseline drivers – reactive, historical and ad hoc – were perceived as being so engrained in practice that the achievability of the desired drivers – proactive, future-oriented and rational – was questioned. Overall, participants indicated that achieving priority setting based on the desired drivers depends on contextual readiness and capacity, leadership commitment, and stakeholder acceptance and buy-in. They argued that initial PBMA goals should include building trust, ensuring high-level commitment, and increasing organizational capacity to move towards more rational and evidence-informed decision making in general. 3.3 Fit discussion Based on the pre- and post-PBMA implementation results, PBMA fit was found to revolve around the way in which it fulfilled decision makers’ desired features of priority setting – which include a process based on proactive goals and guided by a clear vision, and that uses a rational, evidence-informed approach – and how acceptable and usable PBMA was deemed to be. To determine PBMA’s fit in this context, the baseline and desired drivers of priority setting practice, and how PBMA relates to these, is discussed. Then, the discussion turns to PBMA acceptability and usability as these relate to limitations in adopting the rational approach facilitated by PBMA. 3.3.1 Pre-PBMA: Baseline versus desired drivers of priority setting Participants felt constrained by the drivers of baseline priority setting activities over which they had little control, such as government mandates and the routinized practice of basing future resource allocations on historical allocation decisions. Past decisions, once made, lock the system into evolving in certain ways. For example, union contracts commit dollars and also determine clinician scope of practice in a way that also restricts budget flexibility. Other significant drivers – public/media scrutiny (reactive) and lack of a process (ad hoc) – took decision makers further away from what they desired in priority setting practice. The ad 76 hoc process enabled reactivity and historical allocation patterns to assume unwarranted significance. The participants’ main desired drivers – a clear vision, explicit model, and proactive process – move decision making towards a more rational approach, such as that facilitated by PBMA. The importance of having a clear organizational vision drive priority setting activity is well- founded in the literature (e.g., Dionne et al., 2008; Mitton & Prout, 2004). Lack of a clear vision renders a rational and analytic approach to decision making difficult to achieve (Baker et al., 2004). Here, we found priority setting pre-PBMA was not driven by a clear vision. The lack of a clear vision in IH and, specifically, within the Central Okanagan community care team was deemed to be one factor contributing to competition between priorities (e.g., acute vs. preventive services; deficit over-rules care). Participants stated that a clear vision would serve to reduce the influence of public scrutiny and media attention on priority setting, and that a formal process stemming from the vision would legitimize resistance to political or media-driven demands. Participants indicated that PBMA was a potential approach to support priority setting guided by the organizational vision and one that facilitated an evidence-informed and structured process. 3.3.2 Post PBMA implementation: Did PBMA deliver? Participants’ reflections on their baseline and desired priority setting practice built support for implementing PBMA in the community care context as, at least ostensibly, it fulfilled the desired features: it appeared to be a good fit at this level of analysis. PBMA introduces structure, a mechanism to incorporate various forms of evidence into the process and to compare various options, and the ability to align the process to any desired (clear) goals. In other words, it introduces a more rational approach to decision making, which has the desirable effect of reducing the roles of politics and ad hoc judgment that were prevalent at baseline. PBMA is a more structured approach to priority setting than historical allocation. It is as evidence-informed as time and data allow. However, despite PBMA appearing to be a good ‘fit’ regarding desirability prior to implementing it, after using PBMA to inform two budget cycles, PBMA’s usability and acceptability in this context were questioned. 77 Several suggestions were made to enhance PBMA’s fit in this context. These recommendations focused on the following areas: contextual readiness and capacity (including a conducive culture with a guiding vision and adequate resources devoted to the activity), leadership commitment, participant acceptance and buy-in, and a project scope deemed acceptable by participants. Previous PBMA work has identified the importance of strong leadership and a conducive culture (Mitton & Donaldson, 2003a), and scope selection is a key step in the PBMA process; however, assessment of contextual readiness and capacity has not previously been a formalized step in PBMA. Decision makers in this study argued that this assessment would serve to garner leadership support and address commitment, capacity and other contextual issues before starting, and also during, PBMA. A change management or diffusion of innovation model (e.g., Greenhalgh, Robert, Bate, Macfarlane, & Kyriakidou, 2005) 29 could be used to guide the contextual readiness and capacity assessment to ensure key aspects important to facilitating adoption of an innovation like PBMA are attended to. This readiness assessment would also be an ideal time to address other factors (see below) that limit adopting a more rational approach to priority setting and decision making in general. This may be of interest to organizations struggling with adopting PBMA and other evidence-informed practices. 3.3.3 Restrictions to adopting a rational approach Although PBMA has worked in a variety of contexts (Mitton & Donaldson, 2001; Tsourapas & Frew, 2011), in some settings, such as this one, there are restrictions to accepting and/or using, and thereby adopting, such a rational approach to decision making. Rational decision making requires clear goals to lead the process and thorough consideration of all the alternatives (Baker et al., 2004). As this group experienced, clear goals and means to attain them are not always available nor do they remain consistent for extended periods of time. This requires decision makers to be adaptable and use, at least to some degree or at some times, other, less goal-oriented, more emergent strategies as they move forward. This may include more dependence on professional experience and expertise, as is common in intuitive or judgment-based decision making (Klein, 1999). 29 See Appendix B for details of this model. 78 Availability of sufficient information to inform all alternatives, and time to compare them, may also be problematic. Participants had several suggestions for how the organization could move forward in this area. Although financial information is readily available, measures of benefit may be difficult to determine and obtain when working with the chronic care needs of the (often disparate) community care population. Aside from this difficulty, it remains questionable whether a rational approach based on a comparative and analytical decision strategy is the most suitable strategy for this type of decision making. Decision making conditions inherent in this context (e.g., time pressures, dynamic conditions, ill-defined goals, greater dependence on decision maker expertise than evidence) may be more conducive to a decision strategy based on singular (rather than comparative) evaluation and experience-based judgment (rather than formal evidence and rationality) (Klein, 1999), or a combination of different types of decision strategies at different times or as decision making evolves towards a final (for the time-being) decision. …organizations will differ in the extent to which they are able to follow formal planning models as opposed to a more informal strategic improvisation approach. The latter approach is more intuitive and therefore makes less use of formal analysis…but it may be more 'rational' because it matches the conditions of decision-making in the real world. (Thomas, 2004, p. 20). Consequently, an organization can accept that in certain circumstances a rational approach may not be suitable but that when it is desired, a more incremental approach to increasing the rationality of decision making may be needed (Baker et al., 2004; Lindblom, 1959). This approach may require several intermediary steps to be addressed prior to or while implementing a rational approach like PBMA. This is discussed above as a contextual readiness and capacity assessment and may include establishing clear goals and taking steps, both operational (e.g., evidence available in a usable format) and organizational (e.g., power, mandate and time available for rational decision making), to move towards more evidence- informed activities. If an organization can be placed on the ‘original – desired practice spectrum’ (Figure 3.1), targeted strategies can be developed to actively evolve practice closer to the desired approach. Also, studying the specific factors that contribute to successful PBMA use in some settings may illuminate contextual facilitators that can potentially be capitalized on with targeted strategies. 79 Finally, the complex organizational structures which characterize healthcare systems require consideration of complexity theory models. Attending to the complex relationships inherent in large organizations may illuminate potential strategies to facilitate collaborations and other mechanisms that may support more innovative thinking regarding resource allocation options, and also about when and how rational approaches to decision making might fit in the context (Westley, Zimmerman, & Patton, 2007). Complexity theory driven approaches may be more suitable for team decision making, which PBMA facilitates, because of its attendance to relationships and to the complex systems at play in team decision making (Bullen & Sacks, 2003). Although PBMA appeared to fit when compared to participants’ desired priority setting attributes, when compared to the actual baseline practice and results from using PBMA in this setting 30 and elsewhere (e.g., Halma et al., 2004; Miller & Vale, 2001; Mitton & Donaldson, 2003b), perhaps PBMA, as currently operationalized, is not a ready fit in some contexts or situations or at certain times because of its rational approach. For example, the timing of the leadership change during this study, and buy-in by the new leader and some participants, may have been so misaligned with rational and collaborative priority setting approaches that implementing such an approach all in one step may have been too significant a change in practice. It may also be that different times and situations warrant different decision making styles; this requires further investigation. Greater effort to contextualize the approach (e.g., ready the organization, adapt the PBMA tools) and identify when it is best used, and efforts to reconcile the tension between the current more intuitive and ad hoc approach and the desired rational approach, may result in greater acceptability and usability, and thereby adoptability. “The environment exerts a powerful set of influences on practitioners, policy-makers and even researchers. Factors that should be considered include those of a structural nature such as the decision making structure; rules, regulations, and policies” (Graham & Logan, 2004, p. 97), and the organizational decision making style; i.e., how rational versus intuitive and political it is in various situations. 30 The impact results are discussed in the impact chapter (chapter 5) below. 80 3.4 Summary of fit The purpose of this chapter was to determine whether PBMA ‘fits’ within this IH community care context based on decision maker perceptions of baseline priority setting practice vis-à- vis desired practice, and to document their perceptions of PBMA usability and acceptability after using it. The answer is not straightforward. Tensions exist between desired and actual priority setting practice, and between rational and intuitive/judgment-based decision making approaches. Research to date, including these results, has provided clear evidence of decision makers’ expressed desire to move towards more rational decision making approaches; however, whether rational approaches are sufficiently compatible with actual practice to be consistently and successfully adopted in various healthcare contexts and situations remains to be seen. In other words, even when decision makers state that they want a structured, explicit approach to priority setting, is this too big a change from actual practice in some contexts, too much effort to achieve, and/or unwarranted in some situations? Since historical allocation and reactive and ad hoc approaches are such engrained practices, perhaps they have a role to play, at least in some situations, thereby interfering with sweeping change to practice. The answers to these questions should be of interest to both decision makers and researchers interested in priority setting and resource allocation, and also to those interested in adopting other more evidence-informed practices in general. The findings from this study indicate that adding a contextual readiness and capacity assessment stage (and then readying the individuals and context based on the results of the assessment), recognizing organizational complexity (at a minimum when selecting scope and developing reallocation proposals), and reconciling PBMA’s rational approach with prevailing decision making processes through incremental implementation may help to improve PBMA’s effectiveness in some contexts. 81 4 Chapter: Implementation – Results and Discussion 4.1 Synopsis of chapter Since “healthcare organizations are the most complex form of human organization…to manage” (Golden, 2006, p. 10), it is not surprising that change in healthcare is difficult to achieve (Graham & Tetroe, 2007; Grol et al., 2007). The desired end-state “can be a nebulous concept without sufficient specificity to be actionable” (Golden, 2006, p. 11). Even when the desired end-state is clearly envisioned, getting there can be a journey plagued by difficulty (Golden, 2006). This is why the implementation process warrants as much attention as the innovation (in this case, PBMA) being implemented. As noted in the literature review, PBMA has been successfully used many times (e.g., Bohmer et al., 2001; Mitton & Donaldson, 2003a; Mitton et al., 2003; Scott et al., 1998); however, questions about how best to implement PBMA still exist. Implementation-relevant issues identified in the literature include difficulty obtaining adequate information for the program budget (Craig et al., 1995; Twaddle & Walker, 1995), difficulty with disinvestment (Ruta et al., 2005), limited monetary impact (Urquhart et al., 2008), difficult-to-measure outcomes (Mitton & Donaldson, 2003a), organizational barriers to adoption (Mitton & Donaldson, 2003a), and questions of long term sustainability (Mitton & Donaldson, 2001). Implementation has bearing on how well the decision maker users understand the concepts that underlie PBMA and can successfully apply them to their situation. Therefore, understanding the role of implementation in changing priority setting practice is an important step to understanding how to resolve issues with PBMA, such as those identified above. PBMA requires a change in usual priority setting practice, which, as with any change, can be difficult to achieve. The purpose of this chapter is to describe and evaluate the process of, and describe the experience and lessons learned, implementing PBMA in one Interior Health Authority community care portfolio (thereby addressing thesis research objective #2, Section 1.4). Highlighting potential implementation issues and responses to these should be of interest to 82 decision makers who wish to more effectively employ PBMA and other evidence-informed practices. 4.2 Implementation results The implementation results are presented in two sections below. The first section, based primarily on the thematic analysis of the field notes and meeting documents, summarizes the key findings from the PBMA implementation evaluation organized by PBMA stage 1 through 7 (as per Table 2.3 in the methodology chapter above). This addresses the first part of the second research objective in this thesis (Section 1.4) – to describe and evaluate the PBMA implementation process. The second section of results presents the two key qualitative themes that emerged from the post year-1 and year-2 interview transcript analysis: first, participants’ desire for more clarity31 about their roles in and the desired outcome of using PBMA, especially after the change in community care director; and, second, participants’ desired PBMA implementation and approach adaptations for this context. The second section of the results addresses the second part of the second research objective in this thesis – to describe the experience and lessons learned implementing PBMA. 4.2.1 PBMA implementation evaluation findings, by PBMA stage Stage 1 – aim and scope Participants questioned the project scope several times at different stages throughout the study. Although the scope was clarified with attendees at the first advisory panel meeting to include only those programs funded through the community care director’s budget and therefore under her control, questions about the scope emerged at the first meeting. The issue of scope and whether or not it should include relevant acute and residential programs emerged as a concern by the second meeting. It was a reoccurring theme in subsequent meetings, even at the final decision retreat meeting (where proposals were rated and ranked – see stage 6 below). The scope issue was highlighted in the first meeting when participants reviewed those programs with budgets and services that spanned across the continuum; i.e., those programs that included community, residential and acute services. This included social work and dietitian programs. The scope issue was again highlighted when some participants, 31 The term ‘clarity’ as used in this study is defined below in the second section of the results. 83 notably managers of residential and acute services that were closely linked to community care and therefore invited to participate, chose to opt-out of participating in PBMA when the meetings started to focus on resource release discussions. Participants’ primary concerns with the scope were two-fold. First, they wondered if reallocating funds within the community care portfolio was as impactful as reallocating funds between the community, acute, and residential sectors. Second, they were concerned that if community care found resources via disinvestment while other sectors (acute, residential) who were not participating in PBMA did not, then community care would take a financial hit with future targeted funding cuts based on the budget. Participants also struggled with questions about community care boundaries, like where community care starts and stops, and which services should be included in PBMA. Stage 2 – program budget Participants had difficulty conceptualizing a program budget, despite available information. To support development of the program budget, participants agreed to write clear descriptions of their programs to share with the advisory panel and use to build a potential format for the PBMA program budget. However, this was never done despite the IH project manager bringing the item forward for two subsequent meetings (#1 and #2). Instead, the participants based the PBMA program budget on only their usual budget cost centres (e.g., the budget line for each individual community care program included in the PBMA project scope); yet, they were concerned that this approach reinforced ‘silo’d’ thinking regarding service delivery. They also critiqued the cost centres for their historical origins which were deemed arbitrary and thought to perpetuate inequities between service areas. The original plan was for the advisory panel to use the program descriptions to review service patterns for each program area within the project scope to determine if the budget could be conceptualized with a focus on factors other than department cost centres, such as diagnosis, services provided, types of clients served, or types of care required. However, this was not done largely due to lack of time and perceived lack of relevant, easily obtainable data. Therefore, the program budget consisted of simply each program’s name and budget amount. 84 Stage 3 – advisory panel As mentioned in Table 2.3 (Methodology chapter), the advisory panel included representation (program managers) from all programs included in the project scope, the director, a community representative, community care business support, and a consulting physician. Advisory panel meetings were often attended, in a supportive rather than participatory role, by one or two (of three) IH executive-level decision makers who were part of the research team. The advisory panel community representative, recruited to contribute lay-person input to priority setting for publicly-funded programs, was deemed valuable and a successful inclusion. She was well-accepted as a team member and, through her client contact, a suitable person to represent the community care client perspective. Participation in advisory panel meetings by research team members who were also IH executive-level decision makers proved valuable in keeping the broader IH mandate in mind. However, a change in community care leader part-way through year-1 of PBMA implementation led to confusion as to who was in charge (in part because both the previous and new director attended the last four PBMA meetings, including the decision retreat, in the first year) and uncertainty whether and how the process would continue. Because the new director joined community care part-way through the first PBMA cycle, she did not participate in key decisions made early on including decisions about project scope, aim, and rules of engagement that participants had discussed (however never explicitly committed to) in the first few meetings. It also required a repeat of the education provided to the participants prior to implementing PBMA. A meeting was held between the new director and two research team members before meeting number five to discuss the PBMA project details and relevant education with the new director; however, she still brought forward several questions related to project scope and aim during the meetings she attended. Stage 4 – Criteria The criteria 32 developed were consistent with those found in the literature (e.g., maximizing benefits, improving access and equity, reducing waiting times); however, their relevance and validity were questioned several times. The criteria were eventually explicitly linked to the strategic plan which helped to promote their validity to the advisory panel. Some participants 32 See figure D.1 in Appendix D for details. 85 questioned if the chosen criteria, and specifically how they were defined and incorporated into the scoring tool 33 , could effectively be used to rate the investment and disinvestment proposals. After multiple discussions about the criteria and how they could be used in PBMA, participants eventually found another use for the criteria – to guide and assess day- to-day decisions. They accomplished this by incorporating the criteria into a one-page decision guide they could each use as a check-list during non-PBMA related strategic decisions. Stage 5 – Identify options for investment and disinvestment The advisory panel decided that all program managers were responsible for developing at least one resource release and one growth proposal. Although the panel originally discussed a twelve percent budget reduction target for the resource release proposals, this was deemed too difficult to achieve. Furthermore, despite discussing smaller targets for the smaller or even all programs, the community care director decided against targets altogether. Deficits and funding discrepancies between services were also complicating factors. Some participants deemed it unfair that they should contribute funding from their program budgets to other programs experiencing a deficit. Some participants argued that funding inequities between program budgets that have existed for years would only be perpetuated unless a zero-based approach was used; i.e., each program should start with no funding and submit proposals for all services for which they required funds. Furthermore, the communication strategy discussed and agreed-to by the director and advisory panel early in the process was not adhered to which lead to uncertainty, lack of buy-in, and limited proposal development. These effects were exacerbated by the mid-project change in community care leader. Using a broad definition of what evidence comprises, varied forms of evidence were used in the investment/disinvestment proposals. Expert opinion and cost data were most common forms of evidence or data used; link to the vision and evidence of benefit were absent. Although the process did encourage participant collaboration in exploring new ways to provide service, two participants stated that they held back on disinvestment proposals for 33 See Figure D.2 in Appendix D for details. 86 fear of losing the money – they planned instead to use these ideas to make reallocations within their own programs outside of the PBMA process. Stage 6 – Evaluate options Participants struggled in year-1 to employ the new criteria and score-sheet. Some (though not all) found it difficult to rate the proposals against the criteria using objective measures (e.g., percentage change), preferring a most-to-least scale. They indicated that a practice-run using the score-sheet would have been helpful. The advisory panel had difficulty deciding whether initiatives already in process to address deficits or budget overruns – some being developed and implemented shortly after PBMA started – could be counted as disinvestment proposals. Although the ideas behind some of these initiatives may have emerged or been further developed through the PBMA process, and researchers supported the inclusion of these initiatives as disinvestment proposals, some advisory panel members indicated that inclusion of these initiatives would prevent other potential disinvestment proposals from being generated. So the decision was made by the group to not include these initiatives. Participants found some proposals were easier to ‘sell’ through this process – in particular those with a solid, readily-available and applicable evidence-base – giving these a (presumably) unfair advantage. Stage 7 – Validation and resource reallocation Participant uncertainty with the scope and how proposals could or should link to the strategic plan hindered the process at the point of validation in year-1. The new leader was uncertain about how results from the PBMA process aligned with her mandate as the new community care director. Participants were unsure whether the proposals aligned with the evolving strategic plan (new directions in primary care and chronic disease management were being discussed); so some were uncomfortable with using the ranked list 34 generated through the proposal evaluation process (stage 6) as a starting point for resource reallocation decisions. The advisory panel ultimately determined that the proposals needed more work before they could be used to reallocate resources. 34 See Appendix E for details. 87 Summary of PBMA implementation evaluation Key barriers to PBMA implementation in this setting revolved around participant lack of clarity of, and commitment to, the project scope, aim, roles, and responsibilities, and uncertainty about organizational support of and commitment to the process. These issues were exacerbated by the change in community care director. Key facilitators of PBMA implementation in this study included the successful inclusion of a community representative on the advisory panel, new collaborations between advisory panel members to develop proposals for service redesign, and acceptance of the criteria for use in strategic planning decisions outside of PBMA. 4.2.2 Qualitative themes Two overarching qualitative themes emerged during analysis of the interview transcripts. The first was a desire for more clarity and, the second, a desire for PBMA approach and implementation adaptations. These adaptations included improving PBMA flexibility by making each phase functionally independent of the others and adding a contextual readiness and capacity assessment stage. These adaptations were primarily offered up as suggestions for how PBMA could be adapted to both better fit this context for the second year of using PBMA and to be implemented elsewhere in the organization, as was the original plan (see stage 1, Table 2.3 – Methodology chapter). Desire for more clarity Participant lack of clarity was consistently expressed throughout the two sets of interviews, one set after each PBMA implementation (the first implementation with one and then both directors, and the second with only the new director). In this study, desire for more clarity encompassed a desire for clarification and increased understanding, confidence, certainty and trust; with these latter sub-themes also being anticipated outcomes of improved clarity. The clarity theme also includes participants questioning aspects of PBMA implementation and organizational commitment to the process. Thus, the clarity theme includes issues of understanding, legitimacy, trust, respect and inclusion. 88 Particularly after the change in community care director, participants desired greater clarity around several aspects of PBMA implementation: what should their role(s) and responsibilities be in the process (including who ultimately owns the process), and how should PBMA and its tools be used to achieve PBMA’s potential benefits in this setting. Participants also provided suggestions for how clarity might be improved. Participants questioned their responsibilities in and ownership of the process and their role within it, stating that roles and responsibilities were not adequately discussed and clarified, especially after the change in community care director. “Is this truly ours to work through or is this owned by our senior executive because they are setting some of the key parameters [e.g., project scope] around what we are looking at” (Year-1 Participant #335). Despite confirmation of project aim and scope with both the initial and new director, and researcher and project manager support, meetings often got sidetracked by tangential discussions. Participants, in retrospect, appeared to be unsure of what was expected of them at times. “If I look at our experience and our processes, what would have made it better is the leadership, and how it was introduced…what were the expectations of those participating? So leadership, some clarity around roles” (Year-2 Participant #6). [We needed] clear direction. Partly because of that mistrust and not knowing where the line is and what we can do and what we can’t do, and what is ours to play with, what isn’t? So it’s the communication piece. There needs to be a clear direction given to [us] on what this means and what it can mean and what [we] have ultimate authority over. (Year-1 Participant #10) Although researchers confirmed that PBMA had successfully been used in a community care context before, participants questioned whether the project scope contained the relevant services for feasible and impactful reallocation. This lack of confidence in the scope reduced participant buy-in: “I think what was bad about it or is still a challenge is the lack of other sector involvement…applying PBMA in silos, instead of applying PBMA with that overarching collaboration across the sectors” (Year-1 Participant #12). Participants indicated 35 Some participants were the same through year-1 and year-2; however, the participant numbering here is random – the same number is not necessarily assigned to the same person for each year. 89 that a clear and firm scope and aim may have helped stabilize the process through the leader change. In terms of PBMA benefits, participants were clear that the process was not self-justifying: “[PBMA] is good but only if it can be done in a way that it supports what we want it to achieve and it is not a project just in itself” (Year-1 Participant #2). Participants argued that PBMA’s full range of benefits – e.g., opportunity for innovative thinking; a way to generate ideas; a mechanism to help decision makers contextualize and embed economic principles and clinical, managerial and other knowledge and evidence into the decision making process – were not highlighted sufficiently. I think we got kind of stuck on just moving the resources, and didn’t think about what other things can we take out of this and maybe we can’t move the resources around but is there an opportunity for us to look at the data, look at the evidence and turn it into information? (Year-1 Participant #6) Despite researchers providing a full day of formal education prior to PBMA and informal education during PBMA implementation, participants were not clear on how the tools should be used or how the whole approach was supposed to come together. For example, it was not until after PBMA was implemented that participants realized they did not understand the key role that the decision making criteria (stage 4, Table 2.3) play in the PBMA process and that evaluating the proposals using the score-sheet serves to develop a ranked list (stage 5, Table 2.3) which is then a launching point for discussion and not the final list of reallocations to be made. Decision makers indicated that clarity could be improved through both communication and education plans, making specific suggestions for each. They stated that “[Better] communication – upfront” (Year-1 Participant #6) within community care and between community care and the broader organization would have enhanced clarity. “[We needed] documentation in the very beginning of what decisions were made in terms of the scope, the project and the reasons for it. Because decisions were made but they weren’t documented, and it made it really hard for people who came along later” (Year-1 Participant #2). They stated that adequate communication was required both within the group participating in 90 PBMA and with other organizational levels and departments that participants interact with, and that communication was lacking at both in this exercise. One of our principles was if you can get others to work with you around some of your proposal development, then by all means. So you don’t have to stay within your Community Care program…get the other stakeholders on board so if they at least understood what it was we were going through, then we could work with them. (Year-1 Participant #3) Despite the education provided by researchers, participants indicated that education should be provided both pre-PBMA and throughout implementation so that questions and concerns could be discussed and addressed in real-time. “[There] was not enough time necessarily up- front for clarification, dialogue around the parameter setting…we only had that one day as kind of an introduction without an opportunity to digest and dialogue” (Year-1 Participant #3). Participants requested more education on how to contextualize the education they did receive, e.g., “what is evidence, how do we use it” (Year-1 Participant #6). Adaptability of implementation and approach In addition to refining the education and communication plans as discussed above, participants articulated a few adaptations to the PBMA process to facilitate adoption of PBMA in this context. Suggestions were made after year-1, with the plan to use these suggestions to adapt the process for year-2. However, with the change in community care director towards the end of year-1 and other contextual changes that occurred around the same time, PBMA was discontinued at stage five (identify options for resource release or service growth) in year-2 before recommended revisions could be fully applied in this context. Despite this, in the second set of interviews participants continued to suggest, often in more detail, PBMA adaptations to facilitate PBMA adoption. The two main suggested adaptations included making PBMA more adaptable through the use of stand-alone PBMA phases and the addition of a contextual readiness and capacity assessment phase. Participants argued that PBMA should be implemented in a fashion that enabled each of the seven stages to be useful independent of the others. They indicated that this would enable PBMA to be more adaptable to the current culture and activities, rather than trying to make these fit PBMA. 91 I am not sure [PBMA] is the only tool that we would want to use, but I think it definitely is one of the tools we would want to use. And whether again we use all of it or pieces of it, I do believe there are things that we need to have involved. (Year-2 Participant #6) Participants stated that Year-1 should in part be about building trust with each other and the process, rather than be primarily about generating disinvestment proposals, and that this would help to clarify roles and responsibilities in the process. I am not sure once you are in the process there really is room for discussion about trust and power and control. You can acknowledge that yes those are issues, but…you can’t pull out or not participate authentically because of issues around trust, power or control. You had better speak it up and then let it go. (Year-1 Participant #7) They also indicated that a preliminary stage should be built into PBMA to facilitate an organizational readiness and capacity assessment. This would constitute a new stage in PBMA, and should help ensure that the necessary resources and technical and human factors are available to support such a collaborative and evidence-informed decision making approach. [Do] a SWOT [Strengths, Weaknesses, Opportunities, Threats] type of analysis at the outset; or more like an industry analysis so that we can really assess where we are at. So instead of jumping right into programmatic decision making look at the services, outlay the context of the environment first. So what are our strategic influences, what are our external control factors that we need to consider to allow ourselves a better frame of reference for what we can set out to do and what we should set out to do. And then start to look at the specifics of our programs or services relative to criteria…because there is a lot external to us, so really identifying what that is, and what within that do we have control over. As well as capacity, because capacity is a huge issue right now…Before we say that this would be a great thing to do…you have to first see is it feasible within the current context. I don’t think we ever really did that. (Year-2 Participant #3) If we know that evidence is key to making rational decisions that go along with your intuitive understanding of knowledge, then you have to have that evidence in place or you’ve got to have an understanding of what you’ve got…maybe there needs to be a process to get that in place before you move ahead. (Year-2 Participant #9) Finally, participants wanted PBMA to be combined with other business initiatives, so it would not seem like a one-off exercise done in addition to regular budgeting and planning work: “I would change [PBMA] to adapt or combine [it] with other initiatives…PBMA in 92 isolation doesn’t work” (Year-2 Participant #2). “I don’t think the benefits were fully realized because it…was a one-off. It was too constricted. So, elements that were required to be part of the process weren’t there” (Year-2 Participant #4). The key themes emerging from the analysis of the interview transcripts reinforce the key findings from the implementation evaluation. Lack of clarity of and commitment to the project scope, aim, roles and responsibilities, and uncertainty about organizational support of and commitment to the process were barriers in the evaluation that also emerged as themes in the interview transcript analysis. The two key qualitative themes discussed above are also closely connected to each other. Participant desire for PBMA approach and implementation adaptations, including making each phase functionally independent of the others and adding a contextual readiness and capacity assessment phase, were in part requested to enhance participant clarity. For example, clarity can be enhanced during a contextual readiness and capacity assessment when key questions about organizational and individual requirements to implement PBMA are asked and answered in the group. 4.3 Implementation discussion Despite thirty years of testing and application, PBMA remains difficult to implement and is not always successful in some contexts. This is one case example – this analysis qualitatively explores some of the challenges which healthcare decision makers in a particular organization encountered in their efforts to implement PBMA. It emphasized for us that implementation is “always a combination of deliberate and unplanned processes” (Patton, 2011, p. 48) and that those implementing PBMA should be mindful of this as they proceed. The two main practice recommendations stemming from these findings include: the addition of an organizational readiness and capacity assessment stage to the PBMA process 36 and designing each PBMA stage so that it is functionally independent of the other stages. These two recommendations constitute changes to the standard PBMA approach described in the literature to date. 36 The organizational readiness and capacity assessment stage is also discussed, primarily in relation to the need for rather than the details of the assessment phase, in the previous chapter on Fit. 93 Consistent with previous research (e.g., Mitton & Patten, 2004), these IH participants desired a priority setting process that was more structured and evidence-informed than their baseline priority setting practice 37 . However, whether the full standard PBMA approach is required to evolve priority setting to be more structured and evidence-informed is questionable. Alternatively, inability to readily achieve these attributes should not preclude PBMA adoption. For example, data paralysis – lack of usable data – was routinely brought up as justification for not being able to fully implement PBMA. However, whether data availability and usability is a primary ‘system antecedent’ (Greenhalgh et al., 2004)38 for PBMA is unknown. Other system antecedents, like organization and participant ‘absorptive capacity’ and ‘receptive context for change’ (including leadership, clear goals and risk-taking climate) (Greenhalgh et al., 2004) limited PBMA implementation in this study. Couching PBMA implementation in a broader change management strategy may facilitate its acceptance. Change happens at step six of Kotter's (1995) 8-step change management strategy; the first five steps prepare the organization for change. Ambrose (1987) highlights several factors required for successful implementation – including vision, skills, incentives, resources and an action plan – all of which can be considered foundational stages that can facilitate change rather than change itself. Thus, the addition of a pre-PBMA assessment phase to determine (and then prepare) system and decision maker readiness to adopt PBMA seems prudent. This stage can include assessing the local evidence/data availability and usability (e.g., to develop the program budget), and ensuring resources (in particular, decision maker time and power to follow through) are available. Necessary human factors include multi-level leadership, local champion(s), research assistance, conducive group dynamics, and participant capacity. Early recognition of and attention to these factors as potential implementation barriers may have enabled us to work through the issues (e.g., lack of resources and support; leadership change) that ultimately led to the premature termination of this PBMA process. 37 These desired attributes of priority setting practice were introduced and discussed in the previous chapter on Fit. 38 See Appendix B for details of the Greenhalgh model, which includes the concepts system antecedent, absorptive capacity and receptive context for change. 94 Clarifying and confirming the project scope and aim, both of which were questioned throughout this study, should be a key element of an assessment phase. This may help to alleviate concerns that arise during what can be (scope and aim discussions) a political and therefore contestable process. Participants questioned the rationale to restrict PBMA to community care. Although other successful community-care-only PBMA projects (Mitton, Dionne, Damji, Campbell, & Bryan, 2011; Urquhart et al., 2008) have been conducted, these participants found it difficult to think of innovative new approaches to service delivery when confined (by project scope and therefore budget) to community care. They argued that novel approaches to service delivery required a cross-continuum perspective and therefore inclusion of other sectors (acute, residential). They stated that this would help increase the opportunity that sufficient funding was available, and enhance acceptability and implementability of any proposed changes as the impact on all sectors would be considered. This is described as ‘systems thinking’ by Senge (2006) and helps improve clarity (as broadly defined in this study) by highlighting the patterns in a system so that participants can see how to change them effectively. Greenhalgh et al. (2004) call this an ‘assessment of implications’ and recommend that the innovation’s implications be assessed and used to help develop system readiness for the innovation. In our case, this assessment only occurred after using PBMA; it indicated that participants were more open to using some aspects of PBMA (criteria, consideration of opportunity cost, collaborative approach) than others (proposal development, rating and ranking). This links back to the second practice recommendation resulting from this work: designing PBMA with functionally independent stages. If each PBMA stage was functionally independent of the other stages, participants in this study could have implemented those stages that resonated for them first and evolved their priority setting practice accordingly. This may have resulted in increased individual and organizational readiness to further evolve priority setting practice by implementing the other PBMA stages – this is an area requiring further research. Clarifying and confirming the project scope and aim with all participants, as part of the organizational readiness and capacity assessment, should occur before proceeding and again after any significant contextual changes occur or decisions that affect the scope are made. For 95 example, early in PBMA implementation in this study, the original director invited stakeholders from acute and residential to participate as this appeared like a natural fit at the time. However, in retrospect, this should have been identified as a change in scope and brought forward to discuss in detail with implications determined and addressed first before other participants were actually invited. The desired aim of the process and anticipated commitment required of participants should be clear to all participants to enhance their buy-in. Consistent with Halma et al. (2004), and despite education provided by the researchers prior to implementing PBMA, we found participants were unclear about what PBMA could do for them and how much time and effort it would take to implement and realize the intended benefits. This could be due, in part, to a lack of individual and organizational capacity and readiness to adopt PBMA. A lack of capacity may have been responsible, also, for a perception of limited control over and input into the budget – a known barrier to PBMA implementation (Halma et al., 2004) – with a sense that perhaps participants were being asked to complete a task (develop and implement proposals for redesign and reallocate resources accordingly) which was outside of their control and/or capacity. The perception of limited budget control was reinforced with the leadership change. Confirmation and clarity of scope, aim, and desired outcome(s) may have helped stabilize the process through the change in leader and promoted participant buy-in. Reworking the PBMA aim and confirming with participants that it is consistent with their power and scope of influence would have enabled participants to use PBMA in a way that would help them further evolve their priority setting practice in response to the demands of their organizational reality. As a result of our experience with participant perceived lack of control, and to ensure the priority setting process supports achievable resource reallocation, the assessment phase should include an assessment of participants’ scope of control. In this study, lack of participant (perceived) control was one barrier to proposal development. Participants found the emphasis on proposal development for redesign was not compatible with their needs or authority. However, they did find a clear advantage to using the criteria, not in the intended way (for proposal rating), but rather incorporated into daily decision making. In this respect 96 (and in retrospect), participants preferred to spend time and energy clarifying and implementing the criteria in their day-to-day decision making outside of PBMA rather than evolving them for use in proposal rating and ranking. This requires adapting PBMA as per our second practice recommendation, i.e., so that the criteria development stage (stage four) is functionally independent of the other stages. It also requires defining PBMA’s benefits more specifically to the organization and the intended use – strategic planning or budgeting, etc. For example, PBMA may be better clarified as a framework for making sense of situations, for telling us what factors deserve priority based on research and desired results. Such a framework, rather than providing narrow, specific prescriptions, should offer questions to force [participants] to think about and analyze the [options]. (Patton, 2011, p. 100) Returning to the aim of PBMA, our findings indicate that this should be perceived of as evolving or changing each year according to the individual and organizational capacity to participate. Since building relationships in the context of priority setting takes time, perhaps a year-one PBMA goal should be to build relationships and trust rather than focus primarily on identifying areas for service growth, especially vis-à-vis areas for resource release. Trust is fundamental to realize genuine and implementable resource release proposals. And, as Jan (2000) points out, it may be difficult to establish trust given the underlying premise of competition which is generated when managers’ proposals are rated against one another in PBMA. Revising the PBMA aim in year-one to be about trust building may require conceptualizing PBMA as a 'resource management framework' (Donaldson, Bate, Mitton, Peacock, & Ruta, 2007) rather than a priority setting framework, at least in the first year. This year-1 PBMA goal and how it relates to the PBMA goal(s) in subsequent years of PBMA use – e.g., supporting decision makers to set priorities for resource reallocation – should be discussed and clarified with participants during phase one and at other times when scope and aim questions arise. This will serve to enhance clarity for participants, as trust was found to be one component of clarity and one anticipated outcome of improved clarity in this study. Fundamental to PBMA, whether conceptualized as a priority setting or resource management framework, is the ability to glean pertinent information about the programs and their budgets 97 to inform the process. The inability of participants in this study to develop a program budget or even produce clear program descriptions to share with other advisory panel members may have been an early sign of implementation issues. However, as was briefly mentioned in the literature review, arguments have been made both for and against the inclusion of formalized program budgeting in PBMA (Mitton & Donaldson, 2004c) so this inability should not be an insurmountable barrier to PBMA implementation. Having said this, there are several compelling reasons to include the program budgeting stage in a PBMA exercise. Brambleby (1995) indicates that program budgeting provides information, logic, continuity, and openness to support long-term planning and priority setting. With inclusion of sufficient information, a program budget can “give a fairly clear picture of the relationship between current provision, health care needs, the policy options available and the spending decisions made” (Redmayne, Klein & Day, 1993, cited in Craig et al., 1995, p. 108). Thus, along with being an indicator of potential implementation issues, the lack of a program budget may have also stymied innovative thinking for service redesign (proposals) in this study. Since attention to opportunity cost is a fundamental principle of PBMA, and the program budget is the primary vehicle to address opportunity cost in PBMA, inability to develop a program budget should be a signal to attend to this activity in a way that serves the needs (intended use) of the participants. This can be another element of the contextual readiness and capacity assessment phase. Finally, returning to our second practice recommendation (designing PBMA with functionally independent stages), participants in this study indicated that PBMA would be better contextualized if it was implemented with each stage functionally independent of the others. This would enable users to more readily use parts of PBMA rather than the full approach if desired. Designing each PBMA stage so that it is functionally independent of the other stages should also help to mitigate some of the implementation concerns that arise with leadership changes. Buy-in from and education of a new leader is required; however, the whole group participating in PBMA would not necessarily have to return to stage one if the stages are self-contained. Leadership changes are inevitable in healthcare, especially when implementing an approach like PBMA over a long period of time, so redesigning PBMA in this way should be of benefit in other contexts where leadership change occurs. 98 4.4 Summary of implementation Although PBMA has been used in healthcare for over thirty years, unresolved issues exist regarding how best to implement PBMA under real-world conditions. In fact, “history suggests that implementation processes are likely to be the weakest link in turning…proposals [for change] into reality” (Bevan, Ham, & Plsek, 2008, p. 2). The decision can be made to either adopt or reject an innovation like PBMA in its entirety, or to only adopt those stages that users deem pertinent. Rather than focusing on implementing the full PBMA approach (high fidelity implementation as per Patton, 2011), it may be that focusing on adaptation or adoption of those stages that decision makers are ready for may serve decision makers well in terms of using PBMA. Attention to individual PBMA stages, and clarifying the purpose, use and benefits of each stage independent of the others, enables users to adopt those stages that resonate for them. Our findings suggest that this adaptability be built into PBMA implementation, along with a focus on clarity (and the issues encompassed by the term clarity in this study) through targeted education and communication, and the addition of an organizational readiness and capacity assessment stage. Assisting decision makers in this way, rather than focusing on implementing PBMA in its entirety, may have more influence on evolving priority setting practice. This is an important area for future research. 99 5 Chapter: Impact – Results and Discussion 5.1 Synopsis of chapter PBMA is described as an approach or tool to support decision makers with setting health service priorities to inform resource allocation decisions and as a “practical application of economic principles” (Peacock, 1998b, p. 1). As indicated in the literature review, various studies into its use have determined that it effectively incorporates not just economic principles (Mitton & Donaldson, 2004b), but also evidence-based practice and ethical principles (Gibson et al., 2006). Multiple surveys have also shown that decision makers prefer it over priority setting based on historical funding decisions and/or political influences (Dionne et al., 2008; Mitton & Donaldson, 2002; Teng et al., 2007). The literature review discussed how PBMA impact has been evaluated and reported in the literature. Impact is most commonly described in terms of the influence PBMA had on priorities set and/or resources reallocated (Peacock et al., 2010). This may be due to PBMA’s historical roots and fundamental principles being based in economics. Setting priorities and allocating resources accordingly are economic in nature. Both of these impacts are experienced at the system-level; i.e., impact experienced across a set of interrelated units, like an organization, that share a common goal (Rogers, 2003). The literature review also discussed how the differing definitions of impact (often reported in terms of success from implementing PBMA) make it difficult to compare studies with one another. In some PBMA studies, authors have indicated limited impact on specific outcomes, like effectiveness (Bohmer et al., 2001) or monetary impact (Urquhart et al., 2008), yet also reported that participants still found PBMA was “worthwhile and valuable” (Bohmer et al., 2001, p. 47) and that it had a positive impact. When the ultimate goal of using PBMA is to reallocate resources to better meet organizational objectives (Peacock et al., 2010), determining success by this outcome makes evaluative sense (Patton, 1997). However, resource reallocation is one aim of PBMA; there are other aims, so a broader definition of positive impact is required to evaluate for success according to the achievement of these other aims. 100 This thesis chapter reports and discusses the findings from studying the impact of implementing PBMA as a formal priority setting approach (thesis research objective #3, Section 1.4). Despite the early discontinuation of PBMA, participants indicated that PBMA still had a positive impact in this setting. The purpose of this chapter is to describe the impact on the participants, the community care group, and on the system (Patton, 1997) by asking: 1. Did PBMA affect resource allocation in this study? 2. Did PBMA change practice, including decision maker use of evidence in decision making? 3. Did PBMA change decision maker knowledge of and attitudes towards priority setting, including how decision makers learn from and use PBMA? Impact is interpreted in light of the estimated implementation cost – what did these community care decision makers get for their time spent participating in PBMA? Sibbald et al. (2009) outlined the elements germane to successful priority setting (see literature review for details) and thereby expanded the definition of successful priority setting beyond just economic criteria like priorities set and/or resources reallocated. However, they did not provide input into how to determine impact. The findings reported in this chapter provide input into how to determine impact and also contribute to the definition of success, a primary component of impact, by broadening the definition of impact to include not only system-level outcomes like resource re-allocation but also individual and group-level outcomes like changes in priority setting knowledge, attitudes and practice. In this way these findings address an important gap in the literature about determining the impact of implementing a formal priority setting approach, which will be of interest to researchers and decision makers using or considering a formal priority setting approach to manage scarce healthcare resources. 5.2 Impact results Analysis using a social constructivist lens 39 revealed that PBMA had several distinct positive impacts in this study and that these occurred at different levels ranging from the individual to 39 See methodology chapter (Section 2.2.1) above for details of how this philosophical perspective influenced the research activities, including the analysis and documentation of the findings. 101 the system level. These impacts included the desired impact of setting priorities for resource reallocation, as well as impacts on individual participants and the community care group that are seldom formally acknowledged in the priority setting literature. These impacts were found on a continuum, culminating with the intended and desired system-level effects on priority setting as determined by priorities set and resources reallocated. However, impact was also found at the individual and group levels on priority setting knowledge, practice, and attitudes. Results are grouped and discussed below according to the level at which the impact was determined, e.g., at the system, individual and group levels. Individual and group level impacts are discussed collectively as these impacts were so interrelated, and separating them was not required for the purposes of this study. 5.2.1 System-level impact System-level impact occurs across a set of interrelated units that share a common goal (Rogers, 2003). In this study, the system-level impact included setting priorities for resource reallocation. This impact is experienced at the system-level as resources are reallocated across program areas within the community care scope of the project, with subsequent changes in service delivery potentially affecting the broader organization and healthcare system. PBMA was presented to the community care group as an approach to assist decision makers in making choices about competing priorities. The key goals were articulated as supporting the group in setting priorities to allocate resources to both align with strategic objectives and to maximize benefits from limited resources. This was described during the introductory workshop (the main educational component provided) and articulated in the support articles (Gibson et al., 2006; Peacock et al., 2006) provided to the group at the start of the study. Although other potential benefits were mentioned, including stakeholder engagement and legitimacy of decisions, the primary intended and desired outcome was presented as setting priorities for resource reallocation. 102 Regarding priorities set, at the end of the first year using PBMA, participants did rate and then prioritize a list of investment and disinvestment proposals which included approximately $760,000 worth of investment proposals – a figure twenty times greater than the $38,000 generated for disinvestment proposals 40 . It was during the final PBMA validation stage in year-1 (stage 7, Table 2.3) that some members of the group voiced concern over whether these proposals were the right ones to be implemented at that time. The decision was made to table PBMA and revisit the aim of the activity before continuing with the study. After a two-month break over the summer, PBMA began again (year-2) but was then cancelled (permanently) at stage 5 (identify investment and disinvestment options). Concerning reallocations made as a result of PBMA, even though PBMA was stopped mid- stream, participants had a mixed response about whether PBMA was used to set specific priorities that were then used to guide system-level resource allocation. Some participants said “no” (Year-1 Participant #5). Others said that reallocations were made but only in the usual ‘silos’: “we didn’t make any re-allocation decisions really; we made some reduction decisions within silos” (Year-1 Participant #11). Some participants said that they did not know: “I have no idea” (Year-1 Participant #1). And, some said yes: “According to the way we had set it up, then no. But according to the principles and what we actually did, yes. Because we [addressed] some of the budget challenges we had through that whole process of reallocations” (Year-1 Participant #2). I would say that there were resources re-allocated but not at the time, or how can I say this, so we had our meeting where we put weighted values on the initiatives put forward. And then the sense was that the process kind of died there and that nobody did anything with it. But when I look at the strategies put forward for the ’08-’09 run rate for the operating budget, I see some of the initiatives coming into play. (Year-1 Participant #12) It is clear from these findings that it depends on the individual participant perspective as to whether system-level resource reallocation is perceived to have occurred. Although all decision makers participated in the same PBMA exercise, individual responses varied and appeared to depend, in part and as expected, on their individual experience, comprehension 40 See Figure E.1 in Appendix E for details. 103 and acceptance of the activity. Compare these responses which depict either end of the spectrum of comprehension and acceptance: I think what PBMA has done in helping us as a group is help us develop some of those decision making criteria around skill mix – who is the most appropriate person to do the job instead of always just back-filling based on tradition. So another strategy is looking in Home Care Nursing we have had three vacant lines that haven’t been filled for a long time – do we really need those positions? So I think what it’s done is heightened people’s awareness along with perhaps my expectation that first of all we have to save $2.7 million dollars this fiscal year and there is no new money, so anything that you want to do differently you have to find the money from within. And that is kind of what PBMA is about – really having a hard look at what are the right things to do and in what amount and what opportunity do you have to give up to do that. (Year-1 Participant #12) I think I was just kind of so lost in ‘where are we going’, and ‘what are we doing’. So I didn’t take the time. (Year-2 Participant #6) The former participant is able to describe PBMA with specific contextual examples, yet the latter participant indicates a lack of overall understanding and capacity to use the PBMA process. Other participants had levels of understanding and capacity that were in-between these extremes. Although these findings indicate that priorities were set, participants varied in their responses regarding resource reallocation which was the other system-level impact. As a result of the variable perspectives described above and the differing positive system-level impacts described in the literature, we were interested in determining what other ways these individual participants experienced PBMA similarly or differently: what other less- emphasized impacts (other than priorities set or resources reallocated) were found at the individual and group level, and how did these interact with the intended system-level outcome? 5.2.2 Individual and group level impact Along with the mixed response about whether PBMA implementation resulted in setting priorities that guided resource reallocation, participants also had varying responses concerning if and how PBMA affected their individual and group priority setting knowledge and practice, and individual priority setting attitudes. This variability existed between 104 participants and within some participants. For example, participants described positive impacts from PBMA on some aspects of their individual or group knowledge or practice, and negative or no impact on other aspects. The impact on priority setting knowledge and practice was articulated and demonstrated at both the individual and group levels to varying degrees and in an interrelated fashion: participants routinely merged individual and group impacts, and group and individual impacts appeared to depend, at least in part, on each other. As a result, individual and group-level impacts are presented together in these results. Some participants stated that the principles underlying PBMA aligned with their thinking and indicated that they were able to evolve their priority setting practice accordingly. I quite like the process of looking at outcomes, looking at the evidence-base and trying to make proposals about what is a future-oriented direction and where should we be investing either potential new or liquidated dollars. I thought that process was perfect. (Year-2 Participant #7) For these participants, PBMA had a positive impact on at least some aspects of their priority setting knowledge, attitudes and practice which they described as impacting them both individually and at the community care group level. For example, these participants stated that PBMA contributed new insights into priority setting in general and supported a more detailed approach: “I think people generally read each others’ work and became aware of each others’ work differently” (Year-1 Participant #9). [PBMA] gets people to actually look very closely at each component of their programs or their services and at how they are being done and who they are serving and looking at how else could that be accomplished. And then figuring out what they would need to do to accomplish a change. (Year-1 Participant #1) Some participants said PBMA introduced a new way of thinking: “I am thinking a lot more often around opportunities for shifting resources” (Year-2 Participant #6). “It got people thinking of the ‘sacred cows’ and how sacred they aren’t. It also, I think, got people realizing one change that they may make in their area, what impact it had on others” (Year-1 Participant #2). “It is a new way of thinking and I am really proud of all the program leads and managers attempting to come up with a win-win. So how can we save money but not jeopardize the service” (Year-1 Participant #12). 105 And some participants indicated that they learned something new about resource scarcity and economic considerations. [PBMA] heightened people’s awareness…that first of all we have to save $2.7 million dollars this fiscal year and there is no new money, so anything that you want to do differently you have to find the money from within. And that is kind of what PBMA is about – really having a hard look at what are the right things to do and in what amount and what opportunity do you have to give up to do that. (Year-1 Participant #12) The overall process helped to shift us from output-oriented language to outcome- oriented language, which is substantial. I think in Community Care there is such a huge volume of community nursing activity that is output-based: a visit is a visit. If it is referred by a doctor, then there is no question that it is valuable and has to be prioritized, etc. It is a little different, then, to look at it and say, ‘What is the value that we get from a one-hour home visit versus a clinic visit?’; ‘Can you impact nursing time by providing better nutrition and social work and pharmacy and other support services?’ (Year-2 Participant #7) Specific to individual-level impact, PBMA appeared to have a positive influence on some participants’ attitudes toward priority setting: “What has changed is I have a sense of more hopefulness that more of this would be applied on a more regular basis and a heightened awareness of its relevance and role across our team” (Year-2 Participant #3). At the other end of the spectrum, other participants described struggles with seeing how PBMA could fit into their current practice and/or the community care context. I don’t see pure PBMA working in our whole system. It is just too complex. Political forces, reallocating waste, disengagement of staff, people being strategic in the process. There are too many ‘what-ifs’ to have the pure PBMA process work here. (Year-2 Participant #2) These participants were less clear about how PBMA impacted their individual priority setting knowledge, practice and attitudes, and the group’s knowledge and practice. These participants were more likely to express a negative or no impact of PBMA, not overall, but rather for certain aspects of priority setting. There [were] people, including me, not doing their homework and being a little bit slow on the uptake when we actually had to go through the process. Like ‘Remind me what we said about this. How were we supposed to do that again?’ So I think in that sense there was potentially a little bit more academically-friendly than manager- 106 friendly, and that was just because it was occurring at a time when nobody had time. (Year-1 Participant #7) At the end of it, marginal analysis didn’t really deliver, because that evidence wasn’t there or that information wasn’t there in that format. We had to just kind of eyeball it. And I think that was very subjective. (Year-1 Participant #11) People might think [PBMA] is applicable to healthcare but the more I think about it, the less I think so. It has got to be relevant. For things to be effective, they have to be relevant. So it gets back to that change management thing and if it is not relevant to those that make the change, and they are not understanding reallocation, and they are not seeing the benefit of it themselves, then you are not going to get people shifting cultures and practice. (Year-2 Participant #4) Concerning how PBMA impacted specific aspects of priority setting practice, the responses again varied. Some participants were preoccupied with the time and effort required to fully implement and engage in PBMA – e.g., “This process takes time and we just didn’t have the time to do it” (Year-1 Participant #4) – whereas, other participants indicated that PBMA positively impacted practice in several ways. These latter participants were more likely to identify that PBMA facilitated robust decision making, encouraged a more collaborative approach than baseline priority setting practice 41 , changed the criteria used to guide priority setting and facilitated a consistent approach with the use of the participants’ jointly- determined criteria. These participants also stated that PBMA changed the way in which evidence was used in the process. These impacts on priority setting practice were found to be intrinsic to realizing the system-level impact of resource reallocation; e.g., one system-level impact was a group-generated priority list which was based on a collaborative approach to addressing group-identified budget challenges (as described above). [PBMA] set a precedent that there is a different way of doing business than what we had done in priority setting before. So priority setting pre-PBMA was pretty exclusively portfolio budgeting and not remotely kind of cross-managerial. So it put out there the notion that we can sometimes muck in together, that we can establish priorities within all of our programs versus simply establishing priorities within one portfolio. I think it opened up the understanding that knowledge has to contribute to decisions. And that the idiosyncratic knowledge that people have around their own budgets and expenditures is less useful than looking at an evidence-based and outcome-oriented review for priority setting. So I think it introduced all those ideas. 41 Baseline priority setting practice refers to the practice prior to the introduction of PBMA to this group and is discussed in the Fit chapter above. 107 That is why it was so exciting right from the start – it potentially opens up your decision making process to new information, to evidence base and to a different way of looking at things. So what it kind of attacks is that old myopic vision of ‘we need to do everything that we are doing plus we need to do more’. (Year-2 Participant #7) Those participants who connected with PBMA or at least some parts of PBMA found that PBMA facilitated more robust decision making compared to baseline priority setting practice. They found it more structured and transparent, more rational and evidence- informed. “PBMA I think provided us with a structure to look at those difficult decisions in a more measured, rational kind of way” (Year-1 Participant #11). [PBMA] has [us] thinking differently relative to our resources, how they are allocated, what they are achieving, is it what they should be achieving, can we do better, how can we do better…I think PBMA helped make that more robust by coming up with the tools, the criteria, the prioritization, different way to write up the proposal – kind of a nice little package (Year-2 Participant #5) Some participants expressed appreciation for the collaborative approach facilitated by PBMA, comparing it to the ‘silo’d’ approach familiar in their baseline priority setting practice. “The traditional way is very top down and PBMA I think opens the door to a much more collegial, collaborative process” (Year-1 Participant #11). Historically, [my role] has been to defend current budget allocations, make sure that they never, ever get cut and look for any opportunity to vie for new dollars for new programs. So the difference in the PBMA approach to me is it invites me not to look at existing budgets as quite so sacred, not to look at my role as a manager being to necessarily protect my budget but rather to look at my role as a manager on a broader level to enhance and improve services to our clients, which you would think should be part of the job all the time anyway. (Year-1 Participant #7) Some participants also expressed appreciation for the consistent approach to priority setting facilitated by the criteria developed in PBMA. [What] resonated the most for people was the consistent lens in which we were reviewing things. Because that is what the criteria kind of enabled us to do. Are we all looking at this from the perspective of where it fits with the priorities, where it impacts health, where it impacts the system? (Year-2 Participant #6) Some participants stated that PBMA influenced priority setting practice by changing the criteria used to guide priority setting and in how evidence was used in the process. They described how the criteria helped them move away from a ‘flavour of the day’ and/or 108 historical approach and towards aligning decisions with the criteria agreed on during PBMA. Table 5.1 includes the primary pre-PBMA priority setting drivers 42 and the criteria chosen during the PBMA process. Participants said the criteria facilitated a consistent approach, as described above, and a broader perspective for priority setting. They indicated that the broader perspective helped align priority setting with organizational objectives and the strategic plan rather than focusing primarily on the budget. As a result of the PBMA process, what we have committed to is ensuring that our priority setting is done within the context of the IH Strategic Direction. And that was a big part of the criteria that we created, that it had to support that. (Year-2 Participant #5) Table 5.1 Primary priority setting drivers pre-PBMA and criteria chosen as drivers during PBMA process Pre-PBMA drivers 43 Criteria chosen during PBMA process Government/political mandates, e.g., ‘flavour of the day’ Public scrutiny, media attention Based on past budget allocations (historical) Ad hoc (lack of explicit, formal process) Client impact o Community and client needs o Health maintenance/gain o Accessibility and equity Organization and system objectives o Inter-dependencies (internal) o Strategic direction/fit o Alignment – with external directives o Management effectiveness Human resources and innovation o Clinical and staff capacity o Innovation o Engagement Regarding how evidence was used in the process, some participants stated that PBMA “made [priority setting] more evidence-based; that it is ok to leave a traditional service delivery model if you can actually demonstrate that you can deliver value in a different way” (Year-2 Participant #7). Proposals were developed by all participants, those who connected with PBMA and those who indicated concerns with its fit (either with their individual or group priority setting practice), and this may have affected participant engagement in proposal development. As a result, the types of evidence incorporated and how it was used in the 42 The term driver, as used in this thesis, is defined in Chapter 3 – Fit above. 43 These drivers are discussed in detail in Chapter 3 – Fit above. 109 proposals developed in this study varied. Overall, to a greater or lesser extent, proposals included the following: costing and activity data, information on local need, program effectiveness data, links to organizational objectives and Ministry of Health policy, research literature, and expert opinion. Whether they connected with PBMA or not, participants indicated a tension between a desire to be more evidence-informed in their priority setting and difficulty with acquiring and using relevant evidence. “We would if we had the data, we would if we could use the data, there is lack of evidence…that is not true – there is a lack of research evidence, there is lots of other kinds of evidence” (Year-1 Participant #6). A key piece that is missing [is] the data to support decision making. So it is great for me to throw what I perceive to be something I can loosen up and throw on the table but it should be justified and we don’t have the statistical information to do that. (Year-1 Participant #4) However, participants who connected with PBMA were more likely to move past the data paralysis concern and view PBMA’s impacts on evidence optimistically. For example, these participants viewed favourably that PBMA highlighted data and organization issues that limited their ability to make evidence-informed decisions: “what I thought was [PBMA’s] strength…was the awareness of the deficiencies in the organization of program outcomes, of data, of organizational culture” (Year-1 Participant #9). Other participants got stuck on the organizational limits to evidence-informed priority setting. We get caught in the conundrum of wanting information and not having it, and having it and not trusting it. And then having it and not using it. What we do have. So we are not good at using data or information in this organization, specifically in Community Care. So that was another hurdle for PBMA that made it hard to ground itself. (Year-1 Participant #5) These individual and group-level impacts on priority setting knowledge and practice appear to be both essential to and effects of the system-level impact of setting priorities to reallocate resources. Resource reallocation as an outcome of implementing PBMA is experienced at the system-level; however, it requires individual and group action to be realized; e.g., it requires participants to incorporate new knowledge about resource scarcity and economic considerations into their priority setting practice, and to adopt a more collaborative approach 110 to priority setting in order to collectively create a ranked list of prioritized options to inform resource reallocation. And individual and group level impacts depend, in part, on the system- level outcomes achieved; e.g., in this study, lack of organizational follow-through specific to the priorities set as a result of PBMA negatively influenced participant priority setting attitudes – see second quote below. A participant comment reflecting the perspective that system-level impact requires individual and group action to be realized is: It takes someone committed to the learning to act on it. ‘Cause there is a wonderful idea and knowledge coming out of that…and it goes nowhere. That’s the thing. So the whole piece about lack of action is kind of symptomatic of the whole organization – they don’t act. (Year-1 Participant #9) And, a participant comment reflecting the point that individual and group level impacts depend on system-level outcomes achieved is: After we went through the whole process of rating, weighting and then re-articulating what the top re-investment potentials would be, the two things that came out as most important weren’t going to be done because they were still ultimately seen as less crucial than deficit reduction. So I think in the end the top two were the Community Nutrition program expansion or solidification and the guardianship training that staff require. So through I think a fairly clear process of values lens, criteria lens and then relative weighting, those emerged to be the top and everyone was kind of surprised. It was like ‘Oh that’s good, but we are not going to do those because (1) they are not going to have a huge impact on deficit and (2) they seemed a little bit kind of beside core business or a little bit off to the side of core business. And to tell you the truth that was a discouraging moment when we actually came up with “Here is what we should be doing to fully honor and complete the process”, and fairly quickly it was determined that there must have been a mistake in the criteria or the weighting there because Social Work and Nutrition are the ones that are going to get new dollars, and Nursing isn’t. (Year-1 Participant #7) To summarize, participants had varied responses about the individual and group-level PBMA impacts in this study. There was variability both between participants and within participants concerning the impact on individual and group priority setting knowledge and practice, and on individual priority setting attitudes. Participants who connected with PBMA indicated that PBMA positively impacted practice in the following ways: it facilitated robust decision making, encouraged a collaborative and evidence-informed approach, changed the criteria used to guide priority setting, and using these criteria facilitated a consistent approach. We 111 also found a tension between the desire to be more evidence-informed in priority setting and difficulty acquiring and using evidence. Overall, these results demonstrate how impacts at the system, group and individual levels are inter-related and dependant, to some degree, on one another. They also indicate that participants connect with PBMA and/or different aspects of PBMA in different ways, and that this affects how and to what extent PBMA impacts individual and group priority setting knowledge and practice. 5.3 Impact discussion Since PBMA has evolved to include a fairly standard implementation approach, we can determine the approximate cost, in terms of decision maker time, to participate in PBMA. The participants in this study devoted approximately 320 person-hours to attend PBMA meetings; this does not include individual meeting preparation time and is based on a total of ten meetings – seven for the first cycle of PBMA, and three for the second cycle which was not completed. Other PBMA exercises may take more or fewer meetings, depending on the scope and aim of the exercise. We interpret the impact in our study in light of this estimated implementation cost: what did these decision makers get for spending approximately 320 hours of their time participating in PBMA? The impact of using a formal priority setting approach can be determined from the system, group, and individual manager perspectives. When considering whether resources are reallocated between service areas as a result of using a formal priority setting approach, impact is determined at the system level. Although other benefits are described in the literature, this is how success in priority setting is usually defined (Peacock et al., 2010). In this study, although some participants could clearly articulate the resource reallocation that they believed 44 occurred as a result of PBMA, not all participants were sure that system-level resource reallocation was a direct outcome of using PBMA. Difficulty conceptualizing or achieving system-level resource reallocation may be a result of the way resources are 44 Inclusion of individual belief or perception of system-level outcome aligns with the social constructivist conceptualization of knowledge as described in the methodology chapter above. 112 allocated in general in the health authority – on a budgetary rather than an economic basis. This is a structural or institutional barrier to PBMA use (Williams & Bryan, 2007a) and reinforced the ‘silo’d’ approach to priority setting articulated by participants in this study. The lack of consensus on whether or not resource reallocation resulted from PBMA may also have been due to a change in community care director towards the end of the first PBMA cycle. Not only did this require additional meetings and repeat discussions to confirm project scope and aim, but the new director’s mandate appeared to be different from that of the previous director and not necessarily conducive to collaborative decision making. In this study, collaborative decision making emerged as an individual and group-level impact and appeared to be essential to achieving system-level impact (e.g., priority setting): we found that priority setting to inform system-level resource reallocation required collaborative problem and solution identification. The leadership change stalled PBMA momentum. The requirement for additional meetings and lengthy and repeated conversations amongst participants is specific to this context as other PBMA exercises our research team has been involved in have taken fewer meetings and less overall decision maker time to conduct. Research has found that organizational instability and lack of supportive leadership often results in failed (defined by limited system-level impact) priority setting processes (Peacock et al., 2010); so overall PBMA impact must be considered in light of the leadership change in this study. Impact, in this study, was mainly found at the individual and group levels; specifically, impact on priority setting knowledge and practice. Although not the primary impact commonly sought and documented when implementing a formal priority setting process, individual and group level change in priority setting knowledge and practice cannot be discounted. In this study, individual and group-level impacts were found to be closely connected to system-level impact, such that individual and group-level impacts appeared to be prerequisites for system-level impact and, at the same time, outcomes of system-level impact. See Figure 5.1 for a visual representation of the inter-related and inter-dependant impacts, including some examples of how impact has been defined in the priority setting and PBMA literature. Individual priority setting behavior may have more impact on organizational and group priority setting, and therefore system-level resource reallocation, in 113 the long-term as organizational (group) learning and change is often a slow process that requires individual learning and change (Senge, 2006). We found that individual and group- level impacts were interrelated with each other and with system-level impact, and varied between and within individuals regarding different aspects of PBMA. Therefore, attending to these variances during PBMA implementation is prudent in the interest of achieving intended system-level PBMA impact. This connection between individual, group and system-level impacts requires further study. 114 Figure 5.1 Impact: continuum, how it relates to readiness & capacity assessment stage Literature Individual-level impact Group-level impact System-level impact (Mitton & Donaldson, 2001) - Continued PBMA use - Change way of thinking - Principles of framework understood and relied on - Priorities set - Resources reallocated (Tsourapas & Frew, 2011) - Participants’ increased understanding of area under interest - Adopting framework for future use - Disinvesting or resource allocation - Implementing all/some of advisory panel’s recommendations (Sibbald et al., 2009) - Stakeholder understanding, engagement, acceptance, satisfaction - Decision making quality - Decision making quality - Explicit process that considers values and context - Information management - Inclusion of a mechanism to appeal or revise decisions - Resource reallocation - Positive externalities Readiness and Capacity Assessment - Individual, group, & organizational (system-level) capacity and readiness pre/during PBMA - E.g., available resources, time - E.g., sufficient knowledge, control - E.g., acceptable project scope, aim, desired outcomes Individual-level Impact - Knowledge - Practice - Attitudes  Individual mastery Group-level Impact - Knowledge - Practice  Group mastery System-level Impact - Priority setting - Resource reallocation 115 Concerning individual variability in impact, PBMA broadened some participants’ perspectives of what priority setting could entail and aligned well with their priority setting thinking. These participants appeared to connect with PBMA, or some aspects of PBMA in particular, and experienced more positive impacts from using it than those participants who did not connect with PBMA in general or had difficulty with specific parts of PBMA. These results indicate that individual variance in comprehension and/or acceptance of PBMA is important to assess for and address. This is especially important in order to achieve resource reallocation based on collaborative priority setting as collaboration requires all participants to share a common vision and responsibility in the process. Specific to overall comprehension of priority setting and how PBMA can support priority setting practice, participants indicated a personal and organizational need to take time to learn about priority setting and managing scarce resources since they felt that healthcare historically has been focused more on finding ways to provide additional service than on accepting limitations. This is evident in how participants generated almost twenty times the dollar amount of investment proposals than they did of disinvestment proposals, which is not uncommon in the first year of implementing PBMA (e.g., Dionne, Mitton, Smith, & Donaldson, 2009). Healthcare providers appear to be more comfortable in offering service than in discontinuing it, even when an approach like PBMA brings focus to limited benefits being provided by some services. Some individuals connect with this concept more readily than others and therefore individuals require varied amounts of time and effort to adjust priority setting practice accordingly. Regarding priority setting practice, participants who indicated a connection with PBMA perceived that it influenced their individual and group priority setting practice and knowledge in several ways that were essential to system-level impact. These participants indicated that PBMA influenced the actual criteria used in priority setting, which enabled a broader perspective and helped align priority setting with the strategic plan. This may be an important preliminary requirement before economic techniques can be used to reallocate resources. The criteria established during PBMA bring focus to contextually-relevant factors important in determining the opportunity cost of various service options under consideration. From this focus, participants can then gather knowledge to inform economic and other evaluations to 116 compare options. Further research is required to determine whether and how PBMA can help set the stage for economic techniques to be more effective. This may require specific tools and targeted educational approaches depending on different participants’ connections with and understandings of various elements of PBMA, particularly economic considerations and techniques, and PBMA’s rational approach. The consistent approach and broader perspective facilitated by using collaboratively defined criteria enables participants (in particular, those who are ready to do so) to consider priority setting as more than a budgeting activity. The criteria can be used to assess reallocation and service provision ideas from the perspective of how they address organizational objectives and other key factors identified by participants and captured in the criteria, along with how they meet budget requirements. Indeed, this broader perspective has been identified in the literature as a desirable attribute in priority setting (Dionne et al., 2008) and therefore is worth pursuing as a precursor to resource reallocation. Some participants expressed appreciation for the collaborative (cross-managerial) approach facilitated by PBMA, especially compared to the previous ‘silo’d’ approach. They stated that this collaborative approach helped to broaden the manager role in priority setting from one of ‘budget protector’ to one of ‘program enhancer’. Distinct and separate departmental budgets, referred to as budget ‘silos’ by these participants, are common in healthcare settings; however, changing the focus of priority setting from protecting individual budgets to one of enhancing programs to maximize overall client and system benefit from service provision is possible (Mitton et al., 2003). Just the change to a focus on quality care, through enhancing programs, may help to manage scarce resources (Mitton, Dionne, Peacock, & Sheps, 2006). This impact on individual manager priority setting perspectives may not result in immediate resource reallocations; however, it may impact priority setting over time as budgets and priority setting practices evolve with individual decision maker input. Thus, again, this impact on individual and group priority setting practice may be a necessary precursor to resource reallocation. 117 Other impacts on priority setting include group learning and fostering a change in priority setting attitude to be more empowering and enlightening. Participating in PBMA improved understanding of different aspects of priority setting to varying degrees for individual participants, which enabled some participants to look at deficit management strategies differently and to see that ‘sacred’ programs, those which they would not in the past consider taking funding from, should be subject to the same scrutiny as all programs under investigation. Some participants indicated that they learned new information on resource scarcity and economic considerations in priority setting, which facilitated looking within to fund options rather than the usual practice of seeking new funding for new ideas. Again, these individual and group-level impacts were found to be intrinsic to system-level impact and were important impacts of using PBMA in this study. Future research can investigate strategies to fine-tune the education shared during PBMA to address individual participants’ learning needs to evolve all participants to an equivalent level of understanding so that the group can develop their practice together. This may require conceptualizing PBMA not just as a tool to support resource reallocation, but also as a means to develop decision maker priority setting practice and knowledge. New knowledge was a primary outcome of PBMA in this study. If PBMA is conceived of as a tool to develop individual and group priority setting knowledge and practice it is worth determining whether the learning facilitated by PBMA merits the cost compared to participants attending a priority setting workshop. Workshops are a common approach in healthcare manager education. The action research approach used in this study enabled participant learning-by-doing (O'Brien, 2001) which, for adult learners, is more effective than didactic education sessions (Redelmeier, Shafir, & Aujla, 2001). The PBMA implementation approach incorporates an initial education session and subsequent opportunities to revisit educational requirements as needed. Since change is difficult to achieve in healthcare (Graham & Tetroe, 2007; Grol et al., 2007), maximizing individual and thereby group decision maker learning and applying that learning to practice are important for facilitating change. In contrast to a one-time education session, there is residual learning and practice change that result from participants being involved in learning-by-doing and implementing PBMA in their context. As such, it appears that PBMA, especially when 118 implemented using an action research approach, has the potential to be an excellent tool to develop individual and group priority setting knowledge and practice. Some of these study participants also found that PBMA oriented them to a broader understanding of the various sources of potential evidence that can be used in priority setting. In this way, PBMA was perceived as a knowledge translation tool and as a means to organize information to assist with budget planning; the latter of which Dionne, Mitton, Smith, and Donaldson (2009) also found as a benefit of PBMA. PBMA is designed to enable the use of various data inputs and evidence; however, the approach relies on the individual proposal developers to search for and apply the evidence in the proposals. Individual variability in evidence use in the proposals in our study may have been due, in part, to a lack of individual capability and, in part, to a lack of organizational support with finding and applying relevant research evidence. This latter point was highlighted as a problem in this study. This is an important obstacle to overcome as evidence-informed priority setting requires both contextual and individual participant capacity to use evidence in priority setting. It must be noted, however, that some contexts and cultures may be more amenable to the rational, evidence-informed process facilitated by PBMA and that within such contexts PBMA will likely have enhanced impact. The mixed response about whether or not reallocations were made as a result of using PBMA, and how PBMA impacted individual and group priority setting knowledge and practice, is noteworthy. These results indicate that defining impact from using PBMA is subject to individual interpretations and requires clarification of the specific timeline and priority setting activities that should be considered as influencing resource reallocations. In this respect, impact of using a formal priority setting approach should not be subject to the timeline and activity (e.g., resource reallocation) restrictions imposed by the parameters of the particular exercise under consideration. Manager learning and application of that learning in routine budgeting and strategic planning activities outside of priority setting may have more impact on future resource reallocations than will the actual exercise under consideration. This requires further study and may influence the primary measure(s) of impact resulting from formal priority setting activities. This also aligns with Sibbald and 119 colleagues’ (2009) findings that actual resource reallocation is only one aspect of successful priority setting. In this study, other aspects of successful priority setting per Sibbald et al. included improved stakeholder understanding and engagement, decision making quality, stakeholder acceptance and satisfaction, positive externalities (such as organizational changes conducive to the PBMA approach), an explicit process that considers values and context, and improved information management. 5.4 Summary of impact To recommend PBMA as a potential solution to managing scarce resources, one needs to be able to articulate the impact it can have and has had in various contexts. System-level resource reallocation is often seen as the primary impact of using a formal priority setting approach like PBMA. There are, however, other effects of using a formal priority setting approach that also inform the evaluation of impact. The most significant of these, in this study, was that some decision makers found value in the individual and group learning and changing practice and attitudes that resulted from this first run at collaboratively implementing a new way of setting priorities to reallocate resources. Individual and group changes in priority setting practice, knowledge and attitudes, and perhaps organizational changes in procedures and culture, resulted from the experiential learning that occurred during PBMA implementation. The purpose of this chapter was to determine the impact of using a formal priority setting approach, using PBMA as an example. This work broadens the definition of impact to include not only system-level outcomes like resource re-allocation but also individual and group level outcomes like changes in priority setting knowledge, attitudes, and practice. This serves to help develop a comprehensive description of potential impacts from using a formal priority setting approach. Ultimately, this description of impact can be used as justification by decision makers to support their use of a formal priority setting approach like PBMA. 120 6 Chapter: Conclusion 6.1 Overall analysis and integration of fit, implementation, and impact chapters with priority setting research PBMA is about more than program budgeting and marginal analysis. And it is about more than meeting economic and ethical principles of priority setting. However, the current priority setting literature focuses heavily on how PBMA supports economic (e.g., Mortimer, 2010; Peacock, Richardson, Carter, & Edwards, 2007; Wilson, Peacock, & Ruta, 2009) and ethical (e.g., Gibson, Mitton, & DuBois-Wing, 2011; Gibson et al., 2006; Mitton et al., 2011) principles and outcomes. The PBMA literature sometimes comments on how PBMA supports the actual processes and practices, or informs the decision heuristics 45 , decision makers use when setting priorities and making decisions about allocating resources; however, this is seldom the primary focus. The way in which PBMA is described in the current literature suggests it is still well connected to its economic roots. For example, “PBMA is a priority-setting toolkit which aims to assist decision-makers in identifying the most efficient use of resources” (Tsourapas & Frew, 2011, p. 177); “PBMA is an established framework for systematic priority setting in which a ‘weighted benefit score’ for each option is calculated based on all the relevant decision-making criteria” (Wilson et al., 2009, p. 467); and, “The intent of PBMA is to assist local decision makers in directing resources to maximize benefits from health services, considering both opportunity cost and resource shifts ‘at the margin’” (Peacock et al., 2010, p. 539). While recognizing that these are relevant features of PBMA, it can also be argued that these definitions do not provide the full picture of how PBMA can support priority setting. As described in the literature review, the way in which PBMA is implemented has evolved with continued use in various healthcare contexts. The approach currently discussed in the literature includes seven stages. Program budgeting, an in-depth look at current services and fund distribution to provide a map of expenditures and activity, is addressed in the first two stages (Mitton & Donaldson, 2004c). Marginal analysis, the evaluative component of PBMA, 45 Decision heuristics are shortcuts or procedures that decision makers use to help “find adequate, although imperfect, answers to difficult questions” (Kahneman, 2011, p. 98). 121 is represented in the remaining stages. In addition to the economic techniques, program budgeting and marginal analysis, through its seven-stage approach PBMA also supports a collaborative, evidence-informed, structured, and rational process. Research has shown that features, like collaboration, which are sometimes viewed as outcomes of process-oriented innovations can also be viewed as novel characteristics of an innovation (Barnett, Vasileiou, Djemil, Brooks, & Young, 2011). Indeed, these features are an integral part of the PBMA approach and not only an outcome. This thesis project found that these features require specific actions and commitments (like a willingness and ability to collaborate within and across departments, finding and using various sources of evidence, adequate resources, commitment and support from leaders at multiple levels of the organizational hierarchy, decision maker buy-in and participation) from the context and the participants implementing PBMA. It is these specific actions and commitments that can facilitate PBMA implementation in such a way that it has a positive impact on priority setting, whether that impact is defined by priorities set and resources reallocated and/or by changes to priority setting practice, knowledge and attitudes. Understanding how and why specific features of PBMA were effective, and not just whether PBMA was effective, is crucial to advancing the research on healthcare innovation implementation and quality improvement (Dixon-Woods, Bosk, Aveling, Goeschel, & Pronovost, 2011). PBMA is not an entity or a process, distinct from the context, to be implemented. However, the technical presentation of PBMA as discussed in the priority setting literature (e.g., Mitton et al., 2011; Peacock et al., 2006; Tsourapas & Frew, 2011) can easily lead a reader, especially a non-economist or someone with no prior PBMA experience, to conceptualize of PBMA as a separate procedure to adopt or implement into their context. And those inexperienced with PBMA may also be under the impression that by adopting PBMA priority setting practice will then automatically become more in-line with the economic principles (e.g., opportunity cost, the margin 46 ) that underlie the approach (e.g., Dionne et al., 2009; Peacock et al., 2010). An expectation in this case is that a person’s desire or need to change priority setting practice directly correlates with their ability to adopt a new recommended practice. This aligns with the assumption in traditional economics that people 46 These economic principles are defined in a footnote in section 1.5.2.2.2 of the literature review. 122 know what they want and are capable of acting on these preferences (Simon, 1959). However, the field of behavioural economics challenges this assumption. Nussbaum and Sen (1993) indicate that people often do not possess the knowledge and/or the (will)power to make choices that reflect their true desires. In this study, participants expressed a clear desire for specific attributes of priority setting practice that align with the PBMA approach – for example, a proactive, future-based and rational approach (see Figure 3.1). However, many of the gaps between the attributes of their desired and actual priority setting practice remained even after working with PBMA for close to two years. In this case, despite PBMA transcending the knowledge and practice gaps, which participants indicated that PBMA did quite well, participants still lacked the power and capability to transform priority setting practice to be more in alignment with their desired practice. As such, participant power and capability to transform priority setting practice is a key element for participants and organizations to address when considering an approach like PBMA. In this thesis project, the participants were mostly middle managers, and issues regarding power and capability within the middle manager’s role might warrant specific attention. Middle managers’ roles and responsibilities in healthcare are growing, especially their responsibilities related to implementing healthcare innovations like PBMA (Birken, Lee, & Weiner, 2012). Therefore, specifically understanding middle managers’ power and capabilities is critical to help inform strategies to address limitations in these areas. An alternate way of conceptualizing of PBMA is that it is a way of thinking – “it does seem to get the thinking right” (Mitton & Donaldson, 2003b, p. 96) – or a set of techniques that can be used to transform current priority setting practices. In this respect, PBMA is not a distinct process or entity to adopt or implement in addition to current priority setting activities. For example, decision makers can use PBMA’s rational and evidence-informed process to collaboratively formulate questions to ask about and analyze options and to make sense of situations (Patton, 2011). In this way, PBMA supports a different way of thinking about resource scarcity and allocation rather than serving as an additional step in the budgeting process. This different perspective may not be that distinctive to the person making recommendations for change; however, for the end-user, the decision maker, transforming a current practice is different than adopting a new practice. When it comes to 123 supporting organizations with evolving priority setting practice by implementing PBMA, it may be that using approaches founded on counseling 47 (i.e., individual and organizational behaviour change) rather than on educating may be more suitable; i.e., finding techniques to address the gap between desired and current practice by focusing on individual and group power, capacity, and readiness in addition to knowledge. This is an area, much like the field of knowledge translation is struggling with, where further research is required: for example, “…despite 30 years of research in this area, we still lack a robust, generalisable evidence base to inform decisions about strategies to promote the introduction of guidelines or other evidence-based messages into practice” (Grimshaw et al., 2004, p. 66). Another consequence of conceptualizing of PBMA as a distinct entity to be implemented or adopted is that it then seems like it is ‘all or none’ when it comes to adopting PBMA. Many examples in the literature discuss how PBMA’s seven stages can be and are adopted or implemented (e.g., Dionne et al., 2009; Peacock et al., 2010; Urquhart et al., 2008). Seldom do authors discuss how each component of PBMA was used to obtain specific impacts in the setting or link certain features of PBMA to specific outcomes. Arguments have been made in the PBMA literature about whether or not the program budgeting component of PBMA is required (Mitton & Donaldson, 2004c); however, the remaining components of PBMA are still discussed as a distinct process to implement whether the program budget is prepared as recommended in PBMA or not. In this study, participants indicated a preference and ability to adopt or transform their practice to align with some parts of PBMA (e.g., collaborative approach, development and use of criteria to inform decisions) but not other parts (e.g., developing the program budget and proposals for redesign and reallocation). Contextual factors, like the change in community care leader and perceived lack of leadership commitment to the process, were found to complicate PBMA implementation and priority setting in general in this study. This is not surprising: a supportive organizational culture and leadership that demonstrates commitment are key facilitators for successful healthcare innovation (Suter, Oelke, Adair, & Armitage, 2009). As a result of these complications, PBMA as a seven-stage approach was not fully implemented; however, it did transform 47 For example, clinical counseling tools like Motivational Interviewing (Miller & Rollnick, 2002) and related theories like the Transtheoretical Model for Behaviour Change (Prochaska, DiClemente, & Norcross, 1992). 124 priority setting practice for some participants to be more collaborative, rational and to address resource scarcity in a different way. Using the broader definition of impact developed in this study, future studies can be conducted to investigate the specific impacts related to different features of PBMA. This type of information is crucial to understand the mechanisms that influence adoption of healthcare innovations like PBMA. “When QI [quality improvement] initiatives [or innovations like PBMA] are implemented without a proper understanding of what they involve and how they work, they…risk becoming distorted imitations that succeed only in reproducing the superficial appearance but not the mechanisms (or set of mechanisms) that produce the outcomes in the first instance” (Dixon- Woods et al., 2011, pp. 169-170). A related concept is how PBMA is often described in terms of successful adoption or successful use by decision makers to obtain the specific outcome of setting priorities for resource reallocation (e.g., Dionne et al., 2009; Peacock et al., 2010). This can lead the reader to believe that priorities are set either using PBMA or not. In this study, participants had varying perspectives on whether resources were reallocated as a result of PBMA. However, as mentioned above, priority setting practice for some participants was transformed to be more collaborative, rational and evidence-informed during this study. Therefore, although PBMA was not successfully adopted (i.e., the seven-step approach was not fully implemented) and not all participants believed it resulted in resource reallocation, it did have a positive impact in this setting. Other research reports have also indicated both positive and negative outcomes from using PBMA within the same study (e.g., Bohmer et al., 2001; Urquhart et al., 2008). What was also found in this thesis project was that PBMA can be used in conjunction with other less rational decision making techniques (i.e., intuitive and political approaches to decision making) and that, at least in this context, this is what decision makers may find more useful. These participants indicated a preference, ability and capacity to transform part of their priority setting practice rather than change their whole practice. Research on barriers and facilitators to adoption of healthcare innovations has found this type of incremental change to be important to successful adoption (Barnett et al., 2011). And research in the area of behavioural economics and decision sciences indicates a role for both intuitive and rational decision making approaches (Kahneman, 2003). Future research is 125 required to explore how rational and ad hoc/historical approaches to priority setting can be combined or used together to inform priority setting practice. This research should explore if and how different decisions and contextual influences, including time, require different decision making techniques. Re-conceptualizing PBMA as a way of thinking or set of techniques to support decision makers in transforming their priority setting practice requires a different way of describing it to potential users. When PBMA is promoted as a seven-step process to be implemented in a context or adopted by decision makers, it emphasizes PBMA as a distinct ‘add-on’ process, and one that requires effort on the part of decision makers that is in addition to their current responsibilities. Several participants in this study had this impression. In implementation science (Greenhalgh et al., 2005), this in addition or extra work requirement is sometimes described as a barrier to change. Indeed, this perspective was found to be one barrier to PBMA implementation in this thesis project. If PBMA is conceptualized as a set of tools and, in particular, as a way of thinking to develop decision maker priority setting knowledge and practice and support system-level resource allocation and priority setting, it can potentially be seen as replacing and/or transforming, rather than adding on to, existing priority setting activities that do not support an evidence-informed, collaborative, rational and structured process. A recent publication was promoted with the (perhaps misleading but still captivating) tag line ‘Bye Bye to Budgets’ (Minich-Pourshadi, 2012). The article still discussed the requirement for decision makers to manage healthcare finances. However, the emphasis in this article was on how decision makers could transform their budgeting practices to support their resource allocation decision making rather than to support the budgeting process. In this way, some time-consuming, resource-intensive yet redundant or ineffective budgeting activities would be replaced with activities that aligned with and promoted the new approach. Again, the difference is not glaring – it is more about conceptualization – however, to busy healthcare decision makers, perception is critical when it comes to deciding about adopting new practices or transforming old ones. In summary, this thesis project explored PBMA effectiveness and impact, and described and evaluated its implementation. In the process, using iterative cycles of plan-act-observe-reflect 126 that are inherent in action research (Lewin, 1948), the broader decision making and implementation science literatures were examined to help inform the study as it progressed. These fields introduced concepts like decision heuristics and intuitive decision making approaches, and facilitators and barriers to successful healthcare innovation in general, which helped in explaining and interpreting the findings (Sandelowski, 1998). PBMA as commonly conceptualized (i.e., a distinct, seven-step, economic approach to priority setting) may not provide the full picture of how PBMA can support priority setting. It may also limit how users view the potential beneficial impacts attainable when using PBMA to transform priority setting practice. PBMA is not an entity, distinct from the context, to be implemented. Based on this thesis project, PBMA may be more desirable, acceptable and usable to potential users (i.e., a better fit in the context) if it is conceptualized as a way of thinking or a set of techniques that can be used to transform current priority setting practice. In this way, it can be used by decision makers to transform priority setting practice to be more collaborative, evidence-informed, and rational, and to better address economic and ethical principles important in priority setting. 6.2 Overall significance/contribution and potential applications of this research The need to control spiraling healthcare costs is well-documented and multiple strategies to manage healthcare expenditures are discussed in the literature (Berwick & Hackbarth, 2012). Using a systematic, comprehensive, and collaborative approach to set priorities to allocate resources to the most effective services is an appealing strategy to not only help control costs but also to help decision makers maximize value from limited resources. However, as outlined in the literature review, there are several challenges with priority setting that require attention as healthcare organizations and decision makers continue to develop their priority setting practices. This thesis project, by exploring the effectiveness, implementation and impact of one structured and evidence-informed priority setting approach, helps to provide insight into several of the priority setting challenges identified in the literature review (see section 1.5.3.1, page 19), specifically: How to manage the politics 127 Lack of understanding regarding the critical success factors of health system readiness to participate in effective priority setting, including the change management strategies required to facilitate priority setting; and, Limited strategies and awareness of how to effectively engage clinicians (in this case, clinician middle managers) in the process. Based on the findings of this thesis project, PBMA may be more appropriately conceptualized as a way of thinking or a set of techniques that can be used to transform current priority setting practice. Its overall impact can be determined by assessing the system-level impacts on priority setting practice and resource allocation, and also the individual and group impacts on priority setting practice and knowledge. Regarding how PBMA is conceptualized, in this study several recommendations emerged that can help to conceptualize PBMA as a way of thinking and as a set of tools to develop and/or transform decision maker priority setting knowledge and practice. First, in order for decision makers in a specific context to be able to transform their priority setting practice, the context and the participants must be ready for change and must have the capacity for change. Second, if PBMA was comprised of independently useful stages and incremental adoption of its rational approach was encouraged, its adaptability would be increased making it easier for decision makers to use it to transform practice. 6.2.1 Contextual readiness and capacity assessment stage A contextual readiness and capacity assessment stage can be used by participants to identify potential barriers to PBMA and then to help participants develop strategies to overcome these barriers. This will help to ensure that all participants have the requisite knowledge and capabilities to implement PBMA. For complex and multifaceted healthcare innovations like PBMA, organizational receptiveness, in particular preparing an organization both structurally and functionally to receive an innovation, has been found to facilitate adoption (Barnett et al., 2011). Regarding PBMA, participants can focus on barriers to the collaborative, evidence- informed, rational and structured approach inherent in PBMA. When the context and participants act on and make commitments to ensure that adequate resources are available, the culture is conducive, participants have the required power and capacity, and the guiding 128 vision and project scope are clear and supportive of PBMA, then these features will enhance PBMA’s overall fit (desirability, acceptability and usability) in the context. PBMA fit is important to focus on as, in this study, it was found to be a key factor in determining PBMA adoption and subsequent effectiveness. In particular, a contextual readiness and capacity assessment stage can be used to help ensure that sufficient information and time are available to compare the options; this is critical in rational decision making. Herbert Simon described the ‘bounded rationality’ of humans when it comes to decision making, indicating that we do not have unlimited computational capabilities to compare all the options (Simon, 1972). However, if participants want to become more rational in their decision making or some aspects of it, then the intermediary elements (e.g., time, data, ability, and process) to a rational approach must be in place first. This finding – adding a contextual readiness and capacity assessment stage to PBMA – helps address the second priority setting challenge identified above, that of the lack of understanding regarding the critical success factors of health system readiness to participate in effective priority setting, including the change management strategies required to facilitate priority setting. 6.2.2 Incremental adoption of PBMA’s rational approach As found in this study, PBMA’s rational approach must be reconciled with the prevailing decision making approaches, i.e., the ad hoc, intuitive, and political decision making approaches. In some ways, this can be viewed as ensuring a fit between PBMA and the organizational culture which, in this study, appeared to be more conducive to ad hoc, intuitive and political approaches to decision making. This reconciliation is more readily accomplished when PBMA is conceived of as a way of thinking and/or a set of related tools or techniques which can be used to incrementally transform priority setting practice as participants, within their context, are ready and willing. More detailed studies on PBMA use are needed to determine facilitators that can be emulated in other settings. For example, more studies are needed to determine which situations and features of a context facilitate collaborative, structured and comprehensive problem and solution identification in complex 129 organizations. By breaking PBMA down into its various elements – e.g., program budgeting, marginal analysis, collaboration, evidence-informed, structured, comprehensive, and rational process – specific attention can be paid to how each of these elements can be used to transform priority setting practice. Participants can focus on those elements that resonate for them at a particular time. It may be that an incremental approach enables decision makers to move past some of the limitations to rational decision making by enabling them to blend some parts of a rational approach (e.g., structure, evidence-informed, explicitly rate options) with current political and intuitive approaches (i.e., those based on intuitive judgments or expert opinion and/or political mandates) to combine both types of decision making into one priority setting approach. Or it may be that an incremental approach enables decision makers to allow their intuitive decision making (and political influences) to inform the rational approach. This interaction between intuitive and deliberative or rational approaches to decision making is explored in detail by cognitive psychologists, e.g., the dual-process theory and two-mind hypothesis (Evans, 2007), and behavioural economists (e.g., Kahneman, 2003; Kahneman, 2011). Further research, perhaps using techniques from the naturalistic decision making field like cognitive task analysis 48, is required to “demystify intuition by identifying the cues that experts use to make their judgments, even if those cues involve tacit knowledge and are difficult for the expert to articulate” (Kahneman & Klein, 2009, p. 516). Once these cues are identified, research into how these cues can inform the rational decision making in PBMA can be conducted. In this way, the expert opinion of decision makers, as revealed and clarified through the process of identifying these cues, can be shared more broadly with other, less-experienced, decision makers. The broader definition of impact developed in this study aligns with incremental adoption of PBMA’s rational approach. This broader definition of impact highlights individual and group priority setting practice and knowledge changes as important impacts to seek when transforming priority setting practice with PBMA. For example, individual decision makers can transform specific aspects of their priority setting practice to become more rational. Individual impact, as found in this study, then facilitates group priority setting practice 48 Cognitive task analysis methods include interview techniques that elicit cues and contextual considerations that influence the judgments and decisions made by expert decision makers (Kahneman & Klein, 2009). 130 change. The broader definition of impact supports conceptualizing PBMA as both a priority setting approach and as a tool to develop individual and group priority setting knowledge and practice. This finding – the incremental adoption of PBMA’s rational approach – attends to the first priority setting challenge identified above, specifically how to manage the politics by incorporating political (and intuitive) approaches to decision making into rational approaches. 6.2.3 Develop each PBMA stage to be independently useful Another finding in this study, related to the incremental adoption of PBMA’s rational approach, was to develop each PBMA stage to be useful independent of the other PBMA stages. If each of the seven stages of PBMA is functional and useful independent of the other stages, then implementation can be adapted to the organization’s culture and the decision maker users’ priority setting practice and needs, rather than trying to make the culture fit PBMA. For example, an organization and its decision makers can choose to focus on developing those features in PBMA (e.g., the program budget, collaborative problem and solution identification, criteria development) that they are capable of and open to addressing at the time. This can serve to both incrementally change priority setting practice to be in better alignment with the PBMA approach and to ready the context and decision makers to transform other elements of their priority setting practice. Independently useful PBMA stages may also be operationally easier to implement when recognized barriers to implementation exist. Previous PBMA research has identified barriers and facilitators to PBMA implementation (Mitton & Donaldson, 2003a). Some barriers, in particular leadership changes or unsupportive leaders (Peacock et al., 2010), have been described as indicators that PBMA implementation is unlikely to be successful. In other words, leadership change and unsupportive leaders can be seen as insurmountable obstacles to PBMA implementation. Indeed, research into barriers and facilitators to healthcare innovation implementation has found that supportive top management is critical to successful implementation and that people within and outside of an organization can act in ways that 131 either facilitate or inhibit innovation in an organization (Barnett et al., 2011). However, since leadership changes are common in healthcare, it is helpful to potential PBMA users to develop tactics to implement PBMA alongside leadership changes. When leadership changes occur, PBMA buy-in from the new leader would still be required; however, the full group participating in PBMA would not necessarily have to return to earlier stages if the stages are self-contained. As found in this study, this increases PBMA usability and thereby fit. This finding – developing each PBMA stage to be useful independent of the other PBMA stages – addresses the third priority setting challenge identified above, specifically how to effectively engage clinicians (in this case, clinician middle managers) in the process by tailoring the PBMA approach and focusing on incremental adoption. In summary, the key findings of this thesis work – the addition of a contextual readiness and capacity assessment stage to PBMA, incremental adoption of PBMA’s rational approach, and adapting PBMA to have functionally independent stages – address challenges identified in the priority setting literature. These challenges are important to overcome if approaches like PBMA are to be successfully implemented to transform priority setting practice. This transformed practice can then support decision makers in setting priorities to allocate resources to the most effective services in an effort to control spiraling healthcare costs. With a growing emphasis on evidence-based, cost-effective and accountable healthcare (Graham et al., 2006), priority setting practice must continue to evolve to both help inform decisions that meet these attributes (evidence-based, cost-effective and accountable) and result in healthcare that does as well. 6.3 Conclusions based on the research objectives As outlined in the introduction chapter, the objectives in this study were to: 1. Explore PBMA effectiveness (i.e., whether it does actually work) in a community care context, and then determine if and how PBMA can be adapted to make it more effective in this context. 132 2. Describe and evaluate the PBMA implementation process, and describe the experience and lessons learned, to gain insight into the role of implementation in changing priority setting practice. 3. Study the impact on the participants and the context of implementing a formal priority setting approach, using PBMA as an example. Impact is interpreted in light of the estimated implementation cost, i.e., what did these decision makers get for their time spent participating in PBMA? To address these objectives, a multi-disciplinary group of community care decision makers was supported in implementing PBMA to inform resource allocation decisions for two consecutive budget years. The conclusions of this research (that addressed these research objectives) are summarized in three sections below – PBMA effectiveness, implementation and impact. PBMA effectiveness: Effectiveness was explored by examining PBMA’s contextual fit (using the dimensions of desirability, acceptability, and usability) from the decision makers’ perspectives. Although PBMA was discontinued early, prior to the planned year-two conclusion, parts of the PBMA approach – the collaborative, evidence-informed approach, collaboratively defining and then using criteria to inform decisions – were deemed effective for priority setting in this community care context. As such, PBMA was conceived of and used as a set of tools and as a way of thinking to help these decision makers transform their priority setting practice to be in better alignment with their desired features of a priority setting process. The findings from this study also indicated that PBMA’s effectiveness could be improved by adding a contextual readiness and capacity assessment stage (and then readying the individuals and context based on the results of the assessment), appreciating organizational complexity, and reconciling PBMA’s rational approach with prevailing decision making processes through incremental implementation. 133 PBMA implementation: Desire for more clarity, and for PBMA implementation and approach adaptations, emerged as overarching themes in our evaluation of PBMA implementation in this context. Participants desired a clearer understanding of what their roles were and how PBMA and its tools should be used to achieve PBMA’s potential benefits. They argued that each PBMA stage should be useful independent of the other stages so that implementation could be adapted to the organization’s culture, rather than trying to make the culture fit PBMA. To help improve clarity and ensure that the resources and technical and human factors were available to support PBMA, participants also stated that an additional stage that incorporated an organizational readiness and capacity assessment was required. Concepts related to these themes emerged early in implementation; however, the themes became clearer after the change in community care director (toward end of first PBMA cycle) and were clarified further at the end of the project (second PBMA cycle). PBMA impact: Impact was studied by examining whether decision makers considered PBMA useful for priority setting in that it changed policy or priority setting practice (i.e., was used to set specific priorities or to allocate resources), and if and how PBMA influenced participant knowledge, understanding of and/or attitudes about priority setting (Meagher, Lyall, & Nutley, 2008). Positive impact was found on a continuum that included intended and desired effects on priority setting as determined by priorities set: participants prioritized a list of $760,000 worth of investment proposals and $38,000 of disinvestment proposals. However, whether or not resources were reallocated accordingly was interpreted differently by different participants. Impact also included lesser-emphasized and seldom formally acknowledged influences on priority setting knowledge, attitudes and practice which appear to be both essential to and effects of the desired outcomes. PBMA impacts were interrelated and found to varying degrees, both within (regarding different aspects of impact) and between participants, at the individual decision maker level. 134 6.4 Limitations and generalizability (transferability) By using action research to explore PBMA effectiveness and impact, and to describe and evaluate the implementation process in real-time, we were able to capture the lessons learned through this exploratory and evaluative fieldwork for the purposes of sharing with others considering or already using PBMA. These types of studies are needed to describe and help explain how programs like PBMA work, which is important knowledge for future PBMA users (Dixon-Woods et al., 2011). Action research is well-suited to this type of inquiry. However, there are several limitations to this study that relate to conducting applied research in a natural setting. Naturalistic inquiry was necessary to explore PBMA effectiveness (i.e., whether PBMA does actually work given the various contextual factors that can complicate priority setting) because it enabled real-world experiences endemic to the context to influence PBMA implementation and impact. Due to the dynamic nature of healthcare, change is inevitable. During this study, changing personnel and directives, such as new chronic disease management and primary care initiatives that directly impacted community care, complicated the study of PBMA impact and the evaluation of PBMA implementation. These changes slowed PBMA implementation, as they required additional education and re- confirmation of project aim and scope, and resulted in the eventual discontinuation of the project before the end of the second implementation and therefore before system-level impact could be fully realized. As such, using a broad definition of impact – i.e., individual and group-level impact via changes to priority setting practice, knowledge and/or attitudes, and system-level impact via changes to priority setting and resource allocation – enabled a more comprehensive assessment of PBMA impact in this setting. However, effects, such as PBMA’s influence on knowledge, attitudes and priority setting practice, may evolve over time and beyond study completion. Therefore, this study did not and cannot capture all of the long term effects of implementing PBMA. It can only report those impacts realized by the study completion. Working through the challenges that resulted from these contextual changes did however provide valuable information to inform both the study of PBMA effectiveness and how priority setting impact could be determined. Also, addressing organizational change enhances the real-world applicability of these results, which may enhance the transferability (Lincoln & Guba, 1985) of these findings to other healthcare 135 organizations that may also experience contextual changes that affect implementation and adoption of rational practices. Two specific limitations require addressing here. 6.4.1 Leadership change The change in community care leader part-way through year-one resulted in uncertainty about leadership support for and commitment to this project. This made implementation more challenging and may have complicated the study of both PBMA impact and the PBMA implementation evaluation, threatening internal validity. This leadership change also contributed to a long time-span between the initial education session provided by researchers (before PBMA was implemented) and the final stages of PBMA implementation, and this delay may have been one contributing factor to the lack of clarity (as broadly defined in this study) found in this study and discussed in Chapter 4. However, leadership changes are common in healthcare organizations and therefore, working through this change strengthens the PBMA effectiveness and impact findings in this changing context. Future research into how to address leadership change is needed to develop strategies to facilitate PBMA implementation when change occurs. 6.4.2 Education before pre-PBMA data collection The initial educational session (see methods section) occurred before the baseline (pre- PBMA) data were collected. This may have influenced the responses (regarding desired priority setting practice) given by those participants who attended the session: i.e., they may have been biased towards saying that the desired process ought to be a rational one. In some ways, however, this may have helped highlight those issues regarding PBMA fit that required further investigation. 6.4.3 Transferability As mentioned in the methodology chapter, by using action research methodology to address the research objective of exploring PBMA effectiveness and impact in this particular community care context, external validity (generalizability or transferability) was traded, to some degree, for internal validity or credibility (Lincoln & Guba, 1985). Researchers and decision makers worked together in this study to implement PBMA to both inform this 136 particular group of decision makers’ priority setting practice and to explore PBMA effectiveness and impact, and evaluate PBMA implementation. As such, the findings and resulting knowledge claims are grounded in the context in which they were explored. The assumptions – i.e., that the population under study can be representative of another population, and that generalizations made can be independent of time and context – required to make generalizability claims do not align with the philosophical underpinnings of naturalistic inquiry (Lincoln & Guba, 1985). As such, the term transferability is sometimes used instead to describe the extent to which findings from a naturalistic study may be applicable to another setting. Transferability, or extrapolation “of findings from one specific case to another is possible” (Schwandt, 2007). Lincoln and Guba state that this transferability depends on the “degree of similarity between the sending and receiving contexts” (1985, p. 297) which requires clear documentation and ‘thick description’ of the study so that readers can make the determination of ‘similarity’ and thereby transferability to the receiving context. In this study, prolonged engagement in the context enabled ongoing observation and documentation of contextual influences on priority setting practice. And, in this thesis, I have included detailed description of these contextual influences, and the context and findings, so that readers can make the similarity determination. Furthermore, as mentioned above, the change in leader and other changes that occurred during the course of this study are common to other healthcare settings and this also increases the transferability of these findings to other contexts. 6.5 Potential future research directions emerging from the thesis research Several areas for future research have been described throughout the thesis, primarily in the discussion sections of the three results chapters (chapters 3, 4 and 5) and the earlier sections (6.1 and 6.2) of this concluding chapter. Table 6.1 below summarizes these areas for future research. Following this, one specific area for future research that stems directly from the findings of this thesis project is described. This includes using implementation science and diffusion of innovation concepts and methods to develop and evaluate a pre-PBMA contextual readiness and capacity assessment stage, and to develop and evaluate various PBMA approach adaptations to facilitate incremental adoption of PBMA. 137 Table 6.1 Summary of future research areas Thesis section Future research area 1.5.4.3. Barriers and facilitators to using PBMA Develop and explore strategies that decision makers can use to implement PBMA to transform priority setting practice when the context has unsupportive leaders or is otherwise unfavorable. 3.3.3 Restrictions to adopting a rational approach Using methods from cognitive psychology, decision science and behavioural economics, investigate the different decision making approaches used (e.g., intuitive, rational, political) at different times and in different situations. 4.3 Implementation discussion Based on the finding in this study – if each PBMA stage was functionally independent of the other stages, participants could have implemented those stages that resonated for them first and evolved their priority setting practice accordingly – examine if (and if so, how) this increases individual and organizational readiness to further evolve priority setting practice by implementing the other PBMA stages. 4.4 Implementation summary Explore various PBMA adaptations, including to the education and communication components, to determine which adaptations (in which contexts) are most effective in helping decision makers transform priority setting practice. 5.3 Impact discussion Using techniques from adult education and other relevant fields, refine and investigate the education shared during PBMA to address individual participants’ learning needs to evolve all participants to an equivalent level of understanding so that the group can develop their practice together. This supports conceptualizing PBMA not just as a tool to support resource reallocation but also as a means to develop decision maker priority setting practice and knowledge. 5.3 Impact discussion Examine manager learning (that results from PBMA) and application of that learning in routine budgeting and strategic planning activities outside of priority setting. This may have more impact on future resource reallocations than will the actual PBMA exercise in which the learning took place. It may also influence the primary measure(s) of impact resulting from formal priority setting activities. 138 Thesis section Future research area 5.3 Impact discussion Based on the finding in this study – that individual and group-level impacts were interrelated with each other and with system-level impact, and varied between and within individuals regarding different aspects of PBMA – explore this connection between individual, group and system-level impacts. Attending to these variances between and within individuals during PBMA implementation is prudent in the interest of achieving intended system-level PBMA impact. This study provides preliminary evidence that individual and group-level change are preconditions for system-level outcomes (as presented in the impact chapter); however, further research is needed to examine the various relationships between individual, group and system-level impacts. 5.3 Impact discussion Study if and how PBMA can help set the stage for economic techniques to be more effective; for e.g., this may require specific tools and targeted educational approaches depending on different participants’ connections with and understandings of various elements of PBMA, particularly economic considerations and techniques, and PBMA’s rational approach. 6.1 Overall analysis and integration of fit, implementation and impact chapters with priority setting research Using the broader definition of impact developed in this study, investigate the specific individual, group and system-level impacts related to the different features of PBMA. 6.1 Overall analysis and integration of fit, implementation and impact chapters with priority setting research Explore how rational and ad hoc/historical approaches to priority setting can be combined or used together to inform priority setting practice. This research should explore if and how different decisions and contextual influences, including time, require different decision making techniques. 6.1 Overall analysis and integration of fit, implementation and impact chapters with priority setting research Develop and evaluate approaches to implement PBMA that are based on counseling (i.e., individual and organizational behaviour change) rather than on educating; i.e., develop techniques to address the gap between desired and current practice by focusing on individual and group power, capacity, and readiness in addition to knowledge. 139 Thesis section Future research area 6.4.1 Leadership change Explore how to address leadership change during PBMA implementation and then develop strategies to facilitate PBMA implementation when change does occur. 6.2.2 Incremental adoption of PBMA’s rational approach Using techniques from the naturalistic decision making field, like cognitive task analysis 49 , identify “the cues that [priority setting] experts use to make their [resource allocation] judgments, even if those cues involve tacit knowledge and are difficult for the expert to articulate” (Kahneman & Klein, 2009, p. 516). Once these cues are identified, explore how they can inform the rational decision making in PBMA. In this way, the expert opinion of decision makers, as revealed and clarified through the process of identifying these cues, can be shared more broadly with other, less-experienced, decision makers. 6.5.1 Develop and test PBMA adaptations recommended in this thesis It is the “interaction among the innovation [e.g., PBMA], intended adopter(s) and a particular context that determines the adoption rate” (Greenhalgh et al., 2004, p. 598). Thus, tailoring implementation strategies to relevant barriers and facilitators found within the target context appears to be an efficient approach to implementation (Logan & Graham, 1998). In this way, concepts and methods from the field of implementation science can be used to further develop the PBMA implementation approach. Since PBMA implementation is not a simple product-based innovation adoption, and the unit of adoption is not only the individual, diffusion is not simply a matter of imitation. PBMA is a complex, process-based innovation with intended adoption at an organizational, departmental and/or team level. Thus, using a comprehensive approach to research and develop its implementation has broad implications. For example, determining how and the extent to which an innovation like PBMA is used is one way that researchers and decision makers can assess the success of the implementation strategies used in order to share them with others. Also, monitoring implementation and adoption enables investigators to determine if PBMA is being used as designed/intended or if it is being adapted or abandoned. This is valuable for future PBMA research, and to inform PBMA adaptations, to ensure contextual relevance is maintained as the healthcare system continues to evolve. It also enables delineation and validation of the implementation process 49 Cognitive task analysis methods include interview techniques that elicit cues and contextual considerations that influence the judgments and decisions made by expert decision makers (Kahneman & Klein, 2009). 140 so that future implementers/adopters of PBMA can feel more comfortable with the development of the implementation strategies and thereby accept and use them more readily. This thesis project has evaluated PBMA implementation in this community care context and has highlighted where further study of the implementation process is required. Current PBMA implementation is focused on planned action 50 activities, such as forming an advisory panel and working with this group to collectively develop the program budget, relevant criteria, scoring tool and proposal template, with researchers often supporting this group in developing, rating and ranking proposals for service redesign. What is missing from the current PBMA implementation process is a built-in barriers assessment with action planning to develop strategies to overcome these barriers, including those at varying hierarchical levels. This tailored implementation design process is often a weakness (outside of PBMA), even in implementation research, since most interventions are solution-driven versus needs-driven (Vedel et al., 2009). Intervention approaches also requires specific steps designed to describe the relevant change(s) required and identify determinants of these change(s). To develop a built-in barriers assessment and related action plan, a diffusion of innovation framework such as the Greenhalgh Diffusion of Innovation Model (Greenhalgh et al., 2005) is useful 51 . The Greenhalgh Model can be used to develop and evaluate the pre-PBMA contextual readiness and capacity stage and various PBMA approach adaptations as described in this thesis that may facilitate incremental adoption of PBMA’s rational approach. Using implementation science in a focused way – as by using the Greenhalgh Model to plan, evaluate and validate the PBMA implementation process – will have several implications for priority setting research in general. It expands the focus away from development of priority setting tools and approaches to including a stronger emphasis on contextual relevance by also focusing on innovation-system fit and potential adopters’ perspectives. Even if changing 50 Planned action or change theories are primarily predictive and/or prescriptive, and thereby are useful for changing behaviour (Michie et al., 2005; Tiffany & Johnson, 1998). 51 See Appendix B for details about this model. 141 broader outer context issues (e.g., regulations and politics that impact innovation-system fit) is not within the domain of priority setting activities, identifying their presence is still important when planning change (Grol et al., 2007). 6.6 Summary Scarce healthcare resources require that choices be made about which services to fund. PBMA is an evidence-informed and rational approach that can assist decision makers in choosing between the various options. However, as found in this thesis project, contextual factors can complicate priority setting, which can hamper PBMA adoption and effectiveness. The findings from this study suggest that adding a contextual readiness and capacity assessment stage to PBMA, recognizing organizational complexity, and considering incremental adoption of PBMA’s approach may help to improve PBMA’s effectiveness in some contexts. These tactics are suggested to more closely align PBMA with real-world priority setting practice, which may facilitate greater adoption, especially in contexts experiencing difficulty using PBMA. These findings contribute to a growing body of literature on PBMA use in various healthcare settings, and help us to better understand and work with priority setting conditions to advance evidence-informed decision making. Ultimately, healthcare decision makers are faced with making difficult decisions to concurrently address growing client care demands and resource limitations. The goal of this work is to support decision makers in this activity. 142 References Ambrose, D. (1987). Managing complex change. Pittsburgh: The Enterprise Group Ltd. Arminio, J. L., & Hultgren, F. H. (2002). Breaking out from the shadow: the question of criteria in qualitative research. Journal of College Student Development, 43(4), 446- 456. Babbie, E., & Benaquisto, L. (2002). Fundamentals of social research (1st Canadian ed.). Scarborough: Nelson, a division of Thomson Canada Ltd. Baker, G. R., Ginsburg, L., & Langley, A. (2004). An organizational science perspective on information, knowledge, evidence, and organizational decision-making. In L. Lemieux-Charles & F. Champagne (Eds.), Using knowledge and evidence in health care: multidisciplinary perspectives (pp. 86-114). Toronto: University of Toronto Incorporated. Barnett, J., Vasileiou, K., Djemil, F., Brooks, L., & Young, T. (2011). Understanding innovators' experiences of barriers and facilitators in implementation and diffusion of healthcare service innovations: a qualitative study. BMC Health Services Research, 11(1), 342. Berwick, D. M., & Hackbarth, A. D. (2012). Eliminating Waste in US Health Care. JAMA: The Journal of the American Medical Association. doi: 10.1001/jama.2012.362 Bevan, H., Ham, C., & Plsek, P. (2008). The next leg of the journey: How do we make high quality care for all a reality? (pp. 1-9). Coventry: NHS Institute for Innovation and Improvement. Birken, S., Lee, S.-Y. D., & Weiner, B. (2012). Uncovering middle managers' role in healthcare innovation implementation. Implementation Science, 7(1), 28. Bodenheimer, T. (1997). The Oregon Health Plan--lessons for the nation. First of two parts. The New England Journal Of Medicine, 337(9), 651-655. Bohmer, P., Pain, C., Watt, A., Abernethy, P., & Sceats, J. (2001). Maximising health gain within available resources in the New Zealand public health system. Health Policy, 55(1), 37-50. Brambleby, P. (1995). A survivor's guide to programme budgeting. Health Policy, 33(2), 127-145. Bullen, G., & Sacks, L. (2003). Towards new modes of decision making - complexity and human factors. (1a). Retrieved from Carter, S. M., & Little, M. (2007). Justifying Knowledge, Justifying Method, Taking Action: Epistemologies, Methodologies, and Methods in Qualitative Research. Qualitative Health Research, 17(10), 1316-1328. doi: 10.1177/1049732307306927 Cooley, M. (2007). From judgment to calculation. AI & Society, 21(4), 395-409. doi: 10.1007/s00146-007-0106-5 Coulter, A., & Ham, C. (Eds.). (2000). The global challenge of health care rationing. Buckingham: Open University Press. Craig, N., Parkin, D., & Gerard, K. (1995). Clearing the fog on the Tyne: programme budgeting in Newcastle and North Tyneside Health Authority. Health Policy, 33(2), 107-125. Creswell, J. W. (2007). Qualitative inquiry and research design: choosing among five approaches (Second ed.). Thousand Oaks, CA: Sage Publications, Inc. 143 Daniels, N. (2000). Accountability for reasonableness in private and public health insurance. In A. Coulter & C. Ham (Eds.), The global challenge of health care rationing (pp. 89- 106). Buckingham: Open University Press. Daniels, N., & Sabin, J. (1998). The ethics of accountability in managed care reform. Health Aff, 17(5), 50-64. doi: 10.1377/hlthaff.17.5.50 Daniels, N., & Sabin, J. (2002). Setting limits fairly: can we learn to share medical resources? Oxford, England: Oxford University Press. Dionne, F., Mitton, C., Smith, N., & Donaldson, C. (2008). Decision-maker views on priority setting in the Vancouver Island Health Authority. Cost Effectiveness and Resource Allocation, 6(13). doi: 10.1186/1478-7547-6-13 Dionne, F., Mitton, C., Smith, N., & Donaldson, C. (2009). Evaluation of the impact of program budgeting and marginal analysis in Vancouver Island Health Authority. Journal Of Health Services Research & Policy, 14(4), 234-242. Dixon-Woods, M., Bosk, C. L., Aveling, E. L., Goeschel, C. A., & Pronovost, P. J. (2011). Explaining Michigan: developing an ex post theory of a quality improvement program. [Research Support, Non-U.S. Gov't]. Milbank Q, 89(2), 167-205. doi: 10.1111/j.1468-0009.2011.00625.x Dixon, J., & Welch, H. G. (1991). Priority setting: lessons from Oregon. The Lancet, 337(8746), 891-894. Donaldson, C., Bate, A., Mitton, C., Peacock, S., & Ruta, D. (2007). Priority setting in the public sector: Turning economics into a management process. Managing Improvement in Public Service Delivery: Progress and Challenges. Donaldson, M. S., & Sox, H. (Eds.). (1992). Institute of Medicine: Setting priorities for health technology assessment. Washington: National Academy Press. Drummond, M. (1992). Cost-effectiveness guidelines for reimbursement of pharmaceuticals: Is economic evaluation ready for its enhanced status? Health Economics, 1, 85-92. Drummond, M., Cooke, J., & Walley, T. (1997). Economic evaluation under managed competition: Evidence from the U.K. Social Science & Medicine, 45(4), 583-595. Drummond, M. F., Sculpher, M. J., Torrance, G. W., O'Brien, B. J., & Stodart, G. L. (2005). Methods for the economic evaluation of health care programmes (Third ed.). Oxford: Oxford University Press. Elixhauser, A., Halpern, M., Schmier, J., & Luce, B. R. (1998). Health Care CBA and CEA from 1991 to 1996: An Updated Bibliography. Medical Care, 36(5), MS1-MS9. Evans, J. S. B. T. (2007). Hypothetical thinking: dual processes in reasoning and judgment. Hove, East Sussex, England: Psychology Press. Freemantle, N., & Mason, J. (1999). Not playing with a full DEC: why development and evaluation committee methods for appraising new drugs may be inadequate. BMJ, 318(7196), 1480-1482. Gibson, J. (2008). Presentation: Ethics and priority setting in healthcare. MSFHR Team Planning Meeting. Vancouver. Gibson, J., Martin, D., & Singer, P. (2004). Setting priorities in health care organisations: criteria, processes, and parameters of success. BMC Health Services Research, 4(25). doi: 10.1186/1472-6963-4-25 Gibson, J., Martin, D., & Singer, P. (2005). Evidence, economics and ethics: resource allocation in health services organizations. Healthcare Quarterly, 8(2), 50-59. 144 Gibson, J., Mitton, C., & DuBois-Wing, G. (2011). Priority setting in Ontario's LHINs: ethics and economics in action. Healthc Q, 14(4), 35-43. Gibson, J., Mitton, C., Martin, D., Donaldson, C., & Singer, P. (2006). Ethics and economics: does programme budgeting and marginal analysis contribute to fair priority setting? Journal of Health Services Research & Policy, 11(1), 32-37. Golden, B. (2006). Transforming healthcare organizations. Healthcare Quarterly, 10(Special issue), 10-19. Gore, A. (1990). Oregon's bold mistake. Academic Medicine, 65(11), 634-634. Graham, I. D., & Logan, J. (2004). Innovations in Knowledge Transfer and Continuity of Care. CJNR (Canadian Journal of Nursing Research), 36(2), 89-103. Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W., & Robinson, N. (2006). Lost in knowledge translation: time for a map? Journal of Continuing Education in the Health Professions, 26(1), 13-24. Graham, I. D., & Tetroe, J. (2007). Some Theoretical Underpinnings of Knowledge Translation. Academic Emergency Medicine, 14(11), 936-941. Greenhalgh, T. (2006). How to read a paper: the basics of evidence-based medicine (Third ed.). Malden, Massachusetts: Blackwell Publishing Ltd. Greenhalgh, T., Robert, G., Bate, P., Macfarlane, F., & Kyriakidou, O. (2005). Diffusion of innovations in health service organizations: A systematic literature review. Oxford: Blackwell Publishing Ltd. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. The Milbank Quarterly, 82(4), 581-629. Griffiths, S., Reynolds, J., & Hope, T. (2000). Priority setting in practice. In A. Coulter & C. Ham (Eds.), The global challenge of health care rationing. Buckingham, UK: Open University Press. Grimshaw, J., Thomas, R., MacLennan, G., Fraser, C., Ramsay, C., Vale, L., . . . Donaldson, C. (2004). Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technology Assessment, 8(6). Grol, R. P., Bosch, M. C., Hulscher, M. E., Eccles, M. P., & Wensing, M. (2007). Planning and studying improvement in patient care: the use of theoretical perspectives. The Milbank Quarterly, 85(1), 93-138. Guba, E. G., & Lincoln, Y. S. (1981). Effective evaluation: improving the usefulness of evaluation results through responsive and naturalistic approaches. San Francisco, CA: Jossey-Bass. Hagey, R. S. (1997). The use and abuse of participatory action research. Chronic Dis Can, 18(1), 1-4. Halma, L., Mitton, C., Donaldson, C., & West, B. (2004). Case study on priority setting in rural Southern Alberta: keeping the house from blowing in. Canadian Journal of Rural Medicine, 9(1), 26-36. Ham, C. (1993). Priority setting in the NHS: reports from six districts. BMJ (Clinical Research Ed.), 307(6901), 435-438. Ham, C. (1999). Tragic choices in health care: lessons from the Child B case. BMJ, 319(7219), 1258-1261. 145 Ham, C., & Coulter, A. (2000a). Conclusion: Where are we now? In A. Coulter & C. Ham (Eds.), The global challenge of health care rationing (pp. 233-250). Buckingham: Open University Press. Ham, C., & Coulter, A. (2000b). Introduction: International experience of rationing (or priority setting). In A. Coulter & C. Ham (Eds.), The global challenge of health care rationing (pp. 1-12). Buckingham: Open University Press. Hampshire, A. J. (2000). What is action research and can it promote change in primary care? Journal of Evaluation in Clinical Practice, 6(4), 337-343. Hart, E., & Bond, M. (1995). Action research for health and social care - a guide to practice. Buckingham: Open University Press. Herr, K., & Anderson, G. L. (2005). The action research dissertation: a guide for students and faculty. Thousand Oaks, California: Sage Publications, Inc. Hoffmann, C. (2000). The influence of economic evaluation studies on decision making.: A European survey. Health Policy, 52(3), 179-192. Holloway, I., & Todres, L. (2003). The Status of Method: Flexibility, Consistency and Coherence. Qualitative Research, 3(3), 345-357. doi: 10.1177/1468794103033004 Holm, S. (1998). Goodbye to the simple solutions: The second phase of priority setting in health care. BMJ: British Medical Journal, 317(7164), 1000-1002. Interior Health. (2007). Central Okanagan Local Health Area Profile. B.C.: Interior Health Information Support and Research. Jan, S. (2000). Institutional considerations in priority setting: transactions cost perspective on PBMA. Health Economics, 9, 631-641. Jefferson, T., Demicheli, V., & Mugford, M. (2000). Elementary Economic Evaluation in Health Care (Second ed.). London: BMJ Books. Kahneman, D. (2003). Maps of Bounded Rationality: Psychology for Behavioral Economics. American Economic Review, 93(5), 1449-1475. doi: doi: 10.1257/000282803322655392 Kahneman, D. (2011). Thinking, fast and slow. Canada: Doubleday Canada, a division of Random House of Canada Limited. Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: a failure to disagree. Am Psychol, 64(6), 515-526. doi: 10.1037/a0016755 Kaplan, A. (1964). Conduct of inquiry: methodology for behavioural science. San Francisco: Chandler. Kidd, S., & Kral, M. (2005). Practicing participatory action research. Journal of Counselling Psychology, 52(2), 187-195. Kitzhaber, J. A. (1993). Prioritising health services in an era of limits: the Oregon experience. BMJ (Clinical Research Ed.), 307(6900), 373-377. Klein, G. (1999). Sources of power: how people make decisions. Cambridge: The MIT Press. Klein, R. (1993). Dimensions of rationing: who should do what? BMJ (Clinical Research Ed.), 307(6899), 309-311. Klein, R., & Williams, A. (2000). Setting priorities: what is holding us back - inadequate information or inadequate institutions? In A. Coulter & C. Ham (Eds.), The global challenge of health care rationing (pp. 15-26). Buckingham: Open University Press. Kotter, J. P. (1995). Leading Change: Why Transformation Efforts Fail. (cover story). Harvard Business Review, 73(2), 59-67. Lewin, K. (1948). Resolving social conflicts. New York: Harper and Rowe. 146 Lincoln, Y., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage Publications. Lindblom, C. E. (1959). The Science of \"Muddling Through\". Public Administration Review, 19(2), 79-88. Logan, J., & Graham, I. D. (1998). Toward a comprehensive interdisciplinary model of health care research use. Science Communication, 20(2), 227-246. Lomas, J., Woods, J., & Veenstra, G. (1997). Devolving authority for health care in Canada's provinces: 1. An introduction to the issues. Canadian Medical Association Journal, 156(3), 371-377. Martin, A. (2008). Action research on a large scale: issues and practices. In P. Reason & H. Bradbury (Eds.), The SAGE handbook of action research: participative inquiry and practice (2nd ed., pp. 394-406). Los Angeles, CA: SAGE Publications. Martin, D., Pater, J., & Singer, P. (2001). Priority-setting decision for new cancer drugs: a qualitative study. Lancet, 358, 1676-1681. Martin, D. K., & Singer, P. (2000). Priority setting and health technology assessment: beyond evidence-based medicine and cost-effectiveness analysis. In A. Coulter & C. Ham (Eds.), The global challenge of heatlh care rationing (pp. 135-145). Buckingham: Open University Press. McDonald, R. (2002). Using health economics in health services. Rationing rationally? Buckingham: Open University Press. McKie, J., & Richardson, J. (2003). The Rule of Rescue. Social Science & Medicine, 56(12), 2407-2419. McNiff, J., & Whitehead, J. (2006). All you need to know about action research. Thousand Oaks, CA: SAGE Publications. Meagher, L., Lyall, C., & Nutley, S. (2008). Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research. Research Evaluation, 17(3), 163-173. doi: 10.3152/095820208x331720 Michie, S., Johnston, M., Abraham, C., Lawton, R., Parker, D., Walker, A., & on behalf of the \"Psychological Theory\" Group. (2005). Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care, 14(1), 26-33. doi: 10.1136/qshc.2004.011155 Miles, M. B., & Huberman, A. M. (1984). Qualitative data analysis: a sourcebook of new methods. Beverly Hills, CA: SAGE Publications. Miller, P., & Vale, L. (2001). Programme approach to managing informed commissioning. Health Services Management Research, 14, 159-164. Miller, W. R., & Rollnick, S. (2002). Motivational interviewing: preparing people for change (Second ed.). New York, NY: The Guilford Press. Minich-Pourshadi, K. (2012). Byb-bye to annual budgeting? Retrieved April 26, 2012, from http://www.healthleadersmedia.com/page-1/FIN-278180/ByeBye-to-Annual- Budgeting Mitton, C. (2001). Priority setting in regional health authorities: Program budgeting and marginal analysis. PhD, University of Calgary. Retrieved from http://ezproxy.library.ubc.ca/login?url=http://search.proquest.com/docview/30468507 8?accountid=14656 147 Mitton, C., Dionne, F., Damji, R., Campbell, D., & Bryan, S. (2011). Difficult decisions in times of constraint: Criteria based Resource Allocation in the Vancouver Coastal Health Authority. BMC Health Services Research, 11(1), 169. Mitton, C., Dionne, F., Peacock, S., & Sheps, S. (2006). Quality and cost in healthcare: a relationship worth examining. Appl Health Econ Health Policy, 5(4), 201-208. doi: 542 [pii] Mitton, C., & Donaldson, C. (2001). Twenty-five years of programme budgeting and marginal analysis in the health sector, 1974-1999. Journal of Health Services Research and Policy, 6(4), 239-248. Mitton, C., & Donaldson, C. (2002). Setting priorities in Canadian regional health authorities: a survey of key decision makers. Health Policy, 60(1), 39 - 58. Mitton, C., & Donaldson, C. (2003a). Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis (PBMA). Health Policy, 64(3), 335-348. Mitton, C., & Donaldson, C. (2003b). Tools of the trade: a comparative analysis of approaches to priority setting in healthcare. Health Services Management Research, 16, 96-105. Mitton, C., & Donaldson, C. (2004a). Health care priority setting: principles, practice and challenges. Cost Effectiveness and Resource Allocation, 2(3). doi: 10.1186/1478- 7547-2-3 Mitton, C., & Donaldson, C. (2004b). Priority-setting toolkit: a guide to the use of economics in healthcare decision-making. London: BMJ Books. Mitton, C., & Donaldson, C. (2004c). Priority Setting Toolkit: A guide to the use of economics in healthcare decision making: London: BMJ Publishing Group. Mitton, C., & Patten, S. (2004). Evidence-based priority-setting: what do the decision-makers think? [Article]. Journal of Health Services Research & Policy, 9, 146-152. Mitton, C., Patten, S., Waldner, H., & Donaldson, C. (2003). Priority setting in health authorities: a novel approach to a historical activity. Social Science & Medicine, 57(9), 1653-1663. Mitton, C., & Prout, S. (2004). Setting priorities in the south west of Western Australia: where are we now? Australian Health Review, 28(3), 301-310. Morse, J. (2003). Principles of mixed methods and multimethod research design. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research. Thousands Oaks, CA: Sage Publications. Morse, J., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods, 1. Article 2(2). Retrieved from http://www.ualberta.ca/~ijqm/ Morse, J. M., & Richards, L. (2002). Readme first for a user's guide to qualitative methods. Thousand Oaks: Sage Publications. Mortimer, D. (2010). Reorienting programme budgeting and marginal analysis (PBMA) towards disinvestment. BMC Health Services Research, 10(1), 288. Muir Gray, J. A. (2001). Evidence-based healthcare - how to make health policy and management decisions (2nd ed.). London: Churchill Livingstone, Harcourt Publishers Ltd. Nussbaum, M., & Sen, A. (Eds.). (1993). The quality of life. Oxford, UK: Oxford University Press. 148 O'Brien, R. (2001). Um exame da abordagem metodológica da pesquisa ação [An Overview of the Methodological Approach of Action Research]. In R. Richardson (Ed.), Teoria e Prática da Pesquisa Ação [Theory and Practice of Action Research]. João Pessoa, Brazil: Universidade Federal da Paraíba (English version) Available: . Patton, M. Q. (1997). Ulilization-focused evaluation - the new century text, 3rd edition. Thousand Oaks, CA: Sage Publications. Patton, M. Q. (2011). Developmental evaluation - applying complexity concepts to enhance innovation and use. New York: The Guilford Press. Peacock, S. (1998a). An evaluation of program budgeting and marginal analysis applied in South Australian hospitals. Melbourne: Center for Health Program Evaluation, Monash University. Peacock, S. (1998b). An evaluation of programme budgeting and marginal analysis applied in South Australian hospitals. Centre for Health Programme Evaluation, Monash University. Peacock, S., Mitton, C., Ruta, D., Donaldson, C., Bate, A., & Hedden, L. (2010). Priority setting in healthcare: towards guidelines for the program budgeting and marginal analysis framework. Expert Rev Pharmacoecon Outcomes Res, 10(5), 539-552. doi: 10.1586/erp.10.66 Peacock, S., Ruta, D., Mitton, C., Donaldson, C., Bate, A., & Murtagh, M. (2006). Using economics to set pragmatic and ethical priorities. BMJ, 332(7539), 482-485. doi: 10.1136/bmj.332.7539.482 Peacock, S. J., Richardson, J. R. J., Carter, R., & Edwards, D. (2007). Priority setting in health care using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Social Science & Medicine, 64(4), 897-910. Polit, D. F., Beck, C. T., & Hungler, B. P. (2001). Essentials of nursing research: methods, appraisal, and utilization (5th ed.). Philadelphia: Lippincott, Williams & Wilkins. Pope, C., & Mays, N. (Eds.). (2006). Qualitative methods in health research (3rd ed.). Malden, Massachusetts: Blackwell Publishing, Inc. Pope, C., Ziebland, S., & Mays, N. (2000). Qualitative research in health care: Analysing qualitative data. BMJ, 320(7227), 114-116. doi: 10.1136/bmj.320.7227.114 Prochaska, J. O., DiClemente, C. C., & Norcross, J. C. (1992). In search of how people change: Applications to addictive behaviors. American Psychologist, 47(9), 1102- 1114. doi: 10.1037/0003-066x.47.9.1102 Ratcliffe, J., Donaldson, C., & Macphee, S. (1996). Programme budgeting and marginal analysis: a case study of maternity services. Journal of Public Health Medicine, 18(2), 175-182. Reason, P. (1994). Three approaches to participative inquiry. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (Vol. 1, pp. 324-339). Thousand Oaks, CA: Sage. Reason, P., & Bradbury, H. (Eds.). (2008). The SAGE handbook of action research: participative inquiry and practice (Second ed.). Los Angeles, CA: SAGE Publications. Redelmeier, D. A., Shafir, E., & Aujla, P. S. (2001). The Beguiling Pursuit of More Information. Medical Decision Making, 21(5), 376-381. doi: 10.1177/0272989x0102100504 149 Robinson, R. (1993). Economic evaluation and health care: The policy context. BMJ (Clinical Research Ed.), 307(6910), 994-996. Rogers, E. M. (2003). Diffusion of innovations (Fifth ed.). New York: Free Press. Ross, J. (1995). The use of economic evaluation in health care: Australian decision makers' perceptions. Health Policy, 31(2), 103-110. Roy, D. J., Williams, J. R., & Dickens, B. M. (1994). Bioethics in Canada. Scarborough, ON: Prentice-Hall. Ruta, D., Mitton, C., Bate, A., & Donaldson, C. (2005). Programme budgeting and marginal analysis: bridging the divide between doctors and managers. BMJ, 330(7506), 1501- 1503. doi: 10.1136/bmj.330.7506.1501 Ruta, D. A., Donaldson, C., & Gilray, I. (1996). Economics, public health and health care purchasing: the Tayside experience of programme budgeting and marginal analysis. J Health Serv Res Policy, 1(4), 185-193. Sabin, J. E. (1998). Fairness as a problem of love and the heart: A clinician's perspective on priority setting. BMJ: British Medical Journal, 317(7164), 1002-1004. Sandelowski, M. (1998). Writing a good read: Strategies for re-presenting qualitative data. Research in Nursing & Health, 21(4), 375-382. doi: 10.1002/(sici)1098- 240x(199808)21:4<375::aid-nur9>3.0.co;2-c Sandelowski, M. R. N. P. (1986). The problem of rigor in qualitative research. Advances in Nursing Science, 8(3), 27-37. Schackman, B. R. (2010). Implementation Science for the Prevention and Treatment of HIV/AIDS. JAIDS Journal of Acquired Immune Deficiency Syndromes, 55, S27-S31 10.1097/QAI.1090b1013e3181f1099c1091da. Schwandt, T. A. (2007). The Sage Dictionary of Qualitative Inquiry (Third ed.). Los Angeles, CA: SAGE Publications. Scott, A., Currie, N., & Donaldson, C. (1998). Evaluating innovation in general practice: a pragmatic framework using programme budgeting and marginal analysis. Fam Pract, 15(3), 216-222. Senge, P. M. (2006). The fifth discipline - the art and practice of the learning organization. New York: Doubleday. Sibbald, S., Singer, P., Upshur, R., & Martin, D. (2009). Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Services Research, 9(1), 43. Simon, H. A. (1959). Theories of Decision-Making in Economics and Behavioral Science. The American Economic Review, 49(3), 253-283. Simon, H. A. (1972). Theories of bounded rationality. In C. B. McGuire & R. Radner (Eds.), Decision and organization (pp. 161-176). Amsterdam: North-Holland Publishing Company. Singer, P. A. (2000). Medical ethics. British Medical Journal, 321(7256), 282-285. Smith, R., Hiatt, H., & Berwick, D. (1999). A shared statement of ethical principles for those who shape and give health care: a working draft from the Tavistock group. Annals Of Internal Medicine, 130(2), 143-147. Stringer, E., & Genat, W. (2004). Action research in health. Upper Saddle River, New Jersey: Pearson Education, Inc. Suter, E., Oelke, N. D., Adair, C. E., & Armitage, G. D. (2009). Ten key principles for successful health systems integration. [Research Support, Non-U.S. Gov't 150 Review]. Healthc Q, 13 Spec No, 16-23. Swepson, P. (1995). Action research: understanding its philosophy can improve your practice. Retrieved from http://www.aral.com.au/resources/philos.html Teng, F., Mitton, C., & MacKenzie, J. (2007). Priority setting in the provincial health services authority: survey of key decision makers. BMC Health Services Research, 7(8410.1186/1472-6963-7-84). Thomas, P. (2004) Performance Measurement, Reporting and Accountability: Recent Trends and Future Directions. SIPP Public Policy Paper #23. Regina: Saskatchewan Institute of Public Policy (SIPP). Tiffany, C. R., & Johnson, L. L. (1998). Planned change theories for nursing. Review, analysis and implications. Thousand Oaks: Sage. Tobin, G. A., & Begley, C. M. (2004). Methodological rigour within a qualitative framework. Journal of Advanced Nursing, 48(4), 388-396. Tolley, K. H., & Whynes, D. K. (1995). The priority-setting exercise: An instrument for training in health care resource allocation. Medical Teacher, 17(4), 391-398. Tsourapas, A., & Frew, E. (2011). Evaluating 'success' in programme budgeting and marginal analysis: a literature review. J Health Serv Res Policy, 16(3), 177-183. doi: 10.1258/jhsrp.2010.009053 Twaddle, S., & Walker, A. (1995). Programme budgeting and marginal analysis: application within programmes to assist purchasing in Greater Glasgow Health Board. Health Policy, 33(2), 91-105. Twomey Fosnot, C. (2005). Constructivism: Theory, Perspectives, and Practices (2nd ed.). New York, NY: Teachers College Press, Columbia University. Urquhart, B., Mitton, C., & Peacock, S. (2008). Introducing priority setting and resource allocation in home and community care programs. [Research Support, Non-U.S. Gov't]. Journal of Health Services Research and Policy, 13(Suppl 1), 41-45. doi: 10.1258/jhsrp.2007.007064 Vedel, I., De Stampa, M., Bergman, H., Ankri, J., Cassou, B., Blanchard, F., & Lapointe, L. (2009). Healthcare professionals and managers' participation in developing an intervention : a pre-intervention study in the elderly care context. Implementation Science, 4(1). doi: 10.1186/1748-5908-4-21 Viney, R., Haas, M., & De Abreu Lourenco, R. (2000). A practical approach to planning health services: using PBMA. Aust Health Rev, 23(3), 10-19. Vogt, W. P. (2005). Dictionary of statistics & methodology: a nontechnical guide for the social sciences (3rd ed.). Thousand Oaks: Sage Publications. Walley, T., Barton, S., Cooke, J., & Drummond, M. (1997). Economic evaluations of drug therapy: attitudes of primary care prescribing advisers in Great Britain. Health Policy, 41(1), 61-72. Webster's. (1989). Webster's encyclopedic unabridged dictionary of the English language. New York: Portland House, dilithium Press, Ltd. Westley, F., Zimmerman, B., & Patton, M. Q. (2007). Getting to maybe: how the world is changed. Toronto: Vintage Canada, Random House of Canada Limited. Williams, I., & Bryan, S. (2007a). Understanding the limited impact of economic evaluation in health care resource allocation: A conceptual framework. Health Policy, 80(1), 135-143. 151 Williams, I. P., & Bryan, S. (2007b). Cost-effectiveness analysis and formulary decision making in England: Findings from research. Social Science & Medicine, 65(10), 2116-2129. Williams, J. R., & Yeo, M. (2000). The ethics of decentralizing health care priority setting in Canada. In A. Coulter & C. Ham (Eds.), The global challenge of health care rationing (pp. 123-132). Buckingham: Open University Press. Wilson, E. C., Peacock, S. J., & Ruta, D. (2009). Priority setting in practice: what is the best way to compare costs and benefits? [Research Support, Non-U.S. Gov't]. Health Econ, 18(4), 467-478. doi: 10.1002/hec.1380 152 Appendices Appendix A Visual representation of problem statement Figure A.1 Visual representation of problem statement 153 Appendix B Conceptual model for the diffusion of innovations in healthcare organizations The Greenhalgh model (see Figure B.1 below) evolved from a National Health Service (UK) funded research program on change management (Greenhalgh et al., 2005). It started with a systematic literature review and synthesis designed to address the question of ‘how innovations can be spread/sustained in health delivery organizations’ (Greenhalgh et al., 2004). This model is designed primarily as a memory aid to highlight the various relevant factors that influence change (i.e., adoption of an innovation) in complex organizations, including how these factors interact. It should be used to ‘illuminate the problem(s)’ and ‘raise areas to consider’ (Greenhalgh et al., 2004). What Greenhalgh and colleagues have found is that adoption and use of innovations does not usually follow a linear path and can be stymied by factors relevant to the innovation (e.g., complex, risky innovations), adopter (e.g., lacking skills, motivation), organization (e.g., unreceptive context for change), and implementation process (e.g., lacking resources for implementation) (Greenhalgh et al., 2005). A thorough assessment of possible barriers helps implementers identify potential roadblocks to develop appropriate strategies or determine if change is even possible. Also, understanding relevant theoretical influences on behaviour change enables development of targeted strategies to change practice in a specific context. 154 Figure B.1 Conceptual model for the diffusion of innovations in healthcare organizations Diffusion of innovation model* *Greenhalgh et al (2005) Diffusion of innovations in health service organisations: a systematic literature review. USA: Blackwell The innovation: • relative advantage • potential for adaptation • observability, etc. Communication/Influence: Diffusion: informal/unplanned Dissemination: formal/planned Outer context: • sociopolitical climate • incentives • norms/networks, etc. Linkage – design & implementation stages System antecedents for innovation: structure, absorptivity, receptivity System readiness for innovation: • tension for change • innovation-system fit • power balances, etc. The adopter: • needs • motivation • learning style, etc. Assimilation: Complex, non-linear Soft periphery elements Implementation process: • decision-making devolved • dedicated resources, etc. Resource system Knowledge purveyors Change agency Outer context User system Consequences system antecedents system readiness adoption/assimilation implementation Linkage Linkage The innovation Dissemination Social networks Marketing Diffusion 155 Appendix C Study design Figure C.1 Study design 156 Appendix D Weighted criteria and score-sheet Figure D.1 Weighted criteria 157 Figure D.2 Score-sheet 158 Appendix E Investment and disinvestment proposals Figure E.1 List of investment and disinvestment proposals Figure E.2 Ranked list of investment and disinvestment options 159 Appendix F Pre-PBMA focus group and one-on-one qualitative interview question guide 1. What is your perception of what priority setting entails in your organization? 2. What is your overall reflection on the current priority setting practices and those in the past? 3. What have been the major driving forces behind priority setting exercises up to this point? What do you think the major driving forces should be? 4. Is it clear what the values and guiding principles of this organization are? To what extent are these values considered when setting priorities and allocating resources? 5. To what extent does IH Central Okanagan community care have an organizational culture conducive to using evidence in priority setting activities? 6. What capacities currently exist within your organization to build a macro- level priority setting model? (e.g. organizational structure, information sources, links to the university) 7. How could priority setting practices/processes in your organization be improved? 8. What is your vision for priority setting models and practices in the future? 9. What further information or training do you think the organization needs to get to your ultimate priority setting model? 10. Based on our discussion today, are there any additional points that you would like to make or information that you’d like to share? * Adapted from and validated in previous surveys in Australia, UK and Alberta (Mitton & Donaldson, 2003a; Mitton & Prout, 2004; Teng et al., 2007) 160 Appendix G Post year-one PBMA qualitative interview question guide Context 1. What is your understanding of the priority-process just completed? Describe the process. Describe your role in it. 2. How would you describe your level of engagement (or buy-in) in the process? High: what contributed to the buy-in? e.g., High level champion, organizational culture Low: what limited buy-in? e.g., Other demands, lack of leadership, politics 3. Were resources allocated or re-allocated on the basis of this process? If yes, any examples. If not based on this process but reallocations were made, what criteria did you use when making decisions regarding resource allocation? The process 1a. What was good about the PBMA process? 1b. What was bad about it – compared to before? (i.e., previous priority setting approach) Specific issues: a. trust between different levels of the organization b. issues about representation c. were administrators and physician on board (medical director) d. was there a relationship between participation in PBMA and performance evaluation (alignment of incentives) e. what was the role of politics (versus role of research evidence) compared to before 2. Ranking process: What went well? What did not go well? Specifically, any comments about: a. validation, peer review b. ranking criteria: cross portfolio comparisons and relationship to corporate values 161 c. role of evidence – Has your use of evidence (research literature, local data, etc.) changed with the introduction of PBMA; if so, how? d. participation, re: physicians, public member e. fairness f. timeliness g. communication h. transparency i. is any specific assistance needed to prepare business cases 3. Process fairness: Was the PBMA process fair and transparent? How do we ensure it is fairer in the future? Specifically, any comments about: a. Communication b. Relevance c. Appeals d. Enforcement Future 1. Overall, are you in favor of continuing with the process? If no, what change would make you say yes? If yes, what would be your priority for improvement? 2. What would you do differently for future priority setting exercises in this organization (e.g., dealing with gaming, time constraints, political pressures, etc.)? *Adapted from and validated in previous survey work in Alberta and B.C. (Mitton & Donaldson, 2003a; Teng et al., 2007) 162 Appendix H Post year-two PBMA qualitative interview question guide PBMA fit in IH CC context Fit refers here to being of suitable quality and form, including suitable features, to meet required purpose. 1. Is PBMA a mechanism or tool that can assist priority setting in this community care context? Consider the following components: structure (the 7 step implementation, including formation of advisory panel, regular meetings, etc.), priority setting process that PBMA facilitates (transparent, rational approach), tools (criteria, score-sheet, business case template) If yes a. What components of the PBMA framework assisted in priority setting? b. What components were of greatest assistance, i.e. what specifically about PBMA fit? c. How could it be improved to be even more useful/useable (i.e., an even better fit)? d. If no, why not? What specifically about PBMA did not fit? What would be more useful/usable to assist priority setting in the community care context/culture? 2. Is PBMA being used in community care at present, or are there plans to use it in the near future? If not, why not? Is it a question of fit or some other reason? 3. What is it about the community care context that contributes to PBMA fitting or not? PBMA Impact Impact refers here to a noticeable outcome or influence; i.e., PBMA influenced policy or decision making practice as it was used to set specific priorities (it was useful). Impact could also be more wide-ranging regarding PBMA use, including indirect ways in which PBMA may have impacted (influenced) your (as a PBMA participant) knowledge, understanding and/or attitudes regarding priority setting. 4. Now that you’ve gone through PBMA, do you see any benefits (i.e., advantages) in using PBMA compared to previous priority setting approaches? a. If yes, what are the benefits? 163 b. If no, what makes the previous approach better? What does the previous/better approach look like (describe it). c. What changes could one make to the existing priority setting process to bolster it (i.e. improve its impact and/or benefit)? d. What changes could one make to the existing priority setting process to make it more like desired priority setting practice (structured, transparent, evidence- informed; as indicated by pre-PBMA interviews)? e. Are these attributes (listed in d above) even achievable, or are they more an idealized version of what priority setting would include in an ideal world? 5. What was the impact in using PBMA on your priority setting practice? a. If there was an impact what were the specific changes? b. If no impact, what could be done (and by who) differently to increase its impact? c. How should/can impact be measured? Other 6. Public member: a. What role do you feel the public member played on the advisory panel (describe it)? b. Do you think her role was useful and used appropriately? If not, how else could public input/participation be sought in priority setting? 7. Do you have anything else to say about PBMA or the experience this past year, i.e. second cycle of PBMA? "@en ; edm:hasType "Thesis/Dissertation"@en ; vivo:dateIssued "2013-05"@en ; edm:isShownAt "10.14288/1.0073265"@en ; dcterms:language "eng"@en ; ns0:degreeDiscipline "Interdisciplinary Studies"@en ; edm:provider "Vancouver : University of British Columbia Library"@en ; dcterms:publisher "University of British Columbia"@en ; dcterms:rights "Attribution-NonCommercial-NoDerivatives 4.0 International"@en ; ns0:rightsURI "http://creativecommons.org/licenses/by-nc-nd/4.0/"@en ; ns0:scholarLevel "Graduate"@en ; dcterms:title "Implementing a rational priority setting approach in community care in the Interior Health Authority: assessing fit, evaluating implementation, and determining impact"@en ; dcterms:type "Text"@en ; ns0:identifierURI "http://hdl.handle.net/2429/43394"@en .