UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Describing practices of priority setting in publicly funded health care systems of high-income countries Seixas, Brayan V. 2018

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


24-ubc_2018_september_viegasseixas_brayan.pdf [ 1.48MB ]
JSON: 24-1.0371121.json
JSON-LD: 24-1.0371121-ld.json
RDF/XML (Pretty): 24-1.0371121-rdf.xml
RDF/JSON: 24-1.0371121-rdf.json
Turtle: 24-1.0371121-turtle.txt
N-Triples: 24-1.0371121-rdf-ntriples.txt
Original Record: 24-1.0371121-source.json
Full Text

Full Text

DESCRIBING PRACTICES OF PRIORITY SETTING IN PUBLICLY FUNDED HEALTH CARE SYSTEMS OF HIGH-INCOME COUNTRIES by  Brayan V. Seixas  BPharm., Federal University of Rio de Janeiro, 2012  A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF  MASTER OF SCIENCE in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Population and Public Health)  THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver)  August 2018  © Brayan V. Seixas, 2018   ii  The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, a thesis/dissertation entitled:  DESCRIBING PRACTICES OF PRIORITY SETTING IN PUBLICLY FUNDED HEALTH CARE SYSTEMS OF HIGH-INCOME COUNTRIES  submitted by Brayan Viegas Seixas in partial fulfillment of the requirements for the degree of Master of Science in Population and Public Health  Examining Committee: Dr. Craig Mitton Supervisor      Dr. Dean Regier Supervisory Committee Member  Dr. Stirling Bryan Supervisory Committee Member Dr. Jason Sutherland Additional Examiner   iii Abstract  Spending in health care have grown considerably over the last decades in all developed countries. Aging populations and novel technologies are usually deemed as the main drivers for such unsustainable trend in the finances of health systems. In this scenario, setting priorities for investments becomes a central concern. Making hard choices in a rational, evidence-informed, systematic, transparent, legitimate, and fair manner has constituted an increasingly important target. Notwithstanding the voluminous body of literature in this area, most of work focus on developing and improving prescriptive approaches as well as presenting case studies. The present work aimed to describe existing practices of priority setting and resource allocation within the context of publicly funded health care systems of high-income countries. An online qualitative survey was used with decision makers and academics from 18 countries. Four hundred and fifty individuals were invited and 58 answered the survey questionnaire. We found that resource allocation in health care has been still largely done based on historical patterns and through ad hoc decisions, despite the wide understanding that decisions must be formally based on multiple explicit criteria. Health technology assessment (HTA) was the tool most commonly indicated by participants as a formal strategy of priority setting. Several approaches have been tested and published, with special emphasis on Program Budgeting and Marginal Analysis (PBMA), but there is limited evidence of their continuous and systematic use by health organizations across countries. A point of increasing convergence is the reliance of multiple types of evidence to judge the value of investment options. Disinvestment frameworks are very rare and the topic itself has only started to appear with any regularity. iv This work represents the first attempt to identify existing practices of priority setting in a systematic way through a qualitative descriptive study. Despite its methodological limitations, it provides a better understanding of the current scenario of policy making and research in this field.   v Lay Summary  In all developed countries, governments have spent more and more money in health care. Because budgets are fixed while demands can grow indefinitely, difficult choices must be made. Although theories and methodologies have been proposed to guide this difficult process of decision making, no approach has achieved gold-standard status. Thus, priority-setting processes vary significantly across health organizations, regions, and countries. Our study sought to understand the existing practices of decision making in priority setting and resource allocation in publicly funded health systems of high-income countries. We invited decision makers and researchers from 18 countries to answer a survey. We found that resource allocation is still largely based on what has been historically done, without a formal framework. Yet, most participants agree that formal processes with multiple criteria must exist. There is much emphasis on formal assessment of new treatments, but not on resource allocation broadly speaking. Initiatives to review existing spending are rare.  vi Preface  The idea of filling the literature gap concerning the existing practices of priority setting and resource allocation in publicly funded health care systems of high-income countries arose in conversations between me and my supervisor Dr. Craig Mitton about possible projects for my MSc thesis. Right after the 11th International Conference on Priorities in Health held in September 2016 in Birmingham, UK, Dr. Mitton pointed out that an important message of that conference was the confluence of principles and approaches of priority setting across the globe. It was clear that priority setting has become a central topic in developed countries and that decision makers and researchers have more and more understood the importance of developing explicit and consistent multi-criteria methods of decision making. Nonetheless, despite great willingness to advance the research field and improve management practices, we realized that there was a need for better understanding of the current processes of decision making in determining priorities for investing the scarce resources within health care systems of “developed” or “high-income” countries. Most of the study design was developed by me along the course SPPH 519 - Qualitative Methods in Health Research Design, in which I received the inestimable support of Dr. Susan Cox guiding me through the daunting world of qualitative work. All methodological choices were carefully taken under the supervision and advice of Dr. Craig Mitton and Dr. Dean Regier. I was responsible for all subsequent steps, including developing the survey instrument, identifying potential participants, and inviting them to participate. Neale Smith helped create the survey questionnaire, and Dr. Rob Baltussen and Dr. Sam Sheps kindly participated in the pilot study. vii The data analysis was carried out by me with the close support of my supervisory committee (Dr. Craig Mitton, Dr. Dean Regier and Dr. Stirling Bryan). In addition, Dr. François Dionne, an expert in the field of health-care priority setting, also offered insightful feedback. In Chapter 1, section 1.2.3 builds upon insights that were presented in a critical essay published in a peer-reviewed journal (Seixas, B. Welfarism and extra-welfarism: a critical overview. Cad Saúde Pública. 2017;33(8)). Section 1.4.1 describes the evidence synthesis of a systematic literature review led by me, with input from Dr. Craig Mitton, Dr. François Dionne and Tania Conte. Chapter 2 presents the theoretical reflections underpinning the methodology used here. A shorter version of this chapter has been published. (Seixas, B. Smith, N. Mitton, C. The Qualitative Descriptive Approach in International Comparative Studies: Using Online Qualitative Surveys. Int J Health Policy Manag 2018, x(x), 1–4.) I developed the overarching theoretical rationale, the specific arguments, and the metaphors and wrote the first draft of the manuscript. Dr. Mitton and Mr. Smith provided feedback on the subsequent versions of the article. Ethics approval for this study was secured from the UBC Behavioural Research Ethics Board – Certificate # H17-02009. viii Table of Contents  Abstract ......................................................................................................................................... iii Lay Summary .................................................................................................................................v Preface ........................................................................................................................................... vi Table of Contents ....................................................................................................................... viii List of Tables ............................................................................................................................... xii List of Figures ............................................................................................................................. xiii List of Abbreviations ................................................................................................................. xiv Acknowledgements .................................................................................................................... xvi Dedication ................................................................................................................................. xviii Chapter 1: Introduction ................................................................................................................1 1.1 Priority Setting in Health Care ........................................................................................ 1 1.1.1 Research in PSRA ....................................................................................................... 6 1.2 PSRA and Welfare Economics ....................................................................................... 9 1.2.1 Cost-Benefit Analysis (CBA) ................................................................................... 11 1.2.2 Cost-Utility Analysis (CUA) and Cost-Effectiveness Analysis (CEA) .................... 12 1.2.3 A critical overview of the prescriptive paradigms grounded in welfare economics . 14 Criticism on the welfarist approach .................................................................. 15 Criticism on the extra-welfarist approach ......................................................... 19 1.3 PSRA and Decision Sciences........................................................................................ 23 1.3.1 MCDA....................................................................................................................... 24 1.3.2 PBMA ....................................................................................................................... 27 ix 1.3.3 Critical Overview of Prescriptive Approaches Grounded in Decision Sciences ...... 30 1.4 Current Practices of Priority Setting ............................................................................. 32 1.4.1 Literature Review on Applied PSRA Frameworks ................................................... 32 1.4.2 Expanding the Horizon: Findings from a Grey Literature Search and Other Relevant Literature References ............................................................................................................ 35 Chapter 2: Study Objectives and Research Questions .............................................................39 Chapter 3: A Theoretical Reflection on Methodological Aspects: Online Qualitative Surveys as a Qualitative Descriptive Approach in International Comparative Studies .......42 3.1 Preliminary Considerations ........................................................................................... 42 3.2 Theoretical Reasoning .................................................................................................. 43 3.3 Practical Aspects of Study Design ................................................................................ 48 3.3.1 Sampling ................................................................................................................... 49 3.3.2 Data Collection ......................................................................................................... 51 3.3.3 Data Analysis ............................................................................................................ 52 3.4 Final Considerations ..................................................................................................... 56 Chapter 4: Methods .....................................................................................................................58 4.1 Research Design............................................................................................................ 58 4.1.1 Sampling ................................................................................................................... 58 4.1.2 Data Collection ......................................................................................................... 61 4.1.3 Data Analysis ............................................................................................................ 62 Chapter 5: Results........................................................................................................................63 5.1 Survey Response ........................................................................................................... 63 5.2 PSRA Frameworks........................................................................................................ 65 x 5.2.1 National Level ........................................................................................................... 71 5.2.2 State/Provincial, Regional, or Single Organization Levels ...................................... 82 5.3 Disinvestment ............................................................................................................... 89 5.4 Stakeholder Involvement and Public Engagement ....................................................... 94 5.5 Facilitators and Barriers .............................................................................................. 101 5.6 Strengths and Areas for Improvement ........................................................................ 105 5.7 Lesser Themes (Use of Evidence and Appeals Mechanism) ...................................... 110 5.8 Quantitative Aspects ................................................................................................... 112 Chapter 6: Discussion ................................................................................................................116 6.1 The Current Scenario of Priority Setting .................................................................... 118 6.2 The Underlying Context for Priority Setting .............................................................. 124 6.3 Methodological Aspects and Study Limitations ......................................................... 125 Chapter 7: Conclusion ...............................................................................................................129 Bibliography ...............................................................................................................................133 Appendices ..................................................................................................................................142 Appendix A Literature Review Search Strategy ..................................................................... 142 Appendix B Inclusion and Exclusion Criteria of Systematic Literature Review ................... 143 Appendix C Extraction Tool ................................................................................................... 144 Appendix D Summary of the 23 Papers Selected in the Systematic Review ......................... 145 Appendix E Initial Pool of Countries to be Sampled .............................................................. 147 Appendix F List of Organizations’ Websites Used in the Grey Literature Search to Identify Potential Participants .............................................................................................................. 148 Appendix G Survey Instrument (Introductory Message, Consent Form, Questionnaire) ...... 149 xi Appendix H Standard Invitation Message .............................................................................. 156 Appendix I Snowball Sampling Invitation Message .............................................................. 157  xii List of Tables  Table 4.1 List of countries included in the study. ......................................................................... 60 Table 5.1 Participants according to their country and primary professional role. ........................ 64 Table 5.2 Findings on existing processes of PSRA. ..................................................................... 66 Table 5.3 Facilitators and barriers for the development and implementation of formal PSRA processes according to survey participants. ................................................................................ 102 Table 5.4 Strengths and areas for improvement in the current practices according to survey participants. ................................................................................................................................. 106 Table 5.5 Participants' views about perceived fairness of current practices of decision-making in PSRA........................................................................................................................................... 114  xiii List of Figures  Figure 1.1 Demographic trend in Canada.4 ..................................................................................... 3 Figure 1.2 Population aged 85 and older, Canada, 1966 to 2051.5 ................................................. 3 Figure 1.3 Health spending per capita in current dollars, Canada.  Source: Canadian Institute for Health Information.6 Data for years 2016-2017 are based on forecasted numbers. ....................... 4 Figure 1.4 Health spending as a percentage of GDP. Source: OECD.7 .......................................... 5 Figure 1.5 Number of articles containing the term “priority setting” in PubMed. ......................... 6 Figure 1.6 Prescriptive approaches of priority setting and resource according to the normative paradigm of decision-making research where they are rooted. ...................................................... 8 Figure 1.7 Socially optimal point under utilitarian social welfare function. ................................ 17 Figure 1.8 Societal indifference curves under different scenarios. Inspired on Norman et al., 2013.35 ........................................................................................................................................... 22 Figure 1.9 PBMA steps. ................................................................................................................ 29 Figure 3.1 Purposeful sampling strategies for qualitative description in international comparative studies. .......................................................................................................................................... 51 Figure 3.2 Position of our qualitative descriptive approach within Sandelowski and Barroso’s typology of qualitative research findings. ..................................................................................... 55 Figure 5.1 Number of invitees and respondents per country. ....................................................... 63 Figure 5.2 Institutional contexts with/where which participants work or have worked. ............ 113 Figure 5.3 Participants' evaluation of current performance. ....................................................... 114  xiv List of Abbreviations  A4R Accountability for Reasonableness ACRA Advisory Committee on Resource Allocation (UK) CADTH Canadian Agency for Drugs and Technologies in Health CBA Cost-Benefit Analysis CCA Cost-Consequence Analysis CCG Clinical Commissioning Group CEA Cost-Effectiveness Analysis CER Comparative Effectiveness Research CHSC Committee for Health and Social Care (Guernsey) CIHI Canadian Institute for Health Information CMA Cost-Minimization Analysis COHERE The Council for Choices in Health (Finland) CUA Cost-Utility Analysis G7 A group of industrialized countries that consists of Canada, France, Germany, Italy, Japan, the United Kingdom, and the United States  G-BA Federal Joint Committee (Germany) GCEA Generalized Cost-Effectiveness Analysis GDP Gross Domestic Product GKV Health Insurance Funds (Germany) HRQoL Health-Related Quality of Life HTA Health Technology Assessment xv ICD International Classification of Diseases ISPOR International Society for Pharmacoeconomics and Outcomes Research MCDA Multi-Criteria Decision Analysis MSAC Medical Services Advisory Committee (Australia) MSAH Ministry of Social Affairs and Health (Finland) NICE National Institute for Health and Care Excellence (UK) OECD Organization for Economic Cooperation and Development PBAC Pharmaceutical Benefits Advisory Committee (Australia) PBMA Program Budgeting and Marginal Analysis PEC Priorities and Evaluation Committee PHARMAC Pharmaceutical Management Agency (New Zealand) PSRA Priority Setting and Resource Allocation SAPACT South Australian Policy Advisory Committee on Technology STAR Socio-Technical Allocation of Resources WHO World Health Organization ZIN Zorginstituut Nederland [National Health Care Institute]  xvi Acknowledgements  Over the years I become more and more convinced that every successful human endeavour is the result of the actions and dreams of many individuals and a good dose of luck. The self-made man ideology is a myth that makes invisible those who have co-written stories of success. I must start by thanking my parents Claudio and Lucia, who made all possible efforts to help me flourish in a deserted land. Because you never told me there was any place I could not go, I believed I could go anywhere. You gave me the self-confidence and emotional intelligence I needed to go beyond the destiny of most people in my social class. I must also express my gratitude to my entire family, who have celebrated every small accomplishment in my life. I sincerely appreciate the pride and trust of my brother, Claudio Seixas, and my sister, Claudia Viegas. Your enthusiasm is the fuel I use to keep going in the harshest moments. Science cannot explain the magic behind these good vibes that push me forward. Every victory of mine is also a victory for all of you! Special thanks to my wife, Raiane Salles, for being so supportive of all my decisions and encouraging in all my plans. You have contributed to much of my personal and spiritual growth. The shared quest of figuring out narratives to better frame our life has helped me become who I am. Our love has become a refuge for our subjectivities. Thank you! My profound gratitude goes to the most generous person I have met in Canada: my supervisor, Dr. Craig Mitton! I cannot thank you enough for all you have done for me during almost three years. Agreeing to endorse my MSc application, providing financial aid, inviting me to participate in research projects, and stimulating my professional growth—you have been more than a supervisor, you have been a mentor, a source of inspiration, and a friend. This experience xvii created in me a duty and desire to give back as I have received: to be a caring, dedicated, and inspirational leader and supervisor for my future students and collaborators. Sincere gratitude also to Dr. Dean Regier and Dr. Stirling Bryan, my kind and encouraging co-supervisors. It is an inestimable experience to have the advice and follow-up of researchers I admire so much. Your questions and feedback have made me look beyond what was right in front of me. I cannot forget to mention my dear friends and UBC colleagues. Special thanks to Douglas Barrozo, Rafael Harduim, Rayan Ybarra, Túlio Elisiário, Marco Antônio, Tiago Fernandes, Gabriel Freitas, André Maiocchi, Ricardo Fernandes, and Vinicius Peçanha for the endless conversations about politics, philosophy, arts, and public health. Nurturing a collective dream of a better world has been absolutely vital to keep faith on the small actions of our everyday life. In addition, particular thanks to Tania Conte, who helped me so much navigating a foreign institutional context. Finally, my genuine thanks to everyone who somehow contributed to this enterprise, even if very indirectly, like the flapping of butterfly wings influencing a tornado on the other side of the world. And, of course, thanks to the undeserved good luck that has reached me. xviii Dedication To my beloved wife Raiane, without whom I would not have gotten here.  1 Chapter 1: Introduction 1.1 Priority Setting in Health Care No matter how wealthy or large an organization, its available resources can never be deemed unlimited, because demand for new investments can grow indefinitely. Thus, organizations (and individuals) must constantly give preference to some types of spending over others. Individuals, companies, and governments alike are constantly selecting the priorities to which their scarce resources will be allocated. The process of assigning precedence to certain areas or services to receive investments is referred to in the scientific literature as priority setting and resource allocation (PSRA) or simply priority setting. The term rationing is also used as a synonym for priority setting. Although they have commonly been used in an interchangeable manner, the terms are theoretically distinguishable. In the neoclassical economics literature, the notion of rationing arises as a way of allocating resources when the price mechanism is not applicable: “rationing refers to distribution under conditions where a supply that is limited by considerations of cost confronts a demand that is not constrained by considerations of price”.1 (p1) Priority setting refers to the process by which decision makers establish a relative ranking of programs/areas to receive investment. Thus, priority setting represents a form of rationing. The word rationing is built upon the Latin root -ratio, meaning reason, logic, judgement, but the term is by no means emotionally neutral. In fact, rationing commonly has a negative connotation, evoking deprivation, cut-offs, austerity, and undue scarcity. As Klein et al. put it, “the acceptability of rationing seems to depend on its perceived reasonableness, which, in turn, appears to depend on the form it takes”.2 (p8) Priority setting is often the term of choice because of its 2 relatively positive connotation, indicating a sense of control over well-known alternatives and their impacts, even though it is still a delicate and turbulent process. Tragakes and Vienonen further clarify the subtle theoretical distinctions between rationing and priority setting: “While priority setting ultimately achieves rationing, rationing itself may or may not be achieved through priority setting. Rationing can take place in many different ways, of which priority setting is only one.”1 (p3) In health and health care, rationing and priority setting are particularly challenging. Health is a recognized fundamental human right that assures the minimum conditions to enjoy a satisfactory life, whatever that means for individuals. Health care has been under substantial continuous budgetary pressures for decades. Two facts are usually deemed as the major causes for this challenging phenomenon: aging populations and novel expensive treatments. It is well known that people are living longer today than they did in the past. Figure 1.1 and Figure 1.2 show this scenario in Canada, where the number of people over 85 has grown exponentially and where the proportion of seniors (over 65 years old) already exceeds the proportion of children (under 14 years old). This demographic transformation has created significant challenges such as increased prevalence rates of chronic diseases and higher demands of appropriate care for seniors. In Canada, health care spending per capita on seniors is over four times that for non-senior adults (age 20 to 64 years).3 Aging populations necessarily mean a higher demand for health care spending. 3  Figure 1.1 Demographic trend in Canada.4    Figure 1.2 Population aged 85 and older, Canada, 1966 to 2051.5  4 The other important element driving up health care spending is the development of novel treatment regimes, medical devices, and diagnostic tools. Technological advancement is a much-desired achievement in any area of human endeavour. But in health care, unlike in other industries, new technologies do not help decrease production costs. This puts the system under constant pressure to spend more on technologies deemed to generate better outcomes. In Canada, health care spending has grown substantially over the last four decades. Figure 1.3 illustrates the total health spending per capita from 1975 to 2017 as well as per capita spending in the three largest areas of resource allocation in Canada.    Figure 1.3 Health spending per capita in current dollars, Canada.  Source: Canadian Institute for Health Information.6 Data for years 2016-2017 are based on forecasted numbers.  5 This trend in health care spending is not unique to Canada. Virtually all industrialized countries face the same problem: the percentage of the gross domestic product (GDP) designated for health care has ballooned. Figure 1.4 shows the rise of health spending relative to GDP for the G7 countries.   Figure 1.4 Health spending as a percentage of GDP. Source: OECD.7  The scenario of ever-increasing health care spending and demographic and technological transformations, which will likely keep health care systems under budgetary pressures for the upcoming decades, poses enormous challenges for policy makers and health researchers. Thus, developing tools and knowledge to allocate scarce resources in the most efficient manner has become increasingly relevant in the field of health economics and public policy. There has been a growing perception that, as Mitton and Donaldson state8, scarcity is here to stay. The area of 6 research in priority setting has grown considerably over the last 15 years, as illustrated in Figure 1.5.   Figure 1.5 Number of articles containing the term “priority setting” in PubMed.   1.1.1 Research in PSRA  Establishing the epistemological boundaries of the area of PSRA is not easy. Dealing with the problem of resource scarcity before the virtually unlimited possibilities of spending actually represents the founding problem of the discipline of economics. The notions of scarcity, trade-off, and opportunity cost are core principles in economics. But then what distinguishes the area of PSRA from other areas of economics or public policy? What are the theoretical underpinnings, the normative paradigms, and the prescriptive approaches of this scientific area? As discussed in the previous section, when the price mechanism is not applicable to equilibrate the supply and demand sides, rationing arises in the horizon. Thus, instead of consumers and producers acting according to their own interests, that guided by an invisible hand, 7 would ultimately lead to a fair and efficient distribution of the scarce resources, as preached by Adam Smith for regular free markets9, the health care system has constituent exceptionalities, or market failures, as we know at least since Arrow’s seminal work10, that makes it a perfect place for the practice of rationing. In publicly funded health care systems, the need to rationalize the allocation of resources becomes more easily recognizable. Broadly speaking, every effort to use resources efficiently falls within the area of priority setting because the opportunity cost associated with any decision means that choosing where to spend money also means choosing where not to spend it. Thus, all strategies for achieving higher value (in the sense of health outcomes obtained per dollar spent)11 in the health care system can be deemed PSRA strategies. In this sense, value-based purchasing strategies, HTA, and even methodologies without explicit considerations of cost, such as comparative effectiveness research (CER), could be considered practices of PSRA. However, although the distinction between the different areas of scientific research in health care and health economics is somewhat blurred, we can definitely pursue a better delineation of what PSRA actually encompasses and what the epistemological consequences are for the present work. Multiple lenses can be used to understand the PSRA problem and its most common approach. Depending on the overarching epistemological perspective, i.e., whether one is observing the problem primarily from the standpoint of ethics, political science, economics, or even from different paradigms within each field of knowledge, the question gains distinct contours. Here we will discuss the problem of priority setting circumscribed in the field of health economics, where three paradigms are traditionally applied to this topic. The first paradigm, welfarism, is a corollary of the application of welfare economics into the field of health care; the second one, 8 extra-welfarism, arises in opposition to the first, but keeping a lineage relationship; and lastly, a paradigm predominantly grounded in decision sciences. As we will see, these paradigms have elements in common, but they rely on fundamentally different theoretical premises. Figure 1.6 illustrates these epistemological paradigms and the most important practical approaches prescribed upon them.   Figure 1.6 Prescriptive approaches of priority setting and resource according to the normative paradigm of decision-making research where they are rooted.  PSRAWelfarismCBAExtra-welfarismCUACEADecision SciencesMCDA PBMA9 1.2 PSRA and Welfare Economics Both welfarism and extra-welfarism arise from and within the discourse and theoretical sphere of welfare economics. Some would argue that extra-welfarism represents such a fundamental shift from neoclassical welfare economics that it can no longer be understood from this perspective. Within this (extra)-welfarist realm reside the approaches for economic evaluation of health care technologies and programs, including cost-benefit analysis (CBA), cost-utility analysis (CUA), and cost-effectiveness analysis (CEA). The paradigms emerging from neoclassical welfare economics deserve a detailed examination. But first it is important to clearly define two concepts vital for this discussion: efficiency and utility. Efficiency is a broad term that has three major dimensions, usually approached in the neoclassical economics field.  • Technical efficiency addresses the question of efficiency in the production of goods (productivity), i.e., the ways of achieving the highest possible outputs given a specified amount of inputs. • Cost-effectiveness efficiency refers to the comparison among the technically efficient methods, and assumes that the one with the lowest cost of production is the most efficient. • Allocative efficiency refers to the distribution of goods in relation to the way individuals value and judge these goods. Allocative efficiency considers people’s preferences, or lapse it into economic jargon, the utility derived from the goods. The notion of utility, central to the field of economics, relates to the idea of individual satisfaction derived from a given service or good. It works as a heuristic tool and can be defined, according to Hargreaves Heap et al., as “a numerical representation of preferences”.12 (p368) Rooted 10 in the philosophical tradition of utilitarianism—the maximization of pleasure and minimization of pain—utility can also be understood as “the quantity that an individual should maximize or that society should help him to maximize”, as Blaug put it.13 (p337) Bearing these concepts in mind, we can discuss welfare economics, whose central principle is the Pareto Criterion. This principle states that a given allocation of resources is deemed allocatively efficient if it is not possible to reorganize the resource distribution in a way that makes at least one person better off without making someone worse off14. Although the Pareto principle has been widely employed in a conceptual way, it has two significant limitations in practical terms: 1) it does not indicate a single scenario of resource allocation as best (all possible allocations falling within the Grand Utility Possibilities Frontier are allocatively efficient); and 2) there is almost no case in the real world where a reallocation of resources happens without making someone worse off.  The theoretical solution proposed to these limitations was the Potential Pareto Criterion, also known as Expanded Pareto Improvement or Kaldor-Hicks Improvement (named after the economists who formulated this approach). It still relies on the notion that resources should be allocated in a way that maximizes the overall sum of individual utilities but considers that an allocatively efficient arrangement can be achieved if the gains of the winners are sufficiently large to allow the winners to compensate the losers for their losses and still be better off. According to the welfare economics literature, this compensation—supposedly measured in monetary terms—constitutes a hypothetical scenario that does not necessarily have to take place. Thus, the value of this compensation mechanism is precisely its existence in principle; its actual occurrence does not have to be accounted by economic theorists. 11 1.2.1 Cost-Benefit Analysis (CBA) The prescriptive approach for PSRA that emerges directly from this normative theoretical framework is CBA. In CBA, the investment options are assessed and compared in monetary terms by summing up the gains to gainers and losses to losers. The options with the highest values are recommended.15 In this paradigm, too, the compensation between gainers and losers does not have to actually occur; it is enough that it is possible in principle. CBA has been demonstrated to be a powerful tool, widely used by organizations in both private and public sectors in areas like transport and energy. The major strength of CBA is its straightforward method of determining if a project is worthwhile to receive investment. And because the opportunity cost is expressed in monetary terms, governments can in principle compare programs from different areas, allowing an efficient allocation of the overall budget across health care, education, defense, etc.16 The underlying idea is that prioritizing investment according to a rank of programs and services based on a CBA provides a way to maximize the sum of individual utilities, achieving allocative efficiency from the perspective of the Potential Pareto Criterion. Within this paradigm in health care, known as welfarism, health services would be seen the same as any other good produced within the economy, and its consumption would derive its utility. Consequently, the utility is not health itself, but the values individuals attribute to the health care services. These are translated in monetary terms, keeping the underpinning notion of cardinal utility. Thus, maximum efficiency in the health care system would be achieved by maximizing individual utilities derived from the use of services. Nonetheless, due to ethical and practical issues related to attributing monetary value to the life of individuals, CBA has been used in very restricted cases for decision making in PSRA within 12 health care systems. The few examples available in the literature include comparison of ambulance and rescue helicopters,17 and of two models of maternity care.18 Because of these issues, other types of economic evaluations, CUA and CEA, have prevailed in the field.   1.2.2 Cost-Utility Analysis (CUA) and Cost-Effectiveness Analysis (CEA) Extra-welfarism is an approach that addresses allocative efficiency in health care. This paradigm shifted the evaluative space from maximization of utility to maximization of health. In this respect, extra- means to provide this alleged expansion from the sole concept of utility to health itself. The underlying assumption of this paradigm is that all individuals present similar utilities for the same health interventions or health states. Extra-welfarist economists then assume that the role of health services is to increase the overall health of society, and they use it as the basis for cost-utility analyses to decide which investments should be chosen. An innumerable set of indicators of health have been used to analyze which practices and technologies lead to maximization of overall health, with a predominant emphasis on QALYs (quality-adjusted life years), a combination of years and quality of life gained through an intervention. Cost-utility analyses are based on the idea that resources should be allocated according to the ratio between the incremental gains of health and incremental costs (incremental cost-utility ratio [ICUR] or, more commonly, incremental cost-effectiveness ratio [ICER]) of each intervention judged against a threshold of willingness-to-pay (WTP). An efficient allocation of available resources would be one that only invests in interventions whose ICERs are below the WTP threshold, ensuring a maximization of the overall gains of population health. 13 In terms of the normative paradigm in decision making research grounded in welfare economics, CBA and CUA represent the prescriptive approaches that could, in theory, realize a comprehensively rational process of resource allocation solely based on efficiency. As we shall see in the next sections, organizing resource allocation in this manner in the health care system is neither desirable, practical, nor viable. The last common approach of PSRA in welfare economics is the CEA. Situated in the extra-welfarist world and focusing on maximization of health itself, CEA differs from CUA in that the outcome used is no longer a generic measure of health, such as QALY, that can be used across different health conditions; in CEA, interventions are compared in terms of their impacts on a specific outcome related to the disease in question. CEA is widely used and is a valuable tool to compare two interventions aimed the same health condition, but is less important in this discussion because it cannot be used to compare technologies and services from different areas of health care. The link between CEA and the theoretical pillars of welfare economics is weak and difficult to trace. This methodology is less relevant for our discussion of approaches to setting priorities and allocating resources in a broad sense.  Extrapolating welfare economics principles from CBA to CUA/CEA is highly contentious. Phelps and Mushlin19 argue that CBA and CUA/CEA are mathematically equivalent and that existing differences are due to reporting styles. Johannesson20 contends the idea that CEA/CUA represents a subset of CBA. In his view, “cost-effectiveness analysis should then be interpreted as an estimation of the cost function to produce health effects and all costs irrespective of who pays should be included”.(p489) On the other side of the debate, Bala et al.21 points out that, even after relaxing the set of assumptions, it is unlikely that CBA and CUA/CEA are equivalent in real-world settings. Dolan and Edlin22 developed an impossibility theorem that demonstrates that it is not 14 possible to establish a bridge between CBA and CUA/CEA within the realm of welfare economics. Donaldson23 in turn claims that the existence of a (near) equivalence depends on how the approaches are defined. If economic evaluations are defined on the basis of what is measured, it is reasonable to suppose an equivalence between CBA and CUA/CEA. If the definition is based on the question the analyst is seeking to answer, the hypothesis no longer holds true. It is not rare to find articles in which health economists describe other approaches of economic evaluation, such as cost-minimization analysis (CMA), cost-consequence analysis (CCA) and generalized cost-effectiveness analysis (GCEA). The present work does not consider these other approaches because can they be interpreted as variations of the general methodologies discussed above.   1.2.3 A critical overview of the prescriptive paradigms grounded in welfare economics The most striking limitation of the methods of economic evaluation, whether welfarist or extra-welfarist, is that they focus solely on efficiency. Although efficiency undoubtedly constitutes an important criterion for decision making, this is not sufficient in the realm of health care, where other criteria are relevant, such as the budget impact of interventions, the population subset affected, the epidemiological profile of the diseases, etc. The following subsections present some critiques24 of the two paradigms of PSRA grounded in welfare economics: welfarism and extra-welfarism. Relevant critiques pertaining to more methodological aspects, such as the design of health-related quality of life (HRQoL) instruments, the calculation of WTP thresholds, and the consideration of indirect costs, will not be addressed here because our focus is on more fundamental issues.  15 Criticism on the welfarist approach  ➢ Societal values are not necessarily represented in the notion of overall sum of individual utilities.  At least since Durkheim, one of the founders of sociology, social scientists have been aware of the theoretical standpoint that the society is not the same thing as the simple gathering of the individuals that compose it. Durkheim postulated that there are a lot of social facts, such as law, morality, statuses, roles, etc., that make the whole of society. Along these lines, collective representations (the social values, understandings, and notions), are not merely the creation of individuals’ intentions or the overall sum of individuals’ representations. The social facts are phenomena of the society as a whole and are engendered within the social environment. The collective representations cannot be studied through the simple sum of individual stances.25 The concept of utility may serve well for the consideration of markets where there is a predominance of more individualistic characteristics. In the health care system, though, social values lead individuals to behave differently from as predicted by the classical utilitarian and individualistic assumptions of microeconomics. People may value the health of certain groups more or may be more interested in equity than efficiency. Individuals consider humanitarian, caring, and compassionate positions about the delivery of health care. It means more than the simplistic acknowledgement of externalities on these judgements, like the inclusion of interpersonal effects in the function of individual utility. These values are embodied by the individuals and may or may not be reinforced by other social facts.26  16  ➢ Classical welfarist economics presumes an unresolved(able) separation between equity and efficiency. The classical Pareto principle states that the society should reallocate resources to maximize overall utility in a way that makes at least one person better off without making someone else worse off. The two major limitations of this thinking are the impossibility of determining a single best allocation and the empirical remark that virtually no rearrangement of resources produces a benefit to someone without inevitably hurting someone else. Many economists in the first half of the twentieth century sought to develop a more sophisticated theoretical foundation for the idea of utility maximization within Paretian terms. Kaldor and Hicks, who established a new paradigm for Paretian improvement, developed the idea that it is possible to achieve allocative efficiency if the gains to the winners are sufficiently large that they can compensate the losers for their losses and still be better off. Under these new terms, Kaldor elaborated that the process of achieving overall efficiency should happen in two steps.27 First, the economist should be concerned only with the economic decisions about resource reallocation aiming the maximization of utility. Within this paradigm, the social indifference curve is a downward straight line of slope −1, as we can see in Figure 1.7Error! Reference source not found.. The optimal point is the tangency between the grand utility possibilities frontier and the social indifference curve (point a). In this first moment, it does not matter if individual or group β derives much more utility than individual or group α. 17  Figure 1.7 Socially optimal point under utilitarian social welfare function.  Second, the fairness of the distribution of resources should be a concern for politicians and policy makers. As Blaug28 highlights, this approach “offers no opinion, however, on whether such compensation payment should be made; that is, it stops at the point at which it has enumerated the gains and losses to various individuals and ventures no judgement on how these gains and losses should be distributed” (p591). Finally, how do we compensate the “losses” within the context of the health care system? The question of redistribution within the realm of health services poses two main difficulties. First, it establishes a trade-off with the efficiency goal, since redistributing leads to a loss of overall welfare. Second, it gives rise to highly debatable issues such as the valuing of life, the possible differential values attributed to the lives of different individuals, and many other controversial ethical issues.  18 ➢ The use of individual utility to maximize social welfare in health care system may lead to   suboptimal or negative impact on health outcomes.   When individual utility is used as the maximand in the health care market, it is reasonable to think that individuals may find their best interests in interventions that negatively impact their health. Rice29, for instance, defends that individuals have to be protected from their own foolishness. At first glance, this seems the arrogant position of someone who presumes to know what is best for others and is aware that health policies may need to go against people’s best interest. But this paternalistic position may be actually necessary if the society cares more about overall health than about individual utilities. An illustrative example is the anti-vaccination movement. Its partisans prefer not to vaccinate themselves and their children, even though that puts the entire population at risk. Another example is the practice of quarantine during outbreaks of infectious disease. Individuals may not value quarantine, and the act of not isolating infected persons may have a widespread negative impact. Society, therefore, does not always achieve welfare maximization in the health sector through the consideration of individuals’ utilities. The lack of appropriate information, or even the legitimate right to stubbornness, constitute real problems for welfarist approaches in the health sector.  Individuals may not necessarily act “foolishly” or against their own health status, but may simply present an “inability to desire”, as Amartya Sen describes.30 Sen’s rationale is that people’s preferences are not inherent manifestations of individual selves, but are conditioned by the experiences and expectations of the social environment and social pathway we are raised in and live in. Thus, people in lower socioeconomic ranks have truncated expectations, circumscribed by a lifetime (or generations) of limited possibilities. Their horizon of well-being is likely narrowed by the embodiment of the disadvantaged social position. Consequently, a welfarist approach may 19 yield a distribution of resources that is unfair to unprivileged people. And this is even more significant in societies with greater socioeconomic inequalities. Criticism on the extra-welfarist approach  Before we move to examining the weaknesses and limitations of the extra-welfarist approach in economic evaluation and decision making within the health care system, it seems wise to examine the historical transition between these two paradigms and their distinctions. There is no consensus on the precise definition of extra-welfarism and its deviations from welfarism.29,31 As Coast28 highlights, the theoretical foundations of extra-welfarism were not developed prior to their implementation. Practices of economic evaluation of health services that can be deemed extra-welfarist are reported in the literature since the late 1960s, but health researchers including prominent British economist Anthony Culyer explicitly elaborated the theoretical structures of the paradigm only in the late 1980s. Culyer’s criticism on the application of neoclassical welfare economics in the health sector relied on two key points: • Social welfare is not independent of non-utility facets of resources reallocations. • Individual utilities are not independent of non-good aspects of individuals, as the welfarist model assumes. Adopting the notion of capabilities developed by Amartya Sen, Culyer proposed a theoretical framework that would go beyond the focus of individual utilities, contending the need to acknowledge other aspects in the pursuit of social welfare.32 The rise of extra-welfarism in health economics represents an attempt to operate a shift in the evaluative space within economic 20 evaluation and decision making toward a broader spectrum that could encompass capabilities and other aspects including health.  ➢ The alleged expansion of the evaluative space might actually have narrowed it. The addition of the prefix extra- was meant to cover this expansion of the evaluative space. Yet as Birch and Donaldson remark: “although EW (extra-welfarism) explicitly proscribes individual utilities as a measure of social welfare, it does not provide a clear specification of what does determine social welfare”.29 The passage from welfarism to extra-welfarism has been defended in theory as a way to move away from a narrow focus on utility toward including other characteristics important to individuals and the social welfare. But what has been seen in health economics practices deemed as extra-welfarist is an almost exclusive focus on health. As Coast notes, there is a disjoint between the theoretical expositions of extra-welfarism and its practical applications.28 In real-world extra-welfarism, health is valued in itself, no matter how differently individuals may value it in reality. Health is thus seen as “a physical entity of which individuals have a stock, and of which extra (gain) can be produced through the allocation of resources to health production”.28 If the focus on health solves some of the problems of using the neoclassical welfarist approach, it also seems to have shrunk the evaluative area. The excessive emphasis on health as the main outcome of the health care market ignores the well-being individuals obtain from non-health aspects of the use of services. The current practices of extra-welfarism do not take into account what Mooney called “process utility”, i.e., the satisfaction that individuals obtain from the service itself, such as the readiness of assistance with low waiting time, the kindness in the care received, etc.26 21 ➢ Decisions are made upon “non-explicit” considerations on equity Although extra-welfarism does not break with the welfare economics tradition of keeping the modus operandi based on maximization, Kaldor’s reasoning of Paretian welfare cannot be applied, since it is impossible to separate efficiency and equity within this new paradigm. The distribution of resources takes place concomitantly with their production and it is not possible to compensate losses with the exceeding gains obtained by the winners. After all, how can we compensate someone’s loss in health with someone else’s gain in health? One can argue that the extra-welfarist approaches do not appraise vertical equity. The quasi-equalitarian principle “a QALY is a QALY is a QALY” assumes equal weights to all individuals, regardless of the particulars.33 The focus on health maximization based on QALY or other similar indicators endorses the ethical stance that “the total sum of health produced within the health care system is what matters, no matter how that health is distributed”28 (p789) and may, as a result, perpetuate or even exacerbate inequalities in health. This becomes emblematic in the common critique that QALYs are based on an interval scale, assuming the “same gain” for absolute improvements, irrespective the initial health status (e.g., moving from 0.3 to 0.4 is regarded as equal to moving from 0.8 to 0.9). In Figure 1.8, a straight downward black line represents the social indifference curve for the standard application of QALY (or any extra-welfarist health indicator), as it concerns not the distribution of health, but only its maximization. However, it is likely that society members are inequality averse on average, and thus the appropriate social indifference curve would be the dashed curve, indicating that people are willing to lose some efficiency to mitigate inequality.34 Points a and b represent the same overall health gain, but depending on the relative characteristics 22 of Group A, the society may value A more than B, so that the actual societal indifference curve is better represented by the solid curve.   Figure 1.8 Societal indifference curves under different scenarios. Inspired on Norman et al., 2013.35   ➢ Its practices are inconsistent with Sen’s notion of “inability to desire” used against welfarism. Much of the critique of neoclassical welfarist approaches in economic evaluation in health is based on Sen’s idea of the “inability to desire” of individuals living in disadvantaged conditions. Nonetheless the alternative paradigm, extra-welfarism, continues to rely on utility theory to obtain measures of non-utility outcomes. The construction of a QALY, for example, involves individuals deemed as the best judges of attributing weights on different dimensions of health. In the words of Birch and Donaldson, “any limitations associated with the use of individual utilities as a basis for 23 social welfare measurement would seem to have implications for using individuals’ utilities as a basis for determining the weights used for EW measures of social welfare”.29 (p1127)  ➢ Controversial support of empirical evidence There is strong evidence that people are willing to lose overall welfare in the health care sector to prioritize particular characteristics of some patients and groups as well as to reduce inequalities in health. Coast,28 Birch and Donaldson,29 Mooney,26 and Brouwer et al.31 provide a monumental set of bibliographic references in this regard, although many studies show different results.  1.3 PSRA and Decision Sciences The term decision sciences refers to the study of the process of decision making within an organization. It involves research regarding virtually any kind of important decision, including military operations, clinical decisions, and financial transactions. Researchers in this field combine different types of knowledge (such as statistics, behavioural sciences, and microeconomics) to produce tools often based on quantitative modeling, to inform and operationalize decision making. Its origins trace back to the work of operational researchers in the 1960s and it has the textbook ‘Decisions with Multiple Objectives’ by Keeney and Raiffa36 as one of its basilar works.  In the literature about decision making in PSRA, CBA, CUA, and CEA are often placed under the umbrella of decision sciences techniques. This makes sense, but it is helpful to understand the fundamental differences among the prescriptive PSRA approaches found in the literature. The present work employs a conceptual scheme whose differentiating element is the prevailing underpinning normative paradigm. This is why the methods of economic evaluation of 24 health care programs/services previously described were discussed from the perspective of welfare economics, i.e., its basic theoretical pillar. The current section discusses two methodologies that emerge primarily from decision sciences: multi-criteria decision analysis (MCDA) and program budgeting and marginal analysis (PBMA). MCDA does not depict a single approach, but rather a set of techniques to be used when multiple objectives and criteria are involved in the decision-making process. In recent years, most PBMA applications have drawn on some form of MCDA for value assessment. The PBMA methodology has gained significant space not only in the priority-setting literature but also in the political and managerial sphere of health organizations.  1.3.1 MCDA “Definitionally, MCDA is an aid to decision making that relates alternative courses of action to conflicting multiple criteria requiring value trade-offs”.37 (p12) Its early foundations can be traced to the work of operation researchers in the 1970s36. The area has expanded significantly and the techniques developed under the umbrella term MCDA have been applied by governments and private organizations in a wide range of areas, including transport, energy, defense, and education. In health care though, MCDA has hitherto played a modest role despite the recommendation of researchers who are prominent champions of MCDA in this realm38-40. MCDA has gained more and more terrain in the priority-setting literature because it has the potential to address three basic problems of resource allocation in health care: • The objective of health organizations in allocating resources cannot be translated into a single measurement of efficiency, whether expressed in monetary net value or QALYs. Rather it involves a complex set of multiple objectives, including gains of 25 HRQoL and life years for the society as a whole, reduction of health disparities, strategic tackling of certain diseases, health professional well-being, political legitimacy, etc. Applying MCDA techniques in this context represents an attempt to evaluate and aggregate the impact of alternative interventions on multiple weighted criteria, providing thus a global indicator of impact. • Because health organizations do not have a single objective that pertains to all stakeholders, there are competing interests across sectors that often hinder the decision-making process. MCDA serves as a tool to bring together stakeholders representing distinct interests and summarize their views in a single approachable action plan. In the words of Regier and Peacock, “MCDA is a process that integrates objective measurement with value judgement while also attempting to manage subjectivity”.36 (p10) • Another intrinsic problem related to decision making in health care is the availability of evidence. Some consider that it is not possible to carry out consistent decision making in the absence of high-quality evidence. But MCDA provides a way to take into account “heterodox” types of evidence, i.e., those heavily based on subjective judgements or collected in a non-systematic manner (such as experts’ opinions and data obtained from a contingent service shutdown) while also weighting the value of such evidence within the big picture. Given that MCDA comprises several techniques and that health organizations have distinct values and objectives, the exact form of an actual MCDA exercise will depend on problem structuring and model building. Problem structuring refers to the overall process of appraising and establishing policy goals, relevant concerns, the scope of programs, stakeholders and their degrees 26 of power, acceptable actions, and so on41. This task is absolutely vital for a proper outline of the problem, identifying a relevant set of criteria, and adequate choices in model building, which denotes the process of making a behavioural model that can quantitatively translate decision makers’ and stakeholders’ value judgements. Despite the existence of distinct theoretical traditions to underpin MCDA models, two basic components are always present: preferences or value judgements are expressed for each individual criterion, and an aggregation model enables inter-criteria comparison and combination of preferences across the set of criteria. The most common theoretical strategy used in MCDA within the realm of health care has been value measurement42, which encompasses multi-attribute value theory (MAVT), multi-attribute utility theory (MAUT), and analytical hierarchy process. Other underlying theories that have been less commonly employed and researched in health care are outranking and goal programming or reference-level modeling36. With respect to decision making in PSRA, MAUT and MAVT have enjoyed prominence in structuring prescriptive and existing MCDA frameworks, including PBMA itself. MAUT is based upon expected utility theory43 and incorporates the risks associated with outcomes in its utility function, allowing considerations of uncertainty. However, it requires fulfillment of strong assumptions to derive the global utility function, which is very difficult to achieve. Thus, MAVT arises as a pragmatic approach to deal with these hurdles. Using a value function to characterize the outcome of the alternatives, MAVT does not allow for risky outcomes. Whereas in MAUT the utility functions sustain the expected utility hypotheses, MAVT functions preserve deterministic ordering. In fact, the relation between MAUT and MAVT is similar to that between CBA and CUA/CEA. Whereas MAUT is theoretically robust but difficult to operationalize, MAVT is practical but theoretically weaker. 27 For the aggregation component of elicited preferences, whether based on MAUT or MAVT, models are usually either additive or multiplicative. The conventional formulation for these aggregation approaches are:  Where a is a given alternative, u(a) is the overall utility associated with the alternative a, m is the number of criteria, ui(a) is the partial utility of alternative a according to criterion i, and wi is the weight chosen to translate the relative importance of criterion i across the m criteria. The mathematical representation for MAVT would be rigorously the same, except that instead of utility functions u, there would be value functions v. Despite what has been said based on literature, we contend that MCDA in itself is not a tool for PSRA but a value assessment framework. Given its inability to appraise opportunity costs, its application is directed to assessing the value for money of investing in specific technologies.  1.3.2 PBMA PBMA is a technique designed to aid decision makers in allocating resources across alternatives on the basis of their marginal costs and benefits8. It has been used in the health care sector at least since the early 1970s44 and has gained significant emphasis over the last two decades, being employed in a wide range of settings. Although PBMA presupposes a core of principles and procedures, real PBMA exercises can vary considerably according to the health organization’s structure, scope, objectives, and stakeholder involvement45. The fundamental principles underlying PBMA are opportunity cost and marginal analysis. Opportunity cost is explicitly appraised in this method because both alternatives of investment and 28 disinvestment are identified and judged against a defined set of criteria. Interventions are also evaluated according to their benefits and costs at the margin. The two concepts are interrelated in that the costs and benefits of potential areas of investment can be directly compared to the foregone costs and benefits of potential areas of disinvestment. A typical PBMA exercise46 starts with structuring the problem, establishing the aim and the scope, and identifying resources and costs of programs. Then an advisory panel is formed to involve stakeholders that represent different groups within the health organization. The panel chooses a set of criteria and their attributed weights. Options for investment and disinvestment (which involve service growth, resource release from gains in operational efficiency and from scaling back or interrupting some service) are identified. And based on the evidence available and the marginal analysis presented, the advisory panel scores the options on the basis of the defined criteria. This is the MCDA section of a PBMA exercise.  The results are then critically appraised, allowing appeals and review of the process. Ideally, an evaluation of the whole exercise is conducted, providing relevant considerations for further cycles of resource allocation based on PBMA. An ethical evaluative framework called Accountability for Reasonableness (A4R) has been often applied to assess legitimacy and fairness of PBMA exercises.47-49 Grounded on justice theories with a focus on democratic deliberation, A4R specifies four conditions for fairness: relevance, publicity, revision and appeals, and enforcement. Gibson et al. propose empowerment as a fifth condition50. The structure of PBMA steps may consider the implementation of A4R alongside the exercise40 to help foster perceived legitimacy of the process itself irrespective of the ultimate recommendations for reallocation. In Figure 1.9, the human figures with speech balloons represent public engagement. They are placed close to the stages in which the general public usually participates in PBMA exercises 29 (establishing and weighting criteria, voting/scoring alternatives, and reviewing decisions). Generally speaking, this involvement can be either direct, placing representatives of the general public in the advisory committee, or indirect, taking into consideration the elicited values and preferences of the public.   Figure 1.9 PBMA steps.  The notion of public engagement used throughout the present work encompasses both consultative mechanisms (those with a unidirectional flow of information from the public to the system) and participatory mechanisms (those that establish a dialogue between the system and the public). The differentiation between consultative and participatory mechanisms is borrowed from Rowe and Frewe,51 who developed a theoretical framework that distinguishes among three types 30 of public engagement (communication, consultation and participation) according to the flow of information established by the sponsor (the party responsible for the service) and the public. In PBMA exercises and in the decision-making processes of setting priorities in general, public engagement normally means obtaining some input from the general public, which can take place in a consultative or participatory manner. PBMA has been recommended to be based on multi-attribute utility theory.46,52 Normally, MAUT additive models are used to aggregate the scores of interventions across multiple criteria.  1.3.3 Critical Overview of Prescriptive Approaches Grounded in Decision Sciences In spite of its substantial growth in the health care sector, MCDA approaches have been the target of some criticism. The general points raised here apply both to PBMA and other MCDA tools used in PSRA. A common critique on MCDA methodologies is that the uncomplicatedness of scoring, weighting, and aggregation procedures creates an illusion of transparency53. Furthermore, the seeming numerical precision of results may be misleading, giving a false impression of strong scientific objectivity54. Another common problem of MCDA approaches is the inadequate treatment of opportunity costs. Campillo-Artero et al. observe that there is a “confusion between prices, expenditures, harms, accounting cost and opportunity cost (…) amongst the plethora of components in MCDA”.55 (p148) Even in PBMA exercises in which participants are explicitly encouraged to discuss the opportunity cost of interventions, these are often simply read from accounting that despite its managerial relevance does not necessarily capture opportunity costs. 31 Approaching PSRA with MCDA tools also presents problems related to normative validity. Given the employment of ad hoc assessment procedures, with context-specific sets of criteria and weights, the normative validity of the chosen MAUT or MAVT function has to be verified every time56. The most common violation of normative principles of MCDA is double counting or the lack of interdependence among criteria. In most scenarios, the scope of each criterion is not clearly or sufficiently defined and overlaps are very likely. Criteria like social impact, reduced burden, therapeutic effects, and safety effect have possible intersections, requiring difficult demonstrations of interdependence55. A systematic review has revealed that double counting of cost is very common, having an individual criterion for cost and another for cost-effectiveness57. Moreover, Airoldi points out that direct weighting of criteria is inconsistent due to the openness of interpretations and that the decision sciences literature strongly recommends swing weights to discriminate the relative importance of criteria56. The corollary of this argument is that the vast majority of priority-setting exercises based on MCDA lack normative validity, including PBMA. In the view of Airoldi56, a further violation takes place in MCDA tools applied in priority setting. According to the author, “in the context of resource allocation the MCDA literature clearly recommends ranking alternatives according to their cost-effectiveness, defined as the ratio between a (multi-criteria) benefit score and cost”.(p.42) Yet the references underpinning this statement do not demonstrate a theoretically strong reason for the impossibility of applying MCDA without ranking alternatives on the basis of cost-(multi-criteria)benefit ratios. In reality, the ISPOR (International Society for Pharmacoeconomics and Outcomes Research) taskforce report on MCDA58 highlights that using cost-benefit ratios is extremely problematic, they inadequately 32 combine benefits measured in different scales and are less sensitive to benefit impact assessment of alternatives with little impact on cost assessment. In our view, the relevance and strength of these techniques grounded on decisions sciences relies on the fact that, despite the critiques presented, no critique precludes dealing with it to carry out a successful implementation. Although MCDA and PBMA are far from being perfect, these approaches provided very practical tools for decision makers.  1.4 Current Practices of Priority Setting To this point we have discussed the most important prescriptive approaches of PSRA found in the literature of health economics, presenting their normative underlying paradigms and their limitations. As was demonstrated, it has been a very contentious area and no methodology has gained the status of gold standard given their practical, theoretical, or ethical issues. The process of knowledge translation has a tendency to vary across countries, states/provinces, health authorities, and hospitals. This section presents a brief summary of the evidence about existing PSRA practices worldwide. The data were obtained from a literature review on empirical studies describing formal PSRA frameworks, a grey literature search, and other published reviews and articles relevant for broader context.  1.4.1 Literature Review on Applied PSRA Frameworks A literature review was conducted focusing on frameworks used for PSRA in developed countries. The literature review search strategy is outlined in Appendix A. We sought to identify 33 PSRA frameworks that have been used in practice to influence health care decision making. In total, 1,472 titles and abstracts were found, once duplicates were removed. We then used a two-step screening process. First, the 1,472 abstracts were screened by one primary reviewer and two secondary reviewers based on the inclusion/exclusion criteria found in Appendix B. Reviewer 1 (BVS) reviewed all 1,472 articles, Reviewer 2 (CM) reviewed 227 articles, and Reviewer 3 (FD) reviewed 71 articles. The agreement rate between Reviewer 1 and Reviewers 2 and 3 was over 95%. Discrepancies were handled conservatively, resulting in 85 abstracts initially being screened in. A senior reviewer (CM) then took a further detailed read of the abstracts and pared the list down to 23 relevant articles. This second stage screen excluded papers that initially appeared to be a empirical studies but in fact discussed some aspect of priority setting without an actual case study or implementation of a framework. We then applied a data extraction tool (see Appendix C) to identify the relevant information. The 23 papers that met the inclusion criteria and were subject to data extraction provide relevant information from 10 countries: Australia52,59,60; Austria61; Canada46,62–67; Israel68; Korea69,70; New Zealand71; Norway72; Sweden73,74; UK39,75–79; US80. This list of countries gives us a good representation of frameworks developed and implemented in health care systems of the industrialized world. A summary of the 23 papers is provided in Appendix D  Of the 23 studies, eight used PBMA, eleven described a formal process that explicitly used a criterion-based approach to value assessment (with two of these using A4R), three could be described as HTA within a broader framework for decision making, and one was a modeling exercise using multiple criteria. PSRA frameworks have been used in virtually all possible levels of governance and administration (national, state/provincial, regional, hospital) and have served for the prioritization 34 of a wide variety of health services (community care, mental health, drugs reimbursement, immunization, and specific diseases). A number of key process characteristics were identified from the selected papers: • Frameworks employ a variety of criteria to base decisions, not necessarily using a consistent MCDA technique to judge alternatives and summarize preferences. • A variety of stakeholders were involved in almost every case including administrators, government officials and clinicians. • Decisions were typically not subject to review by external stakeholders (i.e., the public). • It was generally not clear from the papers how requests for funding were initiated, nor was it specified if current spending was reviewed as part of the process of decision making (although PBMA, for example, implies such a review). • Several types of data were reported to inform decision making, including published literature, clinical opinions, economic evaluations, HTAs, and data on disease prevalence. • There was limited reporting of evaluation in these studies. Note that because of the objective and design of this review (which focuses on revealing existing formal PSRA frameworks, thus including only papers that provided a full description of a framework employed in a real-world setting), it is likely that articles presenting relevant information on current practices of decision making in priority setting have not been captured.  The next subsection discusses evidence found through a grey literature search in websites of relevant organizations (like ISPOR, IHEA, HTAi, CADTH, NICE, etc.) and found in other existing literature reviews. 35 1.4.2 Expanding the Horizon: Findings from a Grey Literature Search and Other Relevant Literature References Several systematic reviews on PSRA are available in scientific literature, with distinct nuances. Some focus on priority setting at the macro or meso levels, some focus on hospitals, others emphasize frameworks implemented in low- and middle-income countries, etc. So, the results are diverse and provide a dispersive collection of case studies. We do not intend to provide a meta-analysis of all evidence available, but rather touch upon the most relevant aspects for our work. In a review focusing on resource allocation and disinvestment, Polisena et al.81 found 14 studies, all in developed countries. Two of them reported use of HTA to propose disinvestments whereas the majority described applications of PBMA. Studies reported initiatives at the national level (basically the HTA approaches toward disinvestment), at the regional level (health authorities) and at a single health care unit or department. A review carried out by Barasa et al.82 with particular interest in formal PSRA initiative in hospital settings revealed a small number of studies, mostly based in developed countries. Almost all these exercises aimed allocation of resources among hospital departments (usually based on PBMA or MCDA) or decision making about acquiring specific technologies (employing CEA/CUA). Hipgrave et al.83 and Wiseman et al.84 conducted systematic reviews emphasizing PSRA endeavours in low- and middle-income countries. They both point out the relatively little information known about practices of decision making in priority setting within health care systems of peripheral countries in comparison to western nations. In both reviews, no case of PBMA was found in these settings, although Wiseman et al.84 found a study in Papua-New Guinea 36 they consider resembles a PBMA approach. The majority of reports identified by Wiseman et al.84 involve global or regional efforts of GCEA using cost per DALY averted, such as the World Health Organization (WHO) frameworks to identify the most cost-effective interventions to achieve the Millennium Development Goals and Universal Coverage. And in almost all of these cases, the exercise did not consider options for investment and disinvestments. Both reviews identified ranking of alternatives based on MCDA tools, like initiatives in Ghana and Nepal. In addition, other less common approaches were identified, such as case studies of multi-criteria frameworks based on A4R in Tanzania and Uganda and the applications of the investment case approach in India, Indonesia, and Philippines. Another review of the published literature conducted by Cromwell et al.85 to find examples of “real-world” priority-setting exercises that used explicit criteria to guide decision making, identified several case studies, mainly in Canada and the UK. The most common approaches identified were PBMA and MCDA applied in various settings, e.g., national level, health authorities, hospitals, and for specific disease programs. Whereas priority-setting approaches based on decision sciences have been shown to be useful and versatile in allocating scarce resources in a wide variety of levels of governance and administration (hospital, regional and national levels86) as well as within diverse areas of care (like mental health87, heart diseases88 and community care62), approaches grounded on welfare economics have usually been employed only to make recommendations regarding the coverage of specific technologies. CEA and CUA have been widely used within this restricted context of priority setting by HTA agencies all across the globe, such as CADTH in Canada, PBAC and MSAC in Australia, PHARMAC in New Zealand, and NICE in the UK. 37 Theoretical PSRA frameworks without evidence of empirical use are not rare in the literature. Angelis and Kanavos propose a MCDA-based approach called Advanced Value Framework, which uses five domains of criteria (burden of disease, therapeutic impact, safety profile, innovation level, and socioeconomic impact) and a MAVT function to aggregate scores.89 Another framework based on MCDA that was developed to inform decision making in health care and has been proposed for PSRA is EVIDEM90. Airoldi et al.91 propose the socio-technical allocation of resources (STAR) for that purpose. For them, CEA/CUA constitute theoretically robust methodologies and present paramount practical issues, whereas PBMA has a great pragmatic value but is theoretically problematic. STAR was proposed as way to fill this identified gap in the priority-setting literature, being theoretically strong and highly useful for decision makers. STAR employs models to appraise the “cost-effectiveness of all interventions considered for resource reallocation by explicitly applying the theory of health economics to evidence of scale, costs, and benefits, with deliberation facilitated through an interactive social process of engaging key stakeholders”.91 (p965) In this “social process”, the involved stakeholders produce missing estimates of scale, costs, and benefits of the interventions, create visual representations of their relative cost-effectiveness, and then interpret them. STAR was used by a Primary Care Trust (a local NHS planning agency) to allocate a fixed budget in 2008 and 200991. Relying upon the evidence found in previous literature reviews on priority setting, Hipgrave et al.83 comment that “the overarching conclusion was that even in high-income settings where participatory, accountable and rational approaches to health priority setting should be achievable, the process and outcomes of such exercises have been unsatisfactory”.(p.192) The evidence from the published literature is usually about specific case studies and very rarely reports a systematic and continuous use of formal PSRA frameworks. A 2017 review aiming to understand 38 “how have systematic priority-setting approaches influenced policy making” concludes that “while systematic approaches have the potential to improve healthcare priority setting; most have not been adopted in routine policy making”. 92 (p 937) And Tsourapas and Frew found that PBMA applications specifically have shown much success across countries and particularly in Canada.93 Thus, it seems that, as it is often the case, individual details on implementation, and indeed the individuals involved, play a huge part in achieving success or not. 39 Chapter 2: Study Objectives and Research Questions The scientific literature offers a rich view of a multiplicity of settings where formal frameworks of PSRA have been deployed. This indicates the growing relevance of the topic in health care systems worldwide. The available evidence merits two observations. • The majority of PSRA studies have been circumscribed in the context of high-income countries, despite important initiatives being studied and proposed elsewhere. Even within the industrialized world, developing and implementing formal and explicit approaches for PSRA decision making does not receive the same level of attention across countries. This is absolutely reasonable given the variate levels of fiscal constraint, the diversity of institutional and legal structures of health care systems, and the presence of active champions of the topic within some national health research communities. Whereas PSRA studies abound in countries such as the UK, Canada, and Australia, few or none from countries like Italy, Switzerland, or France are published in peer-reviewed English-language journals. • The pieces found in the literature usually depict case studies of specific scenarios where a given PSRA methodology was tested or implemented. To our knowledge, there is no published study aiming to map existing processes of priority setting in a more systematic manner. In spite of the virtual absence of information regarding current practices of PSRA in health care systems in many countries, politicians, managers, and health professionals everywhere are determine priorities for investing scarce money, even when no formal or explicit process takes place. And in all decisions, some sort of rationale is necessarily implied, whether historical allocation, needs assessment, or any other non-expressed justification basis. 40 There is a gap in the literature between what is known and what is practiced. On what basis do hospitals allocate fixed budgets across internal sectors? How do health authorities decide to channel resources for a new capital investment? What rationale do state, provincial, or national governments employ to fund a new public health program or to cut an existing one? Are there formal, explicit, and transparent strategies underpinning such decisions in health care systems? Which stakeholders are normally involved? What types of evidence are used to inform decision making? Although some of these answers in respect to some countries can be found in the literature, there is no data whatsoever—and certainly no single source—for the majority. The present work seeks to address this gap in the literature. But this question has to be properly outlined so that it becomes approachable. First, it is necessary to establish a list of places to be studied given the obvious impossibility of any universally comprehensive attempt. Second, considering the lack of data in the scientific literature, the nonexistence of any relevant international database, and the fact that relying on grey literature pieces to understand organizational/institutional practices in non-English-speaking countries would represent a Herculean task, innovative methodological approaches are necessary. This work’s overall objective is: to describe existing processes of PSRA in publicly funded health care systems of selected high-income countries using an online qualitative survey with decision makers and researchers. Given the higher number of PSRA studies in high-income countries and their expectedly higher capacity building, our study intends to describe practices of priority setting in health care systems within these contexts. Also, to have a consistent and coherent set of health organizations considered in the study, we chose to include only decision-making practices occurring in publicly funded health care systems. A detailed overview of the study design is provided in Chapter 4. 41 Online qualitative survey was the solution chosen for the methodological challenges posed by our research problem. A rigorous and general reflection on the theoretical foundations for the study design is developed in Chapter 3. This study does not aim to provide a comprehensive mapping of PSRA strategies in any given context. The target is to generate more evidence on how decisions are commonly made across different levels of governance and administration (national, state/provincial, health authority, and hospital) in publicly funded health care systems in high-income countries. Rather than building a comprehensive map of existing practices, we aim to craft a mosaic of current procedures and rationales. Thus, our specific objectives are as follows: • To provide a detailed report of common PSRA strategies across numerous countries. • To describe in some detail the specific frameworks uncovered through the survey (both those previously unveiled and those known but not systematically documented). • To compare processes featured in the survey. • To understand the common barriers and facilitators for the implementation of formal processes in the view of participants. 42 Chapter 3: A Theoretical Reflection on Methodological Aspects: Online Qualitative Surveys as a Qualitative Descriptive Approach in International Comparative Studies 3.1 Preliminary Considerations International comparative studies can contribute enormously to public policy research. Understanding the distinct manners in which a given issue may be tackled provides better comprehension of the problem itself as well as useful insights on the design of institutional responses. However international comparisons pose demanding data-collection challenges. Much of the work performed in such studies relies on existing databases kept by international agencies, third sector organizations, or research institutions (e.g., World Bank, OECD, WHO, UN). For studies on some topics, however, there is no reliable database from which the necessary information might be retrieved. Thus, it is vital that alternative, methodologically rigorous approaches emerge. Here we discuss the use of online qualitative surveys as a tool to overcome the difficulties of conducting comparative studies on public policies in different national jurisdictions. Instead of relying on institutional websites, publicly available policy documents, or well-established databases to understand how certain policies have been addressed, researchers can question key stakeholders (such as policy makers, public servants or researchers involved with the topic in question) from each relevant national context to retrieve essential information directly from them. Because researchers cannot anticipate the diversity of responses, it is necessary to have a tool that is open enough to allow any type of information to be captured; qualitative methodologies 43 are highly appropriate for this purpose. However, since the main objective of this type of research is to obtain comparable information from a potentially large number of different countries, the methodology needs also to pre-structure responses to a sufficient extent to allow for viable and efficient data reduction and analysis. And for this particularity, a qualitative description approach through the use of online qualitative surveys, i.e., structured questionnaires with open-ended questions, seem to be an interesting solution. This chapter discusses the theoretical reasoning underlying the qualitative description approach—presenting as an adequate solution to the tensions noted above—as well as practical insights on the development of such work within the realm of international comparative studies on public policies.  3.2 Theoretical Reasoning According to Sandelowski,94 basic or fundamental qualitative description differs from other types of qualitative research such as grounded theory, ethnography, phenomenology, or narrative analysis, in that it is—as the label suggests—descriptive rather than interpretive in focus. This does not mean that a qualitative descriptive approach lacks interpretive efforts or that it intends a supposedly neutral depiction of reality. Qualitative description represents the methodological category that has the least level of inference among the qualitative methods, one that allows “the reading of lines, as opposed to reading into, between, over or beyond the lines”.95 p78 But that does not indicate a derogatory judgement or any sort of depreciation. Rather, qualitative description is viewed as a valuable end-product in itself, and not simply as an entry point. To better comprehend the role of qualitative description, it is interesting to look at Alasuutari’s powerful insight on the distinct manners of viewing the nature of data.96 He 44 differentiates between “factist” and “specimen” views of data. The factist paradigm comprehends data as a somewhat accurately truthful indicator of reality. On the other hand, the specimen approach understands data as part of the reality that researchers should also carefully study. Sandelowski points out that qualitative description is normally compatible with a factist paradigm. This study tries to expose a more nuanced stance. We argue that a qualitative descriptive study is indeed forged with strong hues and textures of the factist paradigm; after all, the objective of this type of scientific investigation is to provide a “report” of the “facts”, i.e., a description of the elements upon which most observers would readily agree. However, it does not mean that researchers involved with qualitative description should necessarily work within a positivist or post-positivist framework. Every researcher pursuing qualitative descriptive endeavours should be aware that qualitative data is always contingently constructed, in that what is called data is the result of the researcher’s efforts to understand a given reality (with all the implications of the subjectivity there involved) and the narratives and perceptions of the interviewees. Thus, the present appraisal of qualitative descriptive studies does not align with Sandelowski’s simplistic adhesion of qualitative description to the factist view of data. Our argument is that the idealistic affiliation of qualitative descriptive efforts with the factist paradigm takes place overall due to the methodological reasons, as a way of creating boundaries between this attempt to “report” the “almost unquestionable” reality from the traditional qualitative methodologies that seek to unveil the discursive pillars of the social reality, the voices of populations silenced by oppression and domination, etc. In actually doing qualitative description, researchers should be cognizant of the inevitable constructive nature of what is called data. One role of the researcher is to be able to manage this nuanced comprehension of reality while seeking what would be considered facts by the vast majority of observers. 45 The novel methodological proposal presented here uses online surveys as a way of operationalizing qualitative description in international comparative studies, allowing retrieval of information on governmental/institutional efforts to develop, implement and evaluate public policies based on the reports of involved stakeholders who draw on their own situated experience and knowledge. Within this context, we offer a reflection on Kvale’s metaphor on the role of a researcher, which presents two ideal types: the researcher as a miner and the researcher as a traveller.97 For the researcher as miner, the reality is out there waiting to be discovered, and their job is to find the precious stones, the gems: the pieces of reality that have value in a given social setting. Thus, the miner-researcher operates under a predominantly positivist framework. The researcher as traveller, however, experiences the reality personally; there is no separation between what the traveller can tell us about the reality and the reality itself. Therefore, the traveller-researcher is not a collector of pieces of information, but is the constructor of the pieces. This type of researcher operates in a predominantly socio-constructivist paradigm. Kvale’s metaphor is indeed insightful to reflect on the role of the qualitative researcher in general, which should not be understood as either miner or traveller, but as an enterprise with a predominance of one or the other paradigm. However, we suggest that another metaphorical representation can be even more powerful, particularly for qualitative descriptive studies. We propose that the role of a researcher in qualitative descriptive efforts is that of a composite-sketch artist. The underlying idea is that this artist’s role is to reconstruct “reality” based on the reports of the witnesses. In other words, the artist’s duty is to draw a picture that accords with the memories of the witnesses, rather than substituting his/her own speculation. Contextualizing this for the field of policy research, the role of the researcher conducting qualitative descriptive study is to retrieve information from stakeholders about their own experiences with the institutions to reconstruct the 46 governmental designs of public policies or organizational management systems. The method employed must faithfully draw a picture on which most of the interviewees from a given setting will agree. Moreover, this metaphor leads us to reflect on the concepts of descriptive validity and interpretive validity, as elaborated by Maxwell,98 within the context of the present work. For Maxwell, descriptive validity refers to the accurate, “correct”, or faithful use of the factual aspects of data. It is predominantly related to the elements “pertaining to physical and behavioral events that are, in principle, observable”.98 p286 Thus, the majority of work conducted in qualitative descriptions is almost exclusively circumscribed to this level of interpretation and validity. Descriptive validity deals with how the composite-sketch artist treats the information provided by the witness. It does not necessarily mean that the researcher would actually work as a copier or a mere reproducer. Indeed, Maxwell, goes further and talks about a second level of descriptive validity, which deals with the inferences that the researcher makes. Qualitative description is not a free-inference approach, but rather the methodological work of least inference among the other categories of qualitative work. Overall, descriptive validity is represented by the faithfulness of use of the “factual” information provided by the witness. In the context of international comparative studies, the researcher has to keep close to the surface of the information provided by the stakeholders to correctly describe the local systems of management or public policies. Although the fundamental concern of qualitative descriptive studies is to provide a sort of report of the events, institutional structures, physical elements, and commonly observable behaviours, it is also important that researchers account for the meaning of these things for the people studied. It does not signify that qualitative description will dive deeply into the web of meanings in which subjects are constantly moving, but there must be at least a conscious 47 movement to acknowledge this phenomenon to obtain a valid drawing of the reality. This domain of validity is what Maxwell call interpretive validity.98 Thinking of our proposed metaphor, the composite-sketch artist needs to take into consideration what the pieces of information provided by witnesses can mean to them. A witness may say that the perpetrator of a crime has big green eyes, but although this constitutes fact, it does not give the artist enough information to draw the eyes in a way the witness would recognize. It is necessary to account for some level of interpretive data, as long as it indeed helps with reconstruction of reality. At this point we have already discussed that qualitative description is a category of qualitative methodology distinct from the traditional qualitative methodologies (e.g., ethnography, grounded theory, phenomenology, and narrative analysis), being less interpretive and being concerned with the description of factual information (i.e., the pieces of data on which most subjects will agree.) Put another way, in qualitative description, “language is a vehicle of communication, not itself an interpretive structure that must be read”,94 (p336) unlike more inferential approaches. Nonetheless, as Sandelowski argues, this does not mean that qualitative descriptive studies do not have elements from these other categories. Qualitative description may have hues, tones, and textures of these other methods, depending on the nature of the phenomenon studied. In the case of international comparative studies of public policies, we argue that, above all, qualitative description must have overtones of grounded theory, given the enormous diversity in the data obtained from different national/regional scenarios. The data analysis requires minimal preconceptions and previous elaborations, leaving the process of “drawing the big picture” to emerge almost entirely from contact with the data. It is important to seek the description of each local context in their own terms, framing their systems of management and development of public 48 policy with their own words. Grounded theory may provide significant theoretical support and practical tools to conduct qualitative descriptive surveys across countries. Fundamental or basic qualitative description ranks as the lowest inferential category of qualitative methodology, but it should not be understood as a low-quality approach or solely as an entry point to really deep research. In the words of Sandelowski, “there is nothing trivial or easy about getting the facts, and the meanings participants give to those facts, right and then conveying them in a coherent and useful manner”.94 p336 Qualitative descriptive studies are generally eclectic studies that, even though they represent a defined category, situate much of their work in the interface between typically quantitative approaches and the traditional qualitative methodologies. The next section will focus on the more practical details of developing an online survey as a qualitative descriptive endeavour for international comparative studies.  3.3 Practical Aspects of Study Design Here we discuss some of the methodological issues of developing an online qualitative survey within the context of international comparatives studies of public policies. We do not presume to cover all relevant issues, but do address those that seem more relevantly envisioned at this point of the study. The more practical problems pertaining to qualitative description will arise in the study itself: “the story of the methodology is the story of the study”, as Agar puts it.99 (p19) As Sandelowski points out, qualitative description is a distributed residual category and, as such, it makes visible the “porous lines between qualitative and quantitative description (…) and between the erosion and re-invention of method”.95 p82 This category of inquiry may incorporate elements from quantitative and qualitative methodologies and thus serve as an innovative research tool. In the particular case of obtaining information from different national contexts, a qualitative 49 description approach allows collection of data that will be analyzed not only from the perspective of traditionally qualitative methodologies, but also from a more quantitative lens, making possible a quasi-statistical analysis of content, providing an overall summary of the findings.  3.3.1 Sampling For this type of research, we propose a combination of some of the purposeful sampling strategies described by Patton.100 Unlike probability sampling, whose main purpose is generalization, purposeful sampling seeks to select the information-rich cases, i.e., “those from which one can learn a great deal about issues of central importance to the purpose of the research”.p169 Patton’s proposed classification system lists several purposeful sampling strategies, each of which has a particular logic serving a particular research objective. His list includes, among others, extreme or deviant case sampling strategy (which emphasizes cases that are rich in information precisely because they are unusual or special) and intensity sampling strategy (which focuses on including the cases that “manifest the phenomenon of interest intensely”). The strategies particularly relevant to compose our proposed sampling scheme are: • criterion sampling (use of specific criteria to select cases); • homogeneity sampling (applied to ensure comparability); • maximum variation sampling (which targets capturing the central themes that cut across a significant diversity of scenarios); • sampling politically relevant cases (used to ensure apprehending the fundamental topics pertinent to knowingly relevant scenarios); • snowball sampling; and • convenience sampling. 50 At the initial level of sampling, the country level, it is important to ensure comparability among selected nations. This ensures the validity of the quantitizing stage of data analysis, which should report a numerical summary of the data and observed patterns. For this level of sampling, it is important to combine two strategies: homogeneity sampling and criterion sampling. For instance, we could select countries by the number of inhabitants or we could include countries only above or below a given value of GDP per capita. Subsequent sampling may focus on strategies to guarantee that there is meaningful variation within the sample and that politically important cases are not missing. For example, it may be appropriate to include cases with distinct institutional models, such as countries with parliamentary and presidential systems, or countries with centralized and decentralized responsibilities for a given public service. Once the countries to be included in an international comparative study are determined, researchers need to identify individual informants. While an explicit sampling frame (e.g., a directory of government department heads) may sometimes be available, strategies such as snowball sampling and convenience sampling may be required to make the study viable. Figure 3.1 depicts this process of sampling within the context of using a survey as a qualitative descriptive tool to study public policies across countries. 51  Figure 3.1 Purposeful sampling strategies for qualitative description in international comparative studies.  3.3.2 Data Collection As mentioned, the objective of data collection within this context is to obtain information about the institutional design of public policies. Researchers will rely on the reports of participants to reconstruct the “reality” of each national scenario. At this point, the survey is the basic tool for collecting data. Respondents located in each national context would be invited to participate in an online qualitative survey. Researchers must circulate a structured survey instrument that allows participants to express their ideas in their own terms, but within a format that facilitates or guides the process of data analysis. Considering that qualitative description aims to record the facts, or describe the things upon which most people would readily agree, the research team needs to develop an effective survey to engage participants in the description of the essential policy elements without narrowing their possibilities of responses. 52 3.3.3 Data Analysis For Sandelowski,94 the analytic strategy of choice in qualitative description is Qualitative Content Analysis. This is a dynamic analytical tool intended to depict the informational content of the data that, according to Morgan,101 is essentially different from the well-known quantitative content analysis for two main reasons: • In qualitative content analysis, the codes are generated from the data (i.e., derived inductively) in the course of the study. This does not mean that researchers cannot have pre-existing conceptions about the topic. In fact, because researchers are typically highly involved with the investigated theme well before the beginning of the study, they have at least a reasonable level of knowledge and often strong positions about the critical issues. However, the analysis of the data must be circumscribed to the sphere of the qualitative data as much as possible. In the context of international comparative studies, this is an important manner to pursue validity, because it can force researchers to stay “close to the surface of what has been said”, instead of making assumptions that do not hold true for other national scenarios. • In qualitative content analysis, the quantitizing phase (the stage when the coded data elements can be numerically organized) allows researchers to go beyond mere summarization of the manifest data (the information readily retrievable from the raw dataset). Researchers conducting qualitative content analysis can use the numerical representations to infer associations and tendencies and make predictions, thus providing an overall understanding of the latent content data (i.e., the type of information that requires a deeper analytic effort to be revealed). 53 Hsieh and Shannon102 argue that there are three different approaches to qualitative content analysis: conventional, directed, and summative. Within their theoretical framework, the conventional approach is the type of analysis that does not rely on preconceived categories, allowing the codes to emerge from the data. In fact, the analytical practice that epitomizes qualitative content analysis in Morgan101 represents only one particular type of qualitative content analysis in the scheme used by Hsieh and Shannon.102 This process, also understood as inductive category development, tends to be used when the research purpose is to describe a phenomenon. In this approach, research theories are only addressed in the discussion section of the study. Its advantage is in gaining direct information from participants, but it possesses the downside of failing to provide thorough comprehension of the context. In directed content analysis, though, Hsieh and Shannon102 explain that researchers explicitly rely on existing theories or prior research to identify key notions for the initial coding process. Besides open-ended inquiries, targeted questions about known relevant categories may be employed. Coding can begin either with immersion in the data to allow non-preconceived codes to surface, followed by discussion based on existing theories, or with the use of predetermined key concepts or categories to analyze the data. In both cases, new codes can always emerge from the data and existing research knowledge is explicitly employed, whether to structure the data analysis or to refine a primary conventional analysis. The directed approach is commonly recommended when the research objective is to extend or improve existing theories. Nonetheless, it displays three main limitations: researchers analyze the data with a knowledgeable but significant bias; in responding to probe questions, participants might be guided to certain pathways to agree with researchers; and there is a risk of putting too much emphasis on existing theory and failing to capture important contextual elements. 54 The summative approach of qualitative content analysis starts with locating and counting specific words or pieces of content. “This quantification is an attempt not to infer meaning, but, rather, to explore usage”.102 p1283 This methodological approach differs from quantitative analysis of qualitative data in that it not only focuses on quantification of manifest data elements but also seeks to reveal the underlying meanings of words, expressions, and content. In other words, it includes latent content analysis. In qualitative descriptive studies on public policies and services within the realm of international comparative studies, the data analysis is expected to consist of a mix of all three approaches of content analysis described by Hsieh and Shannon.102 Its nuances will vary according to the research purpose, its scope, and the level of previous of knowledge. It is important to ensure that the methodology allows unanticipated topics to emerge and contextual information to be comprehensively captured. A tense trade-off between reliance on existing knowledge to structure data analysis and open immersion in the data will pervade the research project and each researcher has to decide on when and where to lose and to gain. Another reflection pertaining to the data analysis in this case is based on the typology proposed by Sandelowski and Barroso103; see Figure 3.2. They describe five levels of findings in qualitative data, from the lowest level of data treatment to the highest level of interpretive efforts. Sandelowski defends that in this type of methodology, the data analysis is done at the level of thematic survey.95 This means that there is an attempt to describe the data according to the topics emerging in or from the data, without much interpretive analysis as in grounded theory or phenomenology, but with a higher level of data treatment than used in quantitative content analysis, which would fall under the category of topical survey. We argue a slightly different “location” for qualitative description. Although its primary objective is to describe the “factual” 55 data with a low level of inference, ultimately it is not possible to conduct a qualitative descriptive without some level of interpretive analysis. The researcher as composite-sketch artist has to engage in some sort of interpretation of the expressions used, such as “big scary mouth”, “penetrating black eyes”, “repugnant face”, etc. Thus, data analysis in qualitative descriptive studies tends to have an imprecise location, moving between thematic survey and conceptual description (Figure 3.2). The degree to which one of this type of finding may appear will also depend on the type of topic studied. Researchers working with the data must be aware of the problematic boundaries between interpretive analysis that is pertinent insofar as it helps the description or the intersubjective agreement of observers, and interpretive analysis that goes beyond and that would actually require more in-depth methodologies, such as grounded theory, phenomenology, narrative analysis, or ethnography.  Figure 3.2 Position of our qualitative descriptive approach within Sandelowski and Barroso’s typology of qualitative research findings.  In the context of international comparative studies, other data analysis strategies may also prove fruitful. For instance, the gaps between qualitative and quantitative analyses can be bridged 56 using Ragin’s Qualitative Comparative Analysis.104 This may be a valuable tool to investigate the generalizability of findings as well as the causal complexity of the variables encountered in the coded data. 3.4 Final Considerations By conducting qualitative descriptive studies with decision makers, public servants, and the local research community, it is possible for international comparative studies to use participants’ contextually situated knowledge to depict the realities of different public policy contexts. The metaphor of the composite-sketch artist can be a powerful device to guide methodological reflections, such as the notions of descriptive validity (the depiction of events as perceived by observers in their apparent sequence) and interpretive validity (the appropriate elicitation of the meanings attributed by the agents to those events). It illuminates the researcher’s role in presenting a picture of the topic investigated that most observers would likely agree with. We argue that qualitative descriptive efforts are neither thick nor thin descriptions, in the sense used by Clifford Geertz.105 By his account, a thick description is an instrument that unveils the web of significance allowing the researcher to differentiate among “the conspiratorial winking”, “the involuntary twitch”, “the parodic-fake winking”, and “the rehearsing winking”, whereas the thin description is the mere report of people rapidly contracting the eyelids. A qualitative description is something else. A fundamental or basic qualitative descriptive endeavour would seek to describe those who are supposedly winking, how many they are, how many times they wink, which other gestures they make, where these people are situated, who else is present, etc. A qualitative description can be understood as a comprehensive description, one that seeks to provide a detailed description of the findings more likely to generate consensus among observers. 57 In the voluminous literature on qualitative research methods, there is no comprehensive study with a systematic reflection about the use of online qualitative surveys in international comparative studies on public policies. Therefore, our current endeavour may provide a valuable contribution to the research community. 58 Chapter 4: Methods 4.1 Research Design In order to describe existing practices of decision making in PSRA within publicly funded health care systems of high-income countries, the present work employed an online qualitative survey with decision makers and researchers that currently or previously worked with priority setting. We intended to describe any area of PSRA that the respondent could have been involved with at any level of governance or administration (micro, meso, or macro) and the type of health care (primary, specialized, hospital, etc.). We had to be aware that distinct depictions from different countries would arise simply because this is not a comprehensive work and those participating have different levels of interaction with the system. The study design will be detailed according to its three main components: sampling, data collection, and data analysis.  4.1.1 Sampling Understanding a given phenomenon across different countries requires a minimum degree of comparability among the settings included in the study. Thus, the first choice was to focus on publicly funded health care systems, which broadly speaking could refer to systems or system components as diverse as the Canadian public single-payer insurance, the American programs Medicare and Medicaid, the British National Health Service (NHS), or the public component of social insurance systems like those in France or Germany. Following, it was necessary to decide which countries could potentially be included in the study. In order to achieve a minimum of comparability, two criteria were defined: countries with a population above 4 million people, and GDP per capita above USD 20,000. This step corresponds 59 to the highest level of sampling indicated in section 3.3.1, encompassing strategies of homogeneity and criteria sampling. Appendix Elists the 27 countries that meet these two criteria, according to data provided by the World Bank for the year 2016. This list represents not all the countries included in the study, but rather the initial pool from where our sample could be obtained. Studying all the 27 countries in the list would not be feasible, not only because of the overwhelming amount of data to be generated and analyzed but also because of the political nature of some of these places, like Kuwait and Saudi Arabia. To determine which countries from the pool would be included in our study, we applied other layers of sampling strategies. Applying the “politically important cases” strategy, Canada, Australia and the UK were included because the majority of published works on PSRA in health care have been carried out in these countries. Also, following the evidence that PSRA has become an important topic in industrialized western nations, at least at the discourse level, almost all of the European nations were included. Applying “maximum variation sampling”, the USA was also selected for the unique nature of its health care system. Thus, the highest level of sampling was completed, yielding a list of countries to be considered in the study, as shown in Table 4.1. Of the initial pool, Japan, Korea, Singapore, Israel, and Ireland also represented important cases to be studied, but because of the time factor involved in a MSc thesis project, we could not include them all. A study addressing these settings remains to be done.   60  Table 4.1 List of countries included in the study. Countries Australia France Portugal Austria Germany Spain Belgium Italy Sweden Canada Netherlands Switzerland Denmark New Zealand United Kingdom Finland Norway United States  The next step was to define and pursue strategies to find potential participants in each of the relevant countries to be included in the study. Two mechanisms were used to find out names to be invited to answer our survey. • We searched the scientific literature using Medline and PubMed, and we scanned the grey literature by conducting searches in websites of relevant organizations such as ISPOR and IHEA. A complete list of sources used for this purpose is presented in Appendix F. The search strategy consisted of combining a term related to PSRA and the name of a country, for instance, “priority setting” and “Finland”, “rationing” and “Finland”, or “resource allocation” and “New Zealand”. The terms used were either “priority setting”, “rationing” or “resource allocation” plus a country name. This was done for all countries in PubMed and Medline. All authors of works related to PSRA were recorded and included in our sampling list. The search results were perused until saturation was achieved; whenever the same authors surfaced repeatedly, the search was stopped. • We searched the membership and contact list kept by the International Society on Priorities in Health (ISPH), provided by the management committee of the Society 61 through Professor Iestyn Williams at the University of Birmingham, to find contact information of individuals working in any of the 18 countries included in our study. All participants were also asked to suggest other potential participants. This snowball sampling was particularly important to reach out to decision makers who usually do not publish papers or are active members of those societies of health economics and related sciences. No specific care setting was predetermined. Participants with experience and knowledge in all possible realms of health care (community care, pharmaceuticals, mental health, etc.) and all levels of governance and administration (national, regional, hospital, residential care, etc.) were considered.  4.1.2 Data Collection A survey instrument was developed and operationalized through the Qualtrics platform. The entire questionnaire (as well as the introductory message and the consent form), is shown in Appendix G. A pilot study was run with four researchers, whose feedback led to improvements in the last version of the survey instrument. In both the standard invitation message and the snowball sampling invitation message (Appendix H and Appendix I), participants were informed that the survey questionnaire could be answered either online through the Qualtrics platform or through phone/Skype calls, according to their preference.   62 4.1.3 Data Analysis As discussed in section 3.3.3, the analytical framework to be employed in qualitative descriptive studies is qualitative content analysis. Given the substantial knowledge generated in the field of priority setting, with both the normative and prescriptive paradigms of decision making in resource allocation, our analysis had a hierarchical composition. Considering our previous knowledge about the consensual importance of specific topics (such as disinvestment, public engagement, use of evidence to inform decision making, etc.) and about existing frameworks, the first underlying analytical layer consisted of a directed content analysis. These preconceived notions and categories were used to structure the survey questionnaire and ensure that participants’ answers would somehow address these elements. Some probe questions were also used to provide contextual information. The second analytical layer consisted of conventional content analysis, i.e., no previous system of codes was prepared beforehand to analyze each answer. The codes relevant for analysis emerged from the data. No information provided by participants was disregarded as irrelevant or impertinent. The typical process of assembling related codes into categories took a different format in the present work. Because participants refer to distinct scenarios and the primary purpose of our work is to describe existing practices as unique phenomena, coalescing different codes into the same category occurred only with participant data that referred to the same institutional context. Thus, data saturation was only sought and achieved (as it could possibly be in this research design) within a given specific scenario being referred by multiple participants. It does not impede us, however, to compare these categories across countries. The dataset was exported from the Qualtrics platform in a CSV file and analyzed using NVivo. 63 Chapter 5: Results 5.1 Survey Response In total, 450 individuals were invited to participate in our study: 135 were identified in our initial search process, 44 were suggested through snowball sampling, and 273 contacts were obtained through the ISPH list. The list of invitees comprised people from all 18 countries included in the study. Fifty-eight individuals responded to the survey, yielding a response rate of 13%. Figure 5.1 shows the number of invitees and respondents by country. And Table 5.1 shows the primary professional role of those involved in the study.   Figure 5.1 Number of invitees and respondents per country.   64 Table 5.1 Participants according to their country and primary professional role.  Academics / Researchers Decision makers Australia 3 - Austria 2 - Canada 5 2 Denmark 1 - Finland 1 - France 2 - Germany 1 - Netherlands 2 - New Zealand 3 1 Norway 8 1 Sweden 5 1 Switzerland 1 - United Kingdom* 15 3 United States 1 - * Includes one decision maker from Guernsey, which is not officially part of the UK.  Three participants (from Canada, Norway, and the USA) did not provide enough information to serve as an input for data analysis, and one participant from the UK answered the survey on the basis of priority setting for health research and not for resource allocation in managing health care systems. Thus, the data analysis was conducted with 54 participants. Of this number, 29 participants answered the survey focusing solely on practices of decision making in PSRA happening at the national level of their respective countries, 10 participants provided a brief overview of PSRA strategies across different levels of governance in their countries, and the remaining 15 either provided data about a specific setting (hospital, health authority, etc.) or 65 provided generic information about practices that generally take place in hospitals, counties, and health authorities. The findings are presented here according to the following topics: • PSRA frameworks • Disinvestment • Stakeholder involvement and public engagement • Facilitators and barriers for implementing formal PSRA processes • Strengths of existing practices and areas for improvement • Use of evidence and appeals mechanism (presented as lesser themes in section 5.7) Whenever appropriate, to obtain an insightful overview of the findings across countries within each topic aforementioned, the results are presented per level of governance.  5.2 PSRA Frameworks A summary of the findings on frameworks/processes/practices of PSRA is presented in Table 5.2. 66 Table 5.2 Findings on existing processes of PSRA.  National level State or Province / Regional level Single health care organizations Australia HTA (PBAC & MSAC) - No specific threshold HTA (SAPACT)  Austria Reimbursement decisions - No single decision-making framework. - For drugs, there is a legally defined process. - Non-transparent decisions  Ad hoc decisions - Loose reliance of several criteria ------ Hospital Clinical Committees - Benefit assessment - Budget impact Canada  HTA of drugs (CADTH) - Overriding principle: will this work have an impact on the system? - Multi-criteria (including clinical burden, economic/financial impact and equity) - Inputs from a variety of sources  Priorities and Evaluation Committee –PEC (a provincial institution) - Single criterion: clinical effectiveness - Decisions: high/medium/low priority for investment - Some consideration of cost-effectiveness and budget impact without a formal influence on the final decision  Several health organizations have implemented formal processes based on A4R and/or PBMA  Several health organizations have implemented formal processes based on A4R and/or PBMA Denmark Formal process for coverage of hospital and prescription drugs - No further details ------ ------ Finland COHERE process - Criteria:  - significance of a health issue ------ ------ 67  National level State or Province / Regional level Single health care organizations  - medical justifiability  - ethical and economic considerations France National HTA No other initiative known ------ ------ Germany No explicit PSRA processes  G-BA - Reimbursement decisions for the statutory health insurance funds (GKV) ------ ------ Netherlands Zinnige Zorg (Appropriate care) - Systematic analyses per ICD-10 chapters - Patient-oriented, effective, and medically necessary care  HTA - Consideration of multiple criteria, not only cost-effectiveness (e.g., budget impact, added therapeutic value)  MCDA / A4R - Resource allocations at the ministry of health ------ ------ New Zealand  No formal framework for resource allocation within the ministry of health - historical allocation - no explicit rationale  Decisions around package of services streamlined into the ministry of health. ------ ------ 68  National level State or Province / Regional level Single health care organizations - It specifies which services should be provided for the population  HTA (PHARMAC) - Consideration of multiple factors situated within four dimensions: need, health benefits, costs and savings, and suitability. Norway National Guidelines on PS - Produced/updated by the National Committee on Priority Setting - Approved by the parliament - Three general criteria:  - benefit  - resource consumption  - severity - No explicit process and much room for interpretation - Despite important influence, there is evidence that it has not been used systematically  Nye Metoder - Analysis for introduction of new technologies in specialized care - HTA represents an important piece - Three types of HTA:  - Mini-HTA (hospital, supporting units)  - STA (single comparison)  - Full HTA (multiple comparisons performed at the national level) ------ ------ 69  National level State or Province / Regional level Single health care organizations Sweden National ethical platform for PS - 3 principles  - human dignity  - need-solidarity  - cost-effectiveness - No explicit process - Lots of room for interpretation/application  National guidelines on disease treatment  HTA - Decentralized - Separation between hospital drugs and prescription drugs No single/unified formal framework Several initiatives implemented and discontinued Different strategies in place based on the national ethical platform  Ordered investments - vertical and horizontal PS based on the national guidelines  HTA - Several county councils have established a group to carry out mini-HTA  Switzerland No initiative known ----- ----- United Kingdom HTA (NICE) - Soft ICER threshold - Well-described process in the literature  Advisory Committee on Resource Allocation – ACRA - Criteria:  - equal access to care for equal need;  - reducing avoidable variations in health inequalities  Scottish Medicines Consortium - ICER threshold with flexibility around drugs for end of life and rare diseases.  Ad hoc decisions and historical allocations still seem to be the norm  No single framework for a given level of governance  CCGs “Muddling through” Multi-criteria processes of variate designs - Common criteria:  - strength of clinical effect  - strength of evidence/  - availability of alternatives Ad hoc decisions and historical allocations still seem to be the norm  70  National level State or Province / Regional level Single health care organizations * Guernsey PSRA framework: G1033 - 3 overarching notions:  - Allocative value  - Technical value  - Personal value - Principles:  - maximization of value  - prioritization as the main tool for maximizing value  - cost-effectiveness  - proven clinical effectiveness  - work with the established budget  - equal treatment funding for equal needs  - no decision taken by third party on its behalf  - no discrimination in treatment - Factors to be considered (among others):  - equity  - budget impact  - the anticipated outcomes of not investing in the proposal  - number of individuals affected - NICE guidance is considered, but it is only advisory, not mandatory  - fit with national and other guidance  - cost-effectiveness   USA ----- ----- -----  71 5.2.1 National Level Participants who relied on practices of priority setting taking place at the national level of their respective countries of work mostly described activities of economic evaluation of health care technologies carried out by HTA agencies or the like (for further discussion on this, see Chapter 6). Let us now examine the findings regarding existing PSRA frameworks or strategies at the highest level of governance of each country. Australian participants described only HTA activities. At the national level, two independent bodies linked to the Australian government ministry of health are responsible for conducting economic evaluations of new therapies: PBAC (Pharmaceutical Benefits Advisory Committee) and MSAC (Medical Services Advisory Committee). The PBAC has a mandate on drugs and the MSAC is responsible for non-drug technologies. New technologies are incorporated in the system only after a formal HTA process by one of these agencies. Technologies already approved for marketing in Australia are submitted to these bodies, normally by their producers. The assessment takes into consideration comparative safety, clinical effectiveness, cost-effectiveness, and total cost, but there is no fixed threshold of WTP. Austrian participants reported a very fragmented health care system, existing diverse public payers, and responsibilities at different levels. No centralized process of decision making in PSRA exists. The hospitals are usually the main decision makers in this regard. Decisions on reimbursement are usually not transparent, do not follow a single formal strategy, and are made differently for drugs and non-drug technologies and across different settings (outpatient services, vaccinations, hospitals). For drugs, there is a legally defined framework to decide which drugs will comprise the reimbursement list. 72 According to Participant 5, the decision-making process on drug reimbursement in Austria has the following design: • The producer applies to have a new drug included into the code of reimbursement. • The drug is appraised in terms of pharmacological innovation, clinical benefit, and economic consequences based on information provided by the manufacturer. • Such evaluation is conducted by administrative staff of the Austrian social health insurance (not by an independent external body) and follows some basic principles of evidence-based medicine and HTA (but not in a systematic way). • Concurrently, the same administrative unit at the health insurance body negotiates price. • A committee of representatives from several social health insurance bodies, chamber of physicians, chamber of commerce, and chamber of pharmacies makes a final recommendation to the executive board of the main association of the social insurance on whether to fund that drug and under which circumstances (for which patients and conditions.) Apart from the existing process for inclusion of drugs in a reimbursement list, no formal and systematic strategy for priority setting process takes place. What takes place is ad hoc and informal, relying on criteria such as patient group affected, alternative treatment availability, etc., to inform decision making. Among those working in Canada, only one participant answered the survey on the basis of practices at national level. This participant described the CADTH process, whose overriding principle is reportedly, “will this work have an impact on the system?” Employed criteria involve 73 clinical burden, economic/financial impact, and equity, which are judged with inputs from a variety of sources. The only Danish survey respondent briefly mentioned a “formal process for hospital-dispensed hospital drugs and prescription drugs”, and “for other health services priority setting takes place through the national and regional budget process”. No information about criteria, underlying rationale, and procedures for reaching a final decision was provided. It is not clear if the participant was referring to regular HTA activities when referring to formal processes for hospital and prescription drugs. In any case, these activities currently take place in a decentralized manner in Denmark, following the structured process established by the extinguished Danish Health and Medicines Agency (information obtained at ISPOR’s website).106 In Finland, the PSRA framework practiced by The Council for Choices in Health Care (COHERE Finland) was reported. According to Participant 14, the council relies on the following three principles to make decisions: “significance of a health issue (both medical and societal), medical justifiability (in terms of the effectiveness and safety of the intervention and taking into account the severity of the condition), and ethical and financial aspects as a whole (e.g., respecting both human dignity and the distribution of resources within society and the health care sector)”. COHERE is chaired by the permanent secretary of the Ministry of Social Affairs and Health (MSAH) and has up to 15 members (with respective deputies assigned) with expertise in medicine, dentistry, nursing, law, health economics. and administration. No further information was provided on how decisions are reached by the council nor how evidence is collected and appraised. According to the COHERE website, the range of health services is monitored, defined, and assessed as a whole by the council, which defines it at a general level. There is no assessment at the patient level regarding service needs, exact treatment, and examinations. A patient’s need for 74 care is assessed by a health care professional. “No comprehensive list is available of the service choices, or the health services financed from public funds in Finland.” One French participant reported a national HTA process for non-drug technologies, without providing further details. Another respondent who claims familiarity with decision making at all levels of governance and administration simply stated that there is no formal PSRA framework and that “decision making principles are based on demand of actors”. Our single German respondent said that there is no explicit PSRA process in the system. Decision making on priority setting is “implicitly done by practitioners (budget constraints)”. He stated that the proper notion of priority setting is “highly unpopular (…) among physicians and politicians; often confused with rationing, taking away services/money”. He highlighted the work of the Federal Joint Committee (G-BA), the “highest decision-making body of the joint self-government of physicians, dentists, hospitals, and health insurance funds in Germany”. The G-BA is responsible for issuing directives for the benefit catalogue of the statutory health insurance funds (GKV) and for specifying which services in medical care are reimbursed by the GKV. G-BA also determines measures for quality assurance in inpatient and outpatient areas of the health care system. Dutch respondents focused on three processes happening at the national level in the Netherlands. One is a program run by the National Health Care Institute (Zorginstituut Nederland [ZIN]), called Zinnige Zorg [Appropriate Care], which conducts systematic analyses per ICD-10 chapter to ensure patient-oriented, effective, and medically necessary care. Furthermore, ZIN has an appraisal committee related to HTA that relies not only on cost-effectiveness but also on criteria like budget impact and added therapeutic value. One participant from the Netherlands also reported the use of a formal process using a value assessment framework based on MCDA and A4R. No 75 specific detail was provided about for which exact purposes and under which circumstances this MCDA/A4R framework is used. Participant 22 states that there appears to be no formal framework for resource allocation, broadly speaking, within the New Zealand Ministry of Health, even though distinct sectors of the Ministry have to submit cases to a decision-making committee, but the underlying rationale and criteria were not described. However, Participant 24, who reports working with elective treatments, says that “the ministry works with clinical groups to develop national clinical prioritisation tools for access to publicly funded elective surgery”, whereas “individual district health boards set their own funding priorities”. Participant 23 reports that priority setting also takes place at this level when the national government specifies services that hospitals or district health boards must provide for the resident population. The National Health Committee used to be an independent body responsible for this criteria-based recommendation of technologies and services to fund. Since 2016, when this body was disestablished, these tasks were incorporated in the general scope of the national ministry. But Participant 23 also stated that historical utilization “would be the best predictor of what is going to be funded in the coming year.” Three of the four respondents from New Zealand reported the work of PHARMAC as a PSRA activity taking place at the national level. PHARMAC is a governmental HTA agency with a mandate on setting the list of drugs to be funded. The decision-making process is based on several factors for consideration covering four dimensions: need, health benefits, costs and savings, and suitability. Norway has a long tradition in the debate on priority setting in health care, dating back to the early 1980s. The national committee on priority setting, which works from time to time to develop/update the country’s PSRA policy, prepares a report and sends it to the parliament. The 76 Norwegian parliament approves a guide with principles and notions for PS in health care. There are three legally defined criteria that all hospitals and decision makers are supposed to follow: benefit (“the priority of an intervention increases in keeping with the expected benefit of the intervention”); resource consumption (“the fewer resources an intervention requires, the greater the priority of this intervention”); and severity (“the priority of an intervention increases in keeping with the severity of the condition”). The national policy does not explain procedures for arriving at decisions and leaves lots of room for interpretation (regarding outcomes measurement, threshold, procedures, etc.). Participant 26 stated that “other and more informal forces also play a role in concrete decisions of how to use resources, e.g., interests by professionals, technology drivers, etc.”. Ultimately, decisions are made by local decision makers who have this general guidance on principles to base their decisions. Until recently, the national council on priority setting provided guidance on coverage of health care technologies. Yet, according to Wester and Bringedal in a comprehensive document analysis of previous council decisions, “while for the most part, the criteria seemed to have played an important role in the Council’s decisions and justifications, it was also evident that they often did not apply the criteria in a systematic and principled manner”.107 p134 Participant 29 reported being involved with a novel initiative called Nye Metoder (new methods), which he described as a “formal process for introduction of new technology in specialist [specialized] care in Norway”. HTA constitutes a fundamental piece of the process, which, following national guidelines, is also based on the three general guiding principles (health benefit, resource use, and severity). According to the official website of “Nye Metoder”: “There are three HTA formats in Norway; mini-HTA, STA, and full HTA. The mini-HTAs are limited assessments performed by clinicians and 77 supporting units within the hospitals. The STAs focus on a single method of health technology related to a comparator and are performed by either the Norwegian Medicines Agency (if the method is a medicine) or the Norwegian Institute of Public Health (all other methods). Full HTAs are broad assessments performed at the national level by the Norwegian Institute of Public Health. The full HTAs may for example be used to compare various methods that have been used in clinical practice for some time. When performing assessments, the Norwegian Medicines Agency or the Norwegian Institute of Public Health works in close dialogue with clinicians that, among others, have been recruited by the four regional health authorities.” The Swedish parliament approved an “ethical platform for priorities”, which includes the human dignity principle, the need-solidarity principle, and the cost-effectiveness principle. According to Participant 34, “the three principles are operationalized into a national model for transparent prioritisation” that is to be used at the national and local/regional levels. At the national level there are formal priority-setting processes where the National Board for Health and Welfare issues national guidelines for different diseases, guidelines that include recommendations for the local level for decisions on priority setting. For Participant 38, though, the parliamentary decision-making guidelines establish general principles but leave lots of room for interpretation. Participant 38 also briefly reported HTA activities for hospital drugs and prescription drugs. He pointed out the decentralized nature of HTA in Sweden and recommended an item in the scientific literature for a comprehensive overview of HTA in the country.108 In general it follows the general pathway, having the new therapy councils put forward proposals to be 78 evaluated by an HTA agency. After an economic and an ethical appraisal, a recommendation is formally issued. The committees work with multiple thresholds depending on the medical need. The only Swiss participant stated that “there is in general enough money in the system (…) so priority setting has less of a role to play”. No precise answer about the current practices of decision making in PSRA was provided. She offered vague insights on what appears to happen at the national level when she says things like “the strength of lobbyists and pharma/industry is high, the process could be better and fairer, but in general the needs are met as required as long as someone is willing to pay”. In general, there was not enough information to adequately describe the PSRA practices taking place in the country. At the national level, most UK participants described the work of the National Institute for Health and Care Excellence (NICE) on HTA. Given the substantial amount of information about NICE in the scientific literature or other well-known elements of grey literature, offering a detailed description of its work would be beyond the scope of the present work. Nonetheless, it is important to highlight some of the quotations from our participants to obtain a deeper view of the process. “In my view, NICE’s method is best thought of as setting a rebuttable presumption or default to prioritize the social value of cost-effectiveness that can be modulated by other social or ethical values, such as reducing health inequalities”, said Participant 43. In the words of Participant 44: “So, there is indeed a formal process, and the rationale provided by NICE for it is essentially about the need to balance a pseudo-utilitarian concern for the maximisation of aggregated population health and what they conceive of as egalitarian concerns, which are focused on who receives the benefits. The resulting process is built around a soft incremental cost-effectiveness 79 threshold set at £20000-£30000/QALY, which is balanced against a set of counterweights (severity, end-of-life premium...).” Another brief description of the overall process was provided by Participant 56: “NICE uses a formal process through its health technology appraisals to evaluate new health care technologies. The process is designed to select out those technologies that are most likely to have uncertainty about their effectiveness, a high price tag, or both. The role of NICE is to set a national standard for these technologies to bring all local health authorities (now called Clinical Commissioning Groups or CCGs) to the same commissioning and thereby reduce regional variation. There is detailed guidance on the process (Guidance on Processes of Technology Appraisal) as well as on the methods (Guidance on the Methods of Technology Appraisal) that set out the priority-setting procedures and how evidence should be reviewed. In addition, the Social Value Judgements guidance sets out a series of ethical principles to take into account when making the decision. In most instances, cost-effectiveness is the clear reason for decisions, and NICE uses a threshold of 20,000-30,000 pounds per QALY as the cost-effectiveness thresholds for ICERs. However, NICE has pushed above this limit in individual cases, and in groups of technologies, like the Highly Specialised Technologies (ultra orphan conditions, a new threshold is being set at 125,000 GBP/QALY where there previously wasn't one) and for life-extending treatments for terminal conditions (given the terms: cancer). This fuzziness about the threshold 80 suggests that NICE does draw on additional rationales for decision making but these have not been consistently and clearly articulated.” Apart from the NICE activities, one participant also mentioned the work of the “Advisory Committee on Resource Allocation (ACRA) that allocates funds geographically according to two criteria: offering equal access to care for equal need; and reducing avoidable variations in health inequalities” (Participant #45). No further detail to comprehend the process was provided. One participant from the UK reported the work of Scottish Medicines Consortium, part of Health Improvement Scotland, which is an HTA agency with a focus on new medicines, operating a cost-utility analysis model with some flexibility built in around drugs for end of life and rare diseases. One respondent was from Guernsey, which though not an official constituent part of the UK still has strong ties to it under the non-sovereign status of a British Crown dependency. In 2011, an ethical framework for prioritizing health care for the States of Guernsey was approved. But it was only in 2017 that new versions were brought in containing operational aspects of PSRA (policy G1033). The governmental body responsible for health care in Guernsey, the Committee for Health and Social Care (CHSC), works with the notion of allocative value (achieving the best distribution of resources across services for different patients and groups), technical value (producing the highest benefit with the available resources), and personal value (making decisions in a way that matters to patients). The overarching framework for PSRA is based on several principles: • Making investments that aim to maximize the value of care delivered to the population it serves. 81 • Adoption of prioritization as the primary methodology for all its decisions making around resources. • Investing in interventions that are cost-effective. • No funding of treatments of unproven clinical effectiveness, unless it is in the context of a well-designed clinical study. • Working within the budget allocated to CHSC by the States of Guernsey. • No funding of one individual if others with the same need cannot be funded. • No funding decision taken by a third party on behalf of CHSC. • No unjust or prejudicial distinction in the treatment of different categories of people, especially on grounds of personal characteristics, such as age, gender, sexual orientation, gender identity, race, nationality, religion, lifestyle, social position, family or financial status, intelligence, disability, physical, or cognitive functioning. The framework also defines a set of factors to be taken into account when making funding decisions, including equity, budget impact, the anticipated outcomes of not investing in the proposal, number of individuals affected, etc. All NICE guidance is considered only advisory. Nonetheless, according to the Guernsey participant, the Health and Social Services Department is under pressure to revise this policy, making NICE decisions mandatory on its territory. She also said that “while a form of G1033 has been operating for a number of years, it would be reasonable to say it has only really been recently corporately owned, and still is not fully implemented”. No specific procedural guidance is provided by the PSRA framework on how to reach the final decision and who to involve, but basically principles are established. 82 5.2.2 State/Provincial, Regional, or Single Organization Levels An Australian participant reported a state level initiative related to HTA. The South Australian Policy Advisory Committee on Technology (SAPACT) advises on the introduction of new high-cost health care technologies into the health system of the state of South Australia. Equivalent HTA programs exist in other states. An Austrian participant reported that hospitals have set up committees of clinicians for priority setting: the rules are based on benefit assessment (gains in quality of life and years of life) and budget impact. The basic underlying principle is utilitarian thinking of maximizing health. No further detail regarding the extent to which the committees have been implemented, evaluation of current practices, or even the exact procedures to make decisions was provided. We collected two sets of data about processes taking place at non-national levels of Canada. One included reports from four respondents about the PSRA process taking place at a provincial level institution. The other consisted of a general report about practices in different settings provided by an academic with extensive experience providing guidance on PSRA to health organizations. The provincial institution has a formal process carried out by the Priorities and Evaluation Committee (PEC) that encompasses drugs and non-drug technologies. Tumour groups submit proposals for new systemic therapy protocols (either new drugs or new schedules/combinations) to the PEC, which reviews the proposals and uses a multi-stakeholder decisions process based on both traditional evidence-based methods and cultural values elicitation109 to compare proposals that are classified as high/medium/low priority for funding based on the strength of the clinical evidence. PEC evaluates submissions on the basis of clinical effectiveness, although some level of discussion around cost-effectiveness and budget impact occurs without formal influence in the 83 final decision. A list of prioritized drugs is then sent to the executive committee, who make final recommendations to the ministry of health. Participant 12 said that “in the past they used a mix of clinical and cost-effectiveness for decision making, but the way in which the criteria were applied was confused and not systematic”. Participant 7 reported the implementation of an explicit values-based decision-making framework informed by accountability for reasonableness and/or program budgeting and marginal analysis (A4R and PBMA) and most often by the Gibson-Mitton integrated priority-setting framework in “a variety of organizational settings (e.g., provincial ministry, large hospitals, local health integration networks) and organizational levels (e.g., ministry emergency operations centre vs. deputy minister's office; ICU vs. organization-wide clinical priority setting)” across Canada. According to the participant, most of these organizations are fairly sophisticated and have formal decision-making processes in place and have requested consultancy from her to “enhance transparency and public defensibility of the priority-setting process, to balance competing goals, values, and interests in a rigorous way, and/or to strengthen effective engagement of affected stakeholders”. A Norwegian family physician working in a primary care clinic setting (with three other family physicians and five nurses/health secretaries) reported that priorities are discussed in biweekly meetings “within this community of practice”. Yet there is no structured framework for making decisions. They have a contract with the municipality that requires them to provide certain services and technologies. This participant pointed out that “on a higher level (…) priorities are directly and indirectly influenced by local, regional, and national priorities in health care”. Another Norwegian respondent reported the existence of “decision forums on regional level (CEOs of 84 hospital trusts) intended to establish common limits across regions and hospitals”, but not enough detail was provided to comprehend the structure of the decision-making process. Two national leaders in the field and staff members of the National Centre for Priority Setting in Health Care (Participants 33 and 58) that answered the survey together provided an insightful overview with respect to Sweden: “All the 21 regional health authorities (county councils/regions) in Sweden have a formal process for the setting of annual budgets. This is of course an example of resource allocation, but it is hardly a process where formal priority setting is done (a structured process for ranking of interventions). How are priorities for investments established on the regional level in Sweden? This differs—initiatives that will determine priorities can emanate from regional politicians and political parties, from local clinical leaders, the interpretations of national guidelines issued by the state authority National Board of Health and Welfare (Socialstyrelsen), national programmes on e.g., cancer care (initiated by the national government, state authorities, professional associations, and interest organizations). Although there were a number of health authorities (approx. six) that in the period 2003-2011 were involved in comprehensive formal priority-setting arrangements that embraced all service areas, almost all such activities have been discontinued. Hence, there are in practice, according to our opinion, no formal processes in use to set priorities in a systematic manner among the regional health authorities. However, there are a few authorities which have separate formal processes for scrutinizing new investments in technologies, mainly drugs. Usually they draw on medical 85 expertise as well as ethical and health economic input. It this case the Swedish “ethical platform”, the three principles of human dignity, need/solidarity and cost-effectiveness, is used as a template for guiding the decisions.” This general view is confirmed by Participant 34, who stated that “on local or clinical level there is no unifying priority [setting] process”. This participant also pointed out that several county councils have established regional HTA organizations to carry out the “so-called mini-HTA to evaluate new technology and advise decision makers”. Nonetheless, the participant also questioned the consideration of such practice as a PSRA approach when they said, “whether or not this should be regarded as a priority-setting process could be discussed, as it is a ‘one decision at a time’ approach”. Participant 36 states that there is no formal PSRA framework being used at the regional level. Participant 35 reported having extensive experience with health organizations across Sweden which establish their priorities based upon the legally defined ethical platform for priority setting as previously discussed, encompassing three principles (human dignity, need/solidarity, and cost-effectiveness). This platform has been operationalized in different manners by health organizations. One organization makes recommendations in terms of what should, could, and should not be used; another sets ranking from 1 to 10 complemented by not-to-do lists; and “a third organization makes rankings from 1–3 (where 1–2 normally gets funding)”. Processes are expected to be transparent and open for review, whether with formal deployment of an evaluation framework like A4R or not. He goes on to provide more details about a single region: “The regional level I am experienced with, there is a so-called Program and Priorities Council for new investments or structured uptake where the health professional nominates new treatments, these are critically examined and 86 prioritized by two preparatory boards with medical experts, HTA experts, ethicists, and health economists. The decision is then finalized by the council where the different care givers in the region (hospitals, primary care), the regional cancer centre are represented (mainly by medical doctors acting in an administrative role) an ethicist and a health economist. However, the final decision is made by the politicians in the region. In all these settings, one might say that the decisions result from a deliberative process within the framework of the ethical platform based on the prepared material by more administrative functions.” Another participant reported a PSRA initiative in a county council for “ordered investment”, although the program has not been used at full scale. According to her description, “the underlying rationale is to use the Swedish national model for priority setting as a tool for both vertical and horizontal priority setting for all kinds of medical technologies (…) The tool per se is based on national guiding ethical principles, and further ethical statements necessary for horizontal priority setting have been developed locally (mainly based on reasoning in the Norwegian investigation on priority setting, NOU 2014:12) and agreed upon. The organisation has performed two systematic, open priority-setting processes”, as published in the scientific literature73,74,110. The data obtained from British participants may appear controversial but seems to reflect an environment with variate levels of capacity to deal with priority setting across health organizations. Participants 40 and 47, academics with expertise in PSRA, reported that most local and regional organizations do not have a formal, explicit, and systematic process. In the words of Participant 48, “most decisions are taken in a fairly ad hoc manner locally or regionally, despite a great deal of thought academically about the right way to undertake priority-setting processes”. 87 Nonetheless, other participants have provided insights on formal processes taking place in some health organizations. At the local/regional level, the Clinical Commissioning Groups (CCG) are the main structures responsible for priority setting. Although some level of guidance is established by national authorities (like NICE), the CCGs need to make their own population spending decisions. Participant 40 reported that, based on his experience assisting several CCGs in this matter, “Many have a “prioritisation” process (the most common term used in the NHS) and that these will typically incorporate some fairly rough and ready version of MCDA. Budgets are extremely tight and so many prioritisation policies ask that the budget for any proposed investment be identified before the investment option will be considered. The criteria often found in these policies include: strength of clinical effect/strength of evidence/availability of alternatives/fit with national and other guidance (e.g., local “sustainability and transformation plans”)/cost-effectiveness. Other criteria used less frequently or with less weighting include: patient and public consultation/affordability and implementation issues/legal risks.” Participant 54, who has experience in PSRA across different health organizations, stated that “at the regional (CCG) level, processes are mixed and generally constitute ‘muddling through’ - each CCG will have its own organizational process for prioritizing investments and (sometimes!) disinvestments, but it is not usually technocratic, and evidence reviews are normally sketchy and rushed - it is well known that prominent clinician advocates have a lot of sway in decision making, which limits the power of commissioners”. Participant 46 reported that priorities in some settings 88 are determined by health needs assessments, employing the NHS Outcomes Framework (NHS OF) to achieve improvement of health and reduction of inequalities. The NHS OF has five domains: • Preventing people from dying prematurely and reducing the number of avoidable deaths; • Enhancing quality of life for people with long-term conditions; • Helping people to recover from episodes of ill health or following injury; • Ensuring that people have a positive experience of care; and • Treating and caring for people in a safe environment and protecting them from avoidable harm. At the micro level, i.e., the patient level, participants highlighted that “clinicians still have a good deal of decision-making power, although this is being increasingly restricted by CCGs putting more firm pathways in place and setting up referral hurdles, particularly at the point of transfer from primary to secondary care”. Participant 49, who works with a “health and social care partnership in one area of Scotland”, reported that no formal process has been used to determine priorities for investment. Their scope has statutory services that must be provided, not allowing room for PSRA, and other services that “appear to be based on historic practice, i.e., they have done them in the past and will continue to do them but with a bit more or a bit less investment”. They seek to provide a balanced budget for the area, completing an equality impact assessment form for proposed investments (no additional information was offered about the nature of this application). In addition, “there is a governance strategy where all decisions are made via a group made up of different stakeholders from different organisations”. 89 A last piece of information in PSRA in local/regional health organizations within the UK was obtained regarding NHS dentistry. According to Participant 52, decision makers rely on an “implicit combination of national guidance (from Department of Health and Social Care) and local guidance from (dental) public health specialists and local clinicians”, although it is not clear what degree of each input impacts decision making. He says that there is no formal process or framework employed by NHS commissioners in this sector, with different procedures taking place across the country. In general, prioritization in this area has be aligned with national aims and objectives, observing the NHS dental commissioning guides, but the actual process does seem to be clear.  5.3 Disinvestment In Australia, in the words of Participant 3, disinvestment “is really only just commencing on the radar of the health system. There is a lot of opposition from clinicians about a recent initiative to remove Medicare item numbers for investigations/procedures deemed to be of no clinical benefit”. With respect to PBAC, the national HTA agency for drugs, other participants point out that disinvestment may occur by replacement when novel drugs are incorporated. Currently used drugs can be delisted, restricted, or suffer price reduction. Yet no formal process exists for disinvestment itself in this case; it takes place within the realm of CEA of new drugs. Participant 5 states that disinvestment rarely happens in Austria. She points out that a drug can be removed from the code of reimbursement in cases where cheaper generic alternatives are available and the manufacturer is not willing to reduce the price of the reference drug (which constitutes a legal basis for disinvestment). Nonetheless, disinvestment of drugs based on lack of evidence of clinical benefit is very unlikely, given that there are no defined criteria and processes. 90 If an attempt to delist a drug goes forward, the manufacturer can file a suit and is very likely to win because the social security act states that, at the patient level, every drug has to be reimbursed if there is medically defined need. The other Austrian participant reported that, even though no formal processes for disinvestment are known, “once in a while the benefit-catalogues (Hospital interventions in the DRG-system, drug-list) are scanned and discussed with clinicians which interventions might be ‘questionable’”. In Canada at the national level, the CADTH respondent said that they are in process of establishing a disinvestment framework. The participant further stated that “HTA by definition is comparative so every decision to invest in something should come with a decision to not invest in comparator (an oversimplification I know but the intent is not to have two unwieldy disconnected processes)”. Participants at the provincial institution reported that there is no formal framework for disinvestment. Nonetheless, two respondents pointed out that disinvestment does take place naturally with incorporation of novel technologies. Participant 7, who has experience in various health organizations within Canada, stated that inn her experience: “Disinvestment processes are less well developed than investment-based priority setting. It is usually for this reason that I am called in to provide advice, recognizing that disinvestment may often be experienced as a loss by affected stakeholders. Hence, there is a sense of urgency to “get the process right” to establish the legitimacy and fairness of priority setting in the eyes of affected stakeholders. Population need and clinical effectiveness are emphasized, although evidence is often not robust or readily available.” 91 Respondents from Denmark, Finland, France, Germany, Netherlands, New Zealand, and Norway were not aware of any formal and systematic approach for disinvestment. Nevertheless, some participants commented how the concept of disinvestment may manifest in their national context. A Dutch respondent highlighted the existence of a program related to “Choosing Wisely”, used for prioritizing HTA research. A New Zealand participant pointed out that PHARMAC chooses the generic drugs to be funded each year in a way that the brand of drugs within a chemical therapeutic class can change from year to year based on price. A participant working in a Norwegian health organization said that although no formal procedures exist, decision makers discuss disinvestment in meetings twice a month considering that, for example, “investment in more advanced technologies may lead to task change for staff, ending up perhaps in reduced staff”. The Swedish participants said they were unaware of formal and systematic frameworks for disinvestment. Several participants highlighted that, regardless of the exact process design, disinvestment is expected to be based on the same ethical platform legally defined for setting priorities for investment. As Participants #33 and #58 pointed out, “It varies quite a lot how [disinvestment] is done and what is influencing. The national guidelines with its rank order recommend that certain low-ranked interventions should be phased out in different clinical areas, but the recommendations are not mandatory. Political decisions on disinvestments are almost non-existent. In most cases it is local clinical leaders that handle their budget dilemmas by rationing, where disinvestment is one type. Hence, the rationale for disinvestment differs” 92 Another Swedish participant commented that “to some extent the result of the priorities above, i.e., the should not, and the not-to-do, will be used for disinvestment to the extent this is done in the health-care sector.” With respect to the research work previously cited (section 5.2.2) of implementing a formal PSRA process in a county council in Sweden, the established procedure included identification of low-priority technologies. In the words of Participant 37, “the newly designed procedure for ordered investment includes presentation of both the desired investment and possible items for disinvestment, described in the same terms (a mini-HTA-protocol) and prioritised by help of the National model mentioned above”. The only Swiss respondent said that disinvestment usually takes place when cost is extremely high. Nonetheless, no further detail was provided so it is difficult to form a relevant picture of practices in the country. Participants in the United Kingdom state that there is no formal framework for disinvestment at the national level. They emphasized that “NICE tends to focus on assessing the cost-effectiveness of new entrants to the benefits package (drugs, new technologies)”. Some participants pointed out some NICE initiatives as “disinvestment decisions”, such as the “do not do” list (a compilation of hundreds of interventions deemed to be low value) and a “cost saving guidance” that includes orientation on the treatment of respiratory tract infection in primary care, which indicates that, apart from reducing antibiotic resistance and medicine related adverse events, the use of a “no prescribing” or “delayed prescribing” policy is expected to decrease antibiotic prescribing by a few million pounds nationally. Whereas NICE issues guidance and recommendations of technologies to be funded, CCGs are in fact the organizations ultimately responsible for disinvestment activities. The practices 93 around it vary considerably across the country. Participants reported that CCGs have been going through a period of financial difficulty, leading to mandating savings targets from different service areas that are then expected to present savings proposals. About this aspect of CCGs, Participant 41 said that, based on his knowledge working with various CCGs, they seemed to have been putting more emphasis on disinvestment than on priority setting, establishing separate procedures for conducting disinvestment. He explains the apparently “illogical” underlying rationale of such phenomenon in the following way: “Prioritisation processes have become in effect mechanisms for slowing down the introduction of new technologies, especially where these are expensive and not mandated by national government bodies. In other words, they are a reaction to the relentless proposals for new investment. By contrast, disinvestment requires an active process of seeking out options for removal/replacement which immediately implies a separate function/process (or at least an additional stage to existing priority-setting processes). The second reason is, I think, the financial context means that CCGs are looking at much more ambitious disinvestment programmes - including not just restricting access to treatments but closing services and whole organisations. Thus, while priority setting has remained slightly piecemeal and peripheral (in terms of scope of decisions and budgets), disinvestment has become a more urgent and large-scale concern in response to concerns about financial sustainability.” Further details on the exact design of such “separate processes for disinvestments” were not provided. Participant 56, on the other hand, commented that the emphasis of NICE on adoption 94 tends to push the whole system to focus on “what is new and coming from NICE” rather than on what is more practical and valuable, but not subject to national appraisal. With respect to NHS dentistry, a respondent indicated that disinvestment does occurs in this sector, “but it is even harder [than PSRA decisions] to know how these decisions are made by relatively small decision-making teams on a regional basis across England”. He revealed that many dentists have told him they want to understand how disinvestment decisions are made; they are a source of frustration given the lack of clarity and predictability of the decision-making processes. In general, he stated being unaware of any formal process for disinvestment in dentistry.  5.4 Stakeholder Involvement and Public Engagement In the HTA activities reported in Australia, numerous actors are involved, including politicians, managers, and health professionals. Under the health professionals umbrella, there are several specialities, like health economists, ethicists, clinical experts, etc. A participant highlights that patient involvement also occurs: “patients and families have the opportunity for input into the decision making through consumer hearings and through opportunities to comment on drugs that are being considered by the PBAC.” No method for engaging the general public has been implemented. The public is only involved insofar as there is space for consumer input throughout the process. In Austria, the HTA activities reported involve managers/administrators and health professionals. Participants pointed out that there is no culture of patient or public involvement in these processes in Austria. In general, the only process identified was a public consultation for defining the overall health goals for Austria. Even for the processes taking place in a hospital, usually only managers and health professionals are involved in decision making. 95 In the Canadian context, the national HTA agency for drugs involves managers, HTA agency staff, and expert committee members in the decision-making process. These stakeholders sit on the deliberative committee where they can vote/rank topics. No method for direct public engagement is employed, except that the public can provide input to topic identification. In the provincial institution, managers/administrators usually drive and oversee the PSRA process, health professionals often engage in developing priority-setting criteria and assessing investment/disinvestment options, and physicians and pharmacists review the clinical evidence. Participant 11 comments that although the PEC work is carried out by managers and health professionals, “in reality, there is significant political interference, usually in the form of direct phone calls to the head [of the department] to exert pressure when limits are set on a new drug-indication pair.” To date, there is no process in place for public engagement. Nonetheless, a participant revealed that a deliberative public engagement model, already successful at a national level, has been considered. Across Canada, there have been a few examples of public engagement, according to Participant 7. These include a health authority that engaged its community advisory committees circa 2002 in developing priority-setting criteria, and a hospital that “created a citizen's council to review evidence and make recommendations on health service investment/disinvestments to aid the hospital in addressing a budget deficit”. The only Danish participant stated that politicians, managers, and health professional are involved in the process of decision making in PSRA. Whereas managers and health professionals provide input to the decision processes, politicians are the ultimate decision makers. However, it was not explicated which exact organization or level of governance he was referring to, and no specific process for public engagement was recalled. 96 At the national level of the Finnish health care system, COHERE has a maximum of 15 members (and assigned deputies) with a variety of fields of expertise. Representatives come from the “main institutions of the Finnish healthcare and social security system and include those from university hospitals, municipalities, regions, research institutions and universities from around Finland, in addition to MSAH representatives.” No process of patient involvement or public engagement was reported at this level. In France, the national activities of HTA reportedly involve politicians, managers, health professionals, and patients in the work of the deliberative committees. The public has not been engaged through a defined process. Participants stated that public is involved “indirectly via regional and national health conferences which report to the ministry of health”. The German respondent stated that politicians, managers/administrators, health professionals, and health care insurance firms are involved in the decision making for setting priorities and allocating resources. No explanation was offered for how these stakeholders interact, at which levels of governance, how they arrive at the final decisions, etc. And no process for public engagement was reported. In the Netherlands, politicians, managers, health professionals, and patients are involved in the decision-making process. The appraisal committee includes members who represent these stakeholders of the health care system. Participants reported that there has been no framework to engage the public. Nonetheless, there is a project in development called “Burgerforum” (civilian forum), aiming to elicit public preferences. In New Zealand, HTA activities mostly involve of managers and health professionals, but also politicians, who influence the decisions made by the ministry of health, and patients, through 97 advocacy groups. The public is not directly involved in the decision-making process, although HTA agencies normally have a consumer board that allows topics to be proposed. In Norway, priority setting involves several agents. • Politicians make the general rules and participating in decision-making instruments like official hearings, macro budget decision making, and setting the agenda for health policy. • Administrators operationalize the nationally established decisions and guidance in PSRA, having direct decision power over their organizations’ budgets. • Health professionals make micro-level decisions on resource use and also develop technical clinical guidelines to decide which technologies to implement. • Patients do not have a direct voice or vote, but eventually patient organizations are invited to be heard on the proposals and general considerations PSRA committees, serving then as an input for decision makers. • Researchers have participated in “preparing the assessments of interventions debated among the managers of regional enterprises”. • Bureaucrats have been invited into the national committees on priority setting since at least 1987. There is no established methodology for public engagement. Yet, the public has been involved at specific points through public hearings. According to Swedish respondents, politicians, managers, health care professionals, health economists, ethicists, and patients are all involved in some way in priority setting. Participants 33 and 58 point out that “although their role in a formal sense is self-evident, the influence between key actors varies due how the health authority is organized (the model for governing health 98 authorities can vary with respect to how commissioning is done, and how evidence is processed, how the local profession is organized in expert bodies etc.).” In general, at the national level, politicians get involved in making decisions on the total budget, allocation of resources to broad clinical areas on the regional level (health authorities), and resource destination to certain patient groups. “Managers/administrators prepare and compile evidence for the political decisions and are usually responsible for the process of commissioning with the clinical units.” Health care professionals support the decision-making process by providing evidence and also make formal decisions regarding the budget in their particular clinical area. Participants report great influence of clinicians on how investments are initiated in the authority. Patients are not regularly involved, but patient representatives may sit on some committees, like HTA boards, and/or be consulted to provide input for review by decision makers. In HTA activities, health economists and ethicists also get involved in the process of evidence review and recommendation making, like in the Swedish Council for Health Technology Assessment. In regard to public engagement, there is no consistent or systematic effort. Several experiments with public consultation have been tried, but they have been circumscribed to research contexts. A Swiss participant reported that it is not clear who is influencing decision making in priority setting at the national level, stating that, apart from politicians, lobbyists seem to have significant influence. How professionals from the health care system participate in the process was not explained. At the national level, NICE has a well-established culture of stakeholder involvement. Politicians are involved in the process in that the health ministry refer topics to NICE for guidance issuing and also interfere with difficult decisions on technology adoption. Normally, these topics have been identified as priorities by NICE, NHS England, and the Department of Health. The 99 NICE committee for HTA, the Technology Appraisal Committee, gathers a wide range of stakeholders such as academics, technical experts, and health economists. Industry representatives submit evidence to support the clinical and cost-effectiveness of the technology under appraisal. In the words of Participant 44, “the process of assessing existing evidence also sees patient and carer representatives and clinical practitioners participating to provide their perspectives on features that are likely to have been forgotten by other stakeholders.” Patients have limited power other than in organized advocacy groups (through lobbying). In addition, NICE has a citizens’ council of 30 members of the public who represent the demographic profile of the population; the citizens’ council provides input regarding social value judgements. However, one participant commented that “there tends to be confusion over what to do with the information once elicited.” For the ACRA, membership is drawn from a broad spectrum of stakeholders, but does not involve the patient or the general public directly. About the Scottish Medicines Consortium, Participant 53 reports: “There is a committee that meets once a month and is (now) also open to the public, which means that for more controversial decisions people can apply to attend. Usually I think those places are mostly taken up with pharmaceutical industry representatives. The committee includes representatives of the SMC's public and patient involvement group. They consider the submissions from the drug companies which are reviewed by economists and HTA experts before the committee, to identify any issues with assumptions and models. The committee votes on whether each drug should be recommended.” 100 With respect to PSRA practices in CCGs, i.e., the local/regional level, Participant 41 states that there is often a stakeholder panel, mostly composed of the main budget holders and provider organizations affected by the decisions, including organizational leaders and senior clinicians. These stakeholders are either scoring options or reviewing a scoring exercise for various investment options. He points out that “it is not always clear a) whether those participating are expected to shed organisational affiliations or b) how recommendations are to be ratified and put into practice. Some bodies will have formal decision processes (e.g., establishing how and when to vote) whereas others handle this much more informally and are prey to who turns up!” About public engagement, Participant 41 commented: “Despite lofty rhetoric the NHS is still resistant to public involvement, especially when it involves contentious decisions about limiting access to services. “Consultation” processes are sometimes undertaken but these are often highly generalised and aspirational in tone (i.e., “what do you think about our new vision for world class health care in your area?”). The nitty-gritty of decision making often involves a few worthy individuals who buy into the “volunteerist” model—i.e., they are there to help the system first and foremost. Big priority-setting decisions are often then subject to opposition and challenge through campaigns, media etc.” In the regional health and social care partnership in Scotland that was described in Section 5.2.2, managers, health professionals, patients, and third sector representatives take part in the decision making in PSRA. There are representatives from each category on the teams which make decisions and “on the planning group who feed guidance and evidence into the decision-making 101 group. Patient and public representatives are invited to sit on the groups and can participate in the meetings. For decisions which change service provision then public consultation will happen.” For NHS dentistry in England, the local dental networks provide expert input to NHS England commissioners on issues of the day. Also, consultants in the topic from Public Health England will also generate input into the local dental networks and then assist commissioners. Participant 52 points out that “input is sought and requested, but it is the commissioners themselves (a rather small number of normally non-clinicians) who have the responsibility for actually making decisions linked to finance”. The general public eventually participates via Healthwatch groups.  5.5 Facilitators and Barriers Participants were also asked to indicate which elements of their institutional or contextual contexts they identify as facilitators and barriers to developing and implementing formal and explicit processes of priority setting. A list of the elements pointed out by participants per country is presented in Table 5.3. For some countries, responses refer to the broad national context whereas a setting-specific analysis is provided for others. The most commonly cited facilitators are budgetary pressures, strong leadership, and existence of key champions. The most common barriers were lack of knowledge about PSRA, vested interests, political resistance, lack of trust among stakeholders, and media pressure.  Some elements suggested are specific to their context. For instance, in Norway the existence of a strong welfare system with emphasis on equity was pointed out as a facilitator because the society tends to better accept the prioritization need. Likewise, Australian participants state that the fragmented nature of their health system, leading to “too many players at too many levels”, is a barrier for the establishment of formal and explicit frameworks of decision making. 102  Table 5.3 Facilitators and barriers for the development and implementation of formal PSRA processes according to survey participants. Country Facilitators Barriers Australia - Strong leadership - HTA capacity - Political understanding of the importance of a formal process - Fragmentation of the health care system - Lack of trust among stakeholders (especially clinicians) - Lack of transparency on evaluation processes Austria - Budgetary pressure (overpriced drugs) - Courage to stand up against pressure from clinicians and industry - Existence of champions in key organizations - Incentives to work too closely with industry - Lack of precise definition of “innovation” within the medical culture - Lack of a culture of transparency - Hierarchical structures and MD dominance Canada (CADTH) - Nimble culture - Service orientation - Vested interests - Short-term perspective - Discomfort in weighting criteria Canada (provincial institution)  - Strong leadership - Budgetary pressures (overpriced drugs and rise in number of cases) - Genuine commitment among committee members to achieve the best outcomes - Perceived legitimacy - Lack of knowledge on PSRA frameworks - Unwillingness to consider other criteria, apart from clinical effectiveness - Discontinuity of high level personnel - Lack of administrative support - Lack of dedicated human resources - Resistance from clinicians and politicians - Lack of public education on cancer and PSRA Denmark ----- - Lack of knowledge on PSRA Finland ----- - Fragmentation - Lack of funding to establish formal processes France - Strong leadership - Reluctance to change - Financial disincentives - Little development of research on PSRA 103 Country Facilitators Barriers Germany ----- - Politicians’ resistance (afraid of electoral consequences) - Resistance from clinicians Netherlands - Learning culture - Integrity - Culture of Transparency - Stakeholders engagement - Vested interests - Lack of trust among stakeholders - Black box procedures New Zealand - Desire for better performance and less variation - Excellence recognition of the HTA agency (PHARMAC) - Too many players and too many levels - Conflict of interests - Culture of secret negotiations (assuming better deals) - Misunderstanding of PSRA and its real causes Norway - Open discussion - Presence of key champions - Good understanding of the topic by politicians - Strong involvement of research expertise - Strong welfare system that emphasizes equity - Highly educated population with strong tradition of fair political decisions  - Lack of precise PSRA guidance beyond general principles - Lack of trust between stakeholders - Resistance to change - Vested interests - Culture of secret price negotiations - Discrepancy between who makes decision and who holds the budget in terms of drugs Sweden - Strong leadership at all levels - Good dialogue between politicians, administrators, and health care professionals at the national level - Openness to change - Sense of responsibility for the entire health system - Good adherence to decisions - Lack of trust among stakeholders at the regional level - Lack of adequate understanding of PSRA - Political resistance to communicate difficult topics - Reliance on historical allocations - Emphasis on adoption - Vague provision of principles - Media pressure Switzerland ----- - Lack of a culture of transparency United Kingdom (national level) - Good data management in health system - Budgetary pressures - Strong leadership - Strong backlash of negative recommendations - Political interference - Media pressure 104 Country Facilitators Barriers - Strong emphasis on performance improvement - Fear of the consequences of unplanned cuts - Independent nature of NICE - Great support of the research community - Vested interests - Lack of time to engage in new activities (lack of personnel) - Costs of stakeholder involvement - Avoidance of tough decisions - Unwillingness to be open and transparent (NHS Dentistry) - Perceived tension between what organization is expected to do nationally….and what local people may want - so they do not engage (risk is perceived to be too high to organization). United Kingdom (local/regional level) - Budgetary pressures (economic downturn) - ‘Air cover’ from NICE - Sharing of resources at local/regional levels - Environment that foster discussion of PSRA in the health care system - Political toxicity of any suggestion of rationing - Lack of funding in local evidence generating and capacity building - Lack of consistency - Constant reorganization - Discontinuity of personnel (commissioners)   105 5.6 Strengths and Areas for Improvement We also asked survey participants about strengths and areas needing improvement in the existing practices of PSRA in their settings/regions/countries. These findings are summarized in Table 5.4.  106 Table 5.4 Strengths and areas for improvement in the current practices according to survey participants. Country Strengths Areas to improve Australia (HTA) - Rigor and consistency in appraisal - Procedural fairness - High level of expertise - Decision making remains flexible in relation to the advice emerging from evaluation - “Greater capacity to assess technologies in the context of the whole system” - Lack of personnel “on the ground”, outside academia Austria ------ - Increasing transparency - Standardization of processes - Collaborative priority setting across sectors - Expand PSRA beyond acute or end-of-life care Canada (CADTH) - Criteria-based - Evidence-informed - Transparency (topics, criteria, and influencing actors) - Improving understanding of all involved that resources are limited and choices need to be made Canada (provincial institution)  - Formal committee - Good buy-in/adherence from/by clinicians - Rigorous evaluation of the clinical evidence - Quick turnaround time - Interval review (higher trust) - Expansion of focus (use of multiple criteria and other types of evidence) - Transparency - Improving knowledge of committee members on PSRA - Patient participation - Public engagement - Inclusion of ethicists and health economists - Appeals mechanism Denmark - Good approach to budgeting at regional and national levels - Reasonably knowledgeable politicians - Systematic use of HTA and CEA/CUA Finland ----- - Legislation - Funding - Staffing France (HTA) - Publicity of HTA decisions - More research - Timeline Incongruence 107 Country Strengths Areas to improve (technologies can be disseminated based upon political decision before the formal assessment is finalized) Germany ---- ----- Netherlands - Transparent process - Use of deliberative MCDA - Continuous process improvement - Skilled personnel - Lack of guidance on interpretation of criteria - No clear link with health system values - Stakeholder involvement New Zealand - Good HTA process (PHARMAC) - Poor PSRA practices within the ministry of health (outside PHARMAC in general) - Transparency / clarity of the decision-making process (how decisions are made, how criteria are weighted, who influence the process) - Capacity building - Consumer involvement - Auditing processes Norway - Legally defined principles - Political legitimacy - Fairness - Explicit talk - Better use of evidence - Stronger follow-up - Transparency - Stakeholder involvement - More awareness on PSRA among stakeholders and students - Speed - Public engagement - Disinvestment Sweden - Transparency (National level) - Explicit ethical platform - Democratic composition of New Therapy Councils (HTA) - Capacity building: a) Establish a common view on why and how more formal priority-setting matters b) how to distribute and define different roles between actor c) how to improve the trust between the key actors d) to create a sense of mutual responsibility for a universal health care system 108 Country Strengths Areas to improve (This internal foundation for PSRA is necessary to engage in formal processes to secure legitimacy within the whole society) - Integration of PSRA with budget establishment in the health authorities (move beyond the common-sense view that PS is a “special activity”, risky and difficult) - Transparency (county/regional level) - Translation of the ethical platform in concrete practices - Focus on adoption of new drugs - Disinvestment - More focus on opportunity cost - Stakeholder involvement - Better distinction between hospital drugs and prescription drugs Switzerland ----- - Transparency / clarity of decision-making processes - Less power for lobbyists - More openness to discuss priorities at the local level United Kingdom (NICE) - Accountability - Clear principles - Transparency - “A track record of learning from mistakes and from a wide range of perspectives” - “Clear distinction between scientific evidence and policy preferences” - Broad spectrum in stakeholder involvement - Infrastructure - High level of public scrutiny - Consideration of social values - Public engagement - Better understanding of health care costs and trade-offs - Make more explicit the final decision-making process - Criteria weighting (consistency among decisions) United Kingdom (regional – CCGs) - Stakeholder involvement - Flexibility - Knowledge of local reality - Clarity and explicitness - Consistency /Continued and systematic processes 109 Country Strengths Areas to improve - Good dialogue both ways between commissioners (the decision makers) and NHS contract holders - Clinicians need more confidence to be open with patients about all relevant factors in decision making at the consultation level. - Public engagement and education  United States ----- -----   110 5.7 Lesser Themes (Use of Evidence and Appeals Mechanism) When asked about the types of evidence used to inform decisions, 41 participants (72%) indicated that there is reliance on epidemiological, clinical, and economic evidence and expert opinions. But this does not mean that these pieces of information have equal or equivalent status. Several participants highlighted that expert opinions are only sought in the absence of “hard evidence”. Others pointed out that data on budget impact analysis or cost-effectiveness have less importance in their contexts when compared to clinical effectiveness. Responses about the existence of appeals mechanisms indicate that there is no well-established understanding about what would constitute an appeals mechanism. Around 35 participants (60%) answered “no” or “uncertain” for the existence of such an instrument within their organizational or contextual setting. The other participants often provided information on processes that they were unsure could be deemed appeals mechanisms. The following quotations from our participants illustrate the vagueness that this topic is commonly perceived. A Swedish participant stated that “the constant technology development is an ‘appealing system’ in itself”. Another individual from Sweden said that although there is “no formal process for appeal, in the end [it is] a political decision (the appeal is then indirect and comes through general elections)”. A respondent from New Zealand stated that appeals takes place at the district level or national level when “the decision may change with new evidence that is forthcoming”. And one from Norway indicated that there is an appeals mechanism in the sense that “patients can ask for a second opinion.” Several participants indicated the legally entitled right to complain about health care services as the appeals mechanisms in their countries. Two participants from Norway stated that patients can file formal complaints about the services they are receiving or are supposed to receive. 111 Similarly, respondents from New Zealand pointed that, at the individual level, “one can apply for compassionate basis for a drug or intervention” or “a patient who has been declined service may ask for a re-referral or complain”. Also, a Swedish participant highlighted that “besides formal systems for patients’ complaints, politicians regularly meet patient organisations and other representatives for the public and get clear views of the status of the quality and function of services”. Within the context of HTA in Austria, coverage decisions can be appealed in court. In Australia there may be a request of a technical review capacity of whether PBAC has made an error through an external expert review; or, as a respondent put it, after rejection, companies may resubmit the application to alter dosage or the price. In the United Kingdom, legal challenges of NICE decisions have occurred. According to participants, these can be the result of political pressure by the society, patient groups, or pharmaceutical industries. Appeals can be lodged for 15 days after NICE has reached a determination, under the presupposition that NICE has failed to act fairly or exceeded its powers, or when the recommendation is unreasonable in the light of the evidence submitted to NICE. Participant 44 stated, “They can only be lodged by stakeholders that have been previously identified as relevant ‘consultees’, including producers, patient and carer representatives, relevant clinicians”. Participant 51 highlighted that not only manufacturers can start an appeals process; he could remember at least one case “where the appeal was made by a Primary Care Trust on the grounds that NICE allegedly violated non-discrimination principles”. In respect to the CCGs in the UK, Participant 41 observed that “many local processes have introduced appeals mechanisms but these typically (following NICE) are confined to whether the 112 decisions making process adhered to the stated processes”. Participant 54 also commented that “individual patients can access an exceptional funding process if CCG policies restrict their access to treatment their clinician says would be helpful to them. If refused, the main recourse is the legal system, which is costly and time-consuming, particularity in light of recent reductions in Legal Aid”.  5.8 Quantitative Aspects Preparing a quantitative report of the findings of our work does not offer much insight into the studied phenomenon and might be problematic and misleading given the research design. Nonetheless, summarizing the quantitative aspects raised in probe questions and opinionative questions may be helpful to understand the profile and heterogeneity of our sample. It is important to emphasize that graphs and tables shown here are not generalizable and can only be interpreted as a descriptive tool of this work. As Figure 5.2 shows, the majority of participants have experience or knowledge with practices of priority setting taken at the national level in their respective countries. This characteristic of our sample is probably one of the reasons for the high frequency that HTA initiatives appeared as the existing PSRA strategy in so many answers. 113  Figure 5.2 Institutional contexts with/where which participants work or have worked.  In addition to their factual answers about PSRA practices, participants were asked to provide their opinion on some aspects. Given the widely recognized importance of fairness in decision-making processes, the survey asked respondents about perception of fairness: their own opinion about fairness of current practices, and how they think stakeholders and the public feel about it. Table 5.5 summarizes these findings. Less than half of survey respondents found the current practices to be fair, and the rest were split between “no”, “do not know” and “prefer not to answer”. In assessing others’ perceptions, around 60% said they did not know or preferred not to answer.     114 Table 5.5 Participants' views about perceived fairness of current practices of decision-making in PSRA. Are current practices perceived to be fair?  By involved stakeholders By the general public In your opinion Yes 17 (29%) 11 (19%) 27 (47%) No 7 (12%) 6 (10%) 14 (24%) Do not know 26 (45%) 34 (59%) 8 (14%) Prefer not to answer 8 (14%) 7 (12%) 9 (16%)   Participants were asked to rate the overall performance of current PSRA practices. As shown in Figure 5.3, 40% of participants judged it to be good or very good, 28% found it fair, and 24% found it poor or very poor.   Figure 5.3 Participants' evaluation of current performance.  115 Again, these findings should be only understood within the present work and are not representative of the perspectives of academics or decision makers working with the topic. The information in this section is relevant to provide richer insights on the upcoming discussion. 116 Chapter 6: Discussion The present work constitutes the first attempt to describe practices of priority setting and resource allocation in a systematic way. Besides filling a gap in the literature, this work has also the contribution of pursuing an innovative and theoretically robust methodology in the field of international studies on public policy. In addition, it offers a better understanding of current practices of decision-making in PSRA and, thus, it also provides a message to researchers and decision-makers about the elements that need to researched, and addressed in policy-making across countries.  In all countries, health care systems have faced the challenging phenomenon of unsustainable rates of growth. The introduction of novel medical technologies in the market and aging populations are deemed the main drivers of this trend. The need for good practices of decision making in PSRA has become increasingly evident, resulting in a substantial expansion of research in this area and policy makers and decision makers developing initiatives to improve practices in this realm. In spite of the voluminous amount of research work addressing this topic, we identified a gap in the literature regarding mapping and documentation of existing practices of decision making in PSRA, often assumed to be limited to historical allocations, HTA, or the published case studies using decision analysis techniques. Our work intended to rely on a methodology that could offer a better understanding of the existing practices in priority setting. The extensive theoretical discussion in Chapter 1 situates our research findings within the current state of the field of priority setting. Given the hierarchical combination of conventional and directed qualitative content analysis used here, the recap of normative and prescriptive 117 paradigms in the field was used not to limit the research question but rather to structure data collection and provide an ex post analysis of the themes and categories arising from data analysis. I have described the countries included in the study as “high-income countries”, “developed countries”, “industrialized countries”, and yet, none of these expressions make me comfortable. “High-income countries” is not perfect, given the diversity of places that fall within this category. The World Bank classifies high-income countries as those whose GDP per capita is higher equal or greater than US$12,236 (reference year 2016); this encompasses countries as diverse as Uruguay, Oman, Hungary, Latvia, France, Switzerland, Saudi Arabia, and the USA. In general, I am fiercely opposed to the unrestricted use of such an approach to make international comparisons, given the clear lack of comparability. “Developed countries” is inadequate, given its vagueness and its colonialist connotations, and “industrialized countries” is inappropriate, given that the strong economies of the twenty-first century are based on knowledge and technology rather than industry in the traditional sense. Being “industrialized”, apart from being difficult to precisely define, is no longer a good sign of socioeconomic development. Some might suggest “western countries”, but the inclusion of Japan makes one hesitate. Thus, these three terms have been used interchangeably throughout the text, for lack of a better descriptor for the countries included in this work. This aspect could appear a minor detail to the careless reader and worthless to be discussed, but it constitutes a non-trivial symptom of the arrogance and Eurocentric perspective of the science produced in the so-called “West”. The numerous studies on low- and middle-income countries tend to treat equally countries extremely different creating thus a binary imagery representation where on one side there is a world of richness, democracy, and reason, and on the other side, there is a world marked by incompetence, scarcity, backwardness, and emotion. Such shallow and 118 equivocate methodological conceptions, although often used by well-intentioned individuals, perpetuate a colonialist view that relegates the majority of people on Earth as second-class human beings. As a citizen and researcher committed to a science that does not pretend to be neutral but that rather takes a stance beside those who have been historically humiliated and excluded, I could not avoid this discussion.  6.1 The Current Scenario of Priority Setting Our findings showed that priority setting has received different levels of attention and action across different countries. Whereas scarcity has been perceived quite acutely in the United Kingdom, a participant from Switzerland reported the perception that there is sufficient money in the system. In Norway the topic has been discussed since early 1980s, whereas in Austria it has not been object of debate in the public sphere. Consequently, observed practices vary considerably from country to country and even within national contexts. In Sweden and Norway, national guidelines of priority setting were approved by the parliament. The establishment of criteria and principles at the national level has been pointed out as a strategy to make decision-making processes accountable, fair, and systematic. Nonetheless, what was demonstrated in this work is that existing practices still vary considerably nationwide and there has not been a systematic and explicit process of making the actual decisions in priority setting. The national guidance has turned out to be vague and of little practical value. Several decision makers emphasized this aspect as an area for improvement, i.e., the development of explicit procedures to be followed to arrive at a final decision. In Sweden, an ethical platform established three principles to base decisions on: human dignity, need/solidarity, and cost-effectiveness. Similarly, in Norway, the national guidance on PS determines three criteria: 119 benefit, resource consumption, and severity. Although some direction is provided, there remain many vital questions, such as: • How should a decision-making process work? • Which stakeholders should be involved? • How should we put together the available evidence and the values of the involved actors? • How should we weight the criteria? • How should we evaluate the process and outcomes? • How different should the processes be across different levels of governance and administration? These pragmatic questions that remain unanswered in the national guidelines are perceived to be urgent for decision makers, who need to have tools to help them allocate resources optimally. What is clearly detectable in the findings is that efforts to allocate resources efficiently and determine priorities are concentrated on one-off decisions about adopting and reimbursing health care technologies. Many participants indicated that decision making processes in HTA are the existing formal strategy of PSRA in their institutional setting. But this is a particularly contentious view. Given the nature of our qualitative descriptive study, we stayed close to the surface of the data, with the least possible level of interpretation. HTA activities were shown as existing practices of PSRA, following the participants’ perspectives. Nonetheless, this is a mistaken view in our opinion. We contend that HTA is a tool for management of health care technologies that offers evidence to support decision making. It is a value assessment framework, i.e., it enables the 120 appraisal of the value for money of a given technology in comparison to others, but it does not offer an answer on prioritization of investments broadly speaking. In fact, HTA has an overemphasis on adoption of new technologies. The vast majority of efforts on HTA are directed to deciding whether to incorporate a new technology in the package of services covered or reimbursed. This stance operates an erroneous notion of opportunity cost. It usually compares the opportunity cost of not adopting an alternative treatment for the same disease or condition. That would be an acceptable conceptualization if we assumed that health care systems operate at maximum efficiency and do not have budget constraints. Yet, the real-world scenario is quite the opposite. Under a fixed budget, deciding on investing in a new technology necessarily means displacing an existing service; there is no other solution. HTA cannot appraise the opportunity costs of displacing an existing service. That is a correct assumption of CBA from a welfare economics perspective. If we could measure the return of all expenses in the health care system (including capital investments, administrative personnel, etc.) in monetary terms, we would have an immediately comparable measure of opportunity cost. But that is not possible, as discussed in Chapter 1. Even establishing a threshold of WTP for an incremental gain of health does not solve the problem. Following this rationale that comparing technologies against an established threshold is enough to appraise opportunity cost, CADTH’s guidelines on economic evaluation of health care technologies stated to “adopt a ‘supply-side’ estimate of the cost-effectiveness threshold, which assumes that reimbursing a new technology will displace some other technology or health care service”.111 (p13) Yet, despite this argument, CADTH provides no explicit guidance on disinvestment. 121 In reality, although the majority of participants did stick to this view of HTA as a tool of priority setting and resource allocation, two academics from the United Kingdom pointed out that HTA does not explicitly account for opportunity costs and is not a PSRA methodology in itself. One of those even stated that because NICE’s emphasis is on evaluation of new medicines suggested by manufacturers, the medicines budget keeps growing. HTA certainly represents an important and necessary constituent part of a global process of priority setting and resource allocation. By defending that HTA should be seen as a value assessment framework (and not as a PSRA tool), we are not saying that health technology assessment does not belong to the semantic and operational realm of priority setting. Obviously, assessing the use of specific health care technologies represents a vital aspect of efficient management of resources. Nonetheless, PSRA goes far beyond that. A framework designed to allocate resources needs to incorporate manners to assess the opportunity cost of investments, broadly speaking, which HTA is not capable (and not expected) to do. In this sense, we argue that it would be highly beneficial to shift the semantics around HTA. This will be helpful to allow the research community, decision-makers and health professionals to obtain clarity of what each practice aims and can do.  The idea of disinvestment in general was demonstrated to be only incipiently discussed within health care systems. Actually, many participants argued that, even though formal strategies of disinvestment have not been implemented, disinvestment takes place in the sense that HTA decisions about adoption of new technologies presupposes replacement or displacement. As we argued above, however, this stance would be correct only if the entire health care budget were being allocated through a fully rational process where all expenses are ranked according to their 122 cost-benefit or cost-utility ratios, which is neither feasible nor desirable as became evident in the Oregon experiment. Despite the lack of formal initiatives on disinvestment, the concept has become more important given the growing budgetary pressures on health care systems. According to Participant 41, disinvestment has been an acute phenomenon at the regional level in the United Kingdom, where CCGs have often been led to focus on processes to disinvest from treatments and even entire services to stay within the established budget rather than focus on developing practices of priority setting in a more systematic way. Going back to the processes of priority setting, apart from the HTA efforts, allocation of resources is still largely based on historical utilization and carried out through ad hoc decision making. Specific initiatives are not rare, and some have been published as case studies in the scientific literature, mainly involving PBMA (and/or A4R) but also MCDA.47,64,66,112–117 As presented in Chapter 5, several initiatives were tested in Canada, Sweden, and the UK, in different types of health organizations including hospitals, health authorities, and national ministries of health. Yet, these experiences have not been translated into standard systematic practice in most places. In spite of the lack of formal approaches to determine priorities for investment beyond the HTA realm, observing the existing practices offered a rich view of certain phenomena. There seems to be wide agreement on the use of multiple criteria to inform decision making. The majority of formal approaches take more than a single criterion into consideration, even though some places still weight certain criteria disproportionately. This is the case of the Canadian provincial institution, for example, where decisions are decided primarily on the basis of clinical effectiveness, although other elements tend to be raised in the discussions. 123 Another point of convergence is the appraisal of the available evidence. Almost all participants reported that all types of evidence are taken into consideration. This represents a shift from the traditional paradigm of “hard evidence” or “gold standard” status attributed to randomized controlled trials to a more holistic understanding of health care systems. It does not mean, however, that all types of evidence are treated equally. Evidence of clinical effectiveness still enjoys more prestige and is deemed superior to others. An aspect upon which there is no common understanding is appeals mechanisms. While some participants reported that there is appeal in some scenarios— individuals can consult another doctor for a “second opinion” or companies and patient groups can take decisions to court—other participants reported the total absence of appeals or even the debate about it within the same settings. The practices described in this work generally do not have formal structures and mechanisms to engage the general public. Notwithstanding the debate around eliciting and incorporating societal values in decision-making frameworks, almost all participants reported processes without an explicit concern in this realm. In fact, when asked about this issue, several respondents commented that the decision-making processes are held accountable through election of members for the legislative and executive powers. Nevertheless, we tend to consider such position an exaggeration of the actual scope of elections, which rarely have the time or the means to bring such technical topics to the debate. In addition, defending the participation of the public in some manner, either directly through representatives sitting on committees or eliciting public values, constitutes a way to make health care decisions fairer and legitimate and to make the system less influenced by political cycles and external influences. 124 A further related topic is transparency. Whether mentioning it as a strength or an area for improvement, most participants emphasized the importance of transparent and explicit processes of decision making. And even in settings where legally defined guidance exists, such as Norway and Sweden, respondents commented that it is important to be fully transparent in respect to the exact procedures leading to the final decisions, i.e., the actors influencing it, the underlying rationale, the voting/rating system, etc.  6.2 The Underlying Context for Priority Setting The countries included in the study, despite belonging to a generic group of high-income countries, constitute very different societies. They differ in institutional design, e.g., the model of health care system (multi-payer or single-payer, social insurance or public service, etc.), political system (presidentialism or parliamentarism, counties or regions, etc.), infrastructure, economic dependence of certain industries, levels of social and economic inequality, and underlying values (about the role of the state and the size of markets, about equity, about an individual’s responsibilities for their own health, etc.). They represent a diversity of underpinning societal contexts upon which health care systems are created. This is reflected in the ways that certain needs are perceived, on the ways that problems are addressed, and the extent to which countries are affected by local and international issues. Participants from the United Kingdom stated that the scenario of economic crisis worked as an important facilitator for debating priority setting and defending the need to make difficult decisions. However, the Swiss participant suggested that there is enough money in the system and this discussion has not been relevant in the country. While Austrian respondents commented that there is a sense of hierarchy that hinders an adequate participation of actors from different sectors 125 and roles, stakeholder involvement has been practiced in a more successful way in several settings. Norwegian respondents stated that politicians have a good understanding of the topic and have dealt with it over decades, but German politicians avoid the topic because of its electoral consequences. All these examples highlight that processes of decision making in PSRA do not constitute technocratic solutions that can be applied all across the board. They reflect the socio-historical conditions and values embedded in the health care system. The list of facilitators and barriers for the development and implementation of formal PSRA processes, as well as the areas for improvement, found in Table 5.3 and 5.4 respectively, should be interpreted bearing in mind the qualitative and quantitative distinctions among these countries and their levels of governance and care provision.  6.3 Methodological Aspects and Study Limitations The methodology chosen was a survey designed within the realm of qualitative descriptive studies. Its underpinning theoretical reflection, that we carried out in Chapter 3, allowed us to comprehend the strengths and limitations of such an approach. On one hand, it was supposed to deal with some important methodological challenges such as the lack of a database, the difficulty of conducting a document analysis through a grey literature search in multiple languages, and the diversity of possible responses to the problem in question; on the other hand, it presented several limitations. The methodology relies on the word of participants to depict existing institutional practices, which can be misleading. The respondents’ testimonials about the processes of decision making in PSRA tend to vary according to the level of “formal knowledge” (i.e., knowledge of what is 126 formally established and documented) and experience with the setting. So, considering that all participants have good faith in answering the questionnaire, the veracity or faithfulness of the answer may be compromised because of a biased or partial understanding of the actual practices. Obviously, describing a process based on reports of several individuals is desirable to provide a fuller representation of the actual process and to confront inconsistencies or discrepancies among answers. The information obtained for certain settings was particularly weak, given the participation of a single or a few respondents. In some cases, it was not even possible to analyze certain responses due to their lack or inconsistency of information, like Denmark, Switzerland, Germany, and the USA, where only one individual completed the survey and provided an insufficient level of information. In the settings with multiple participants, the notion of data saturation was pertinent because the reading and categorization of a further participant would allow the confirmation of emerging themes. The convergence of topics and ideas was important to illuminate what is perceived as utterly important in each setting. Possible divergence was not used to weaken previously emerged themes, but rather it was used to expand the horizon of codes. At the predominantly descriptive process, divergence could mean disparate and equivocal reports of existing practices, but that did not occur in the present work. The answers did not provide controversial description of processes.  Surveys typically generate low response rates, which can be even lower for qualitative surveys. The strategy of finding potential participants through the scientific and grey literatures tends to return mostly researchers. Relying on them to reach local decision makers through snowball sampling was not successful. Only 8 (out of 58) participants described their primary professional roles as decision makers. Reaching out to decision makers who may have been 127 working with PSRA represented a fundamental component of the expectation that this methodological approach could unveil undocumented practices of decision making. In fact, as presented in Chapter 5, much of our findings focused on strategies well known in the health economics community and well discussed in the literature, such as the NICE work and HTA agencies in Australia. The data analysis was based on a qualitative content analytical approach. As discussed in Chapter 3, the approach was to focus on the factual content of the data. Thus, the explicit value and moral judgements had a secondary role, to provide us more context. In Chapter 5, we presented the data and avoided making assumptions, generalizations, and deep interpretations. Instead, we allowed participants to speak in our analytical narrative, often quoting them verbatim. Some respondents provided references to compose their answers, which were also used to present a more detailed and faithful depiction of the institutional realities being discussed. That held true, for instance, for the consultation of the ISPOR webpage on the Danish health care system106, the website of the Norwegian framework “Nye Metoder”,118 and a paper by Wester and Bringedal.107 Despite the focus on revealing the factual content of existing practices of PSRA, the survey instrument also asked participants about their views on some aspects of the system, such as elements in their settings that could work as facilitators or barriers for the implementation of explicit and formal PSRA processes, as well as strengths and areas for improvement of existing practices. This secondary objective of our work was important to better understand the underlying structures upon which decision-making frameworks are developed and implemented as well as the envisioned pathways for public policies and the research field. Questions 11 and 12 of the survey asked about the perceptions of fairness and the overall rating of current decision-making practices. 128 However, the answers followed no identified pattern and were quite difficult to be analyzed in a meaningful way. Therefore, we simply presented the data without conducting any further analysis.  Another limitation related to data analysis is the fact that only one coder was employed. Although the inclusion of a second coder would not eliminate the contamination of bias into the reading and categorization of the data, it would possibly allow a more robust and objective analysis. In addition, using more than a single coder could provide a more refined identification of relevant themes.   There was significant heterogeneity in our sample of institutional environments, as was expected given the sampling strategies employed to capture maximum variation and include known relevant cases. That was particularly helpful to understand the state-of-art picture of PSRA, shedding light on the strengths and gaps of existing practices. A further step in this work would involve the use of a more specific sampling design to select fewer countries, with representatives of all types of health care systems (social insurance system, public system, market-oriented, etc.), and a stratification of participants per type of health care organization (private, public, etc.) to allow comparisons across sectors and systems.  Overall, the methodology demonstrated an insightful and auspicious strategy for qualitative description of institutional aspects of public services in international studies. Nonetheless, identifying potential participants and engaging an adequate number of them presents a challenge. In our case, the strategy of sampling was not successful in reaching regional and local decision makers in the countries included. Despite this weakness, we believe our study provides a useful mosaic of practices (the factual element) deemed to be PSRA processes and offers a good basis for a better grasp of how priority setting has been understood and translated into practice across many different countries. 129 Chapter 7: Conclusion The provision of health care has consumed more and more resources in virtually all countries in the developed world. Demographic and technological changes have played a significant role in this phenomenon, increasing health cost per capita considerably. Recognizing its unsustainability has made researchers, decision makers, and politicians aware of the need to make choices on where to invest scarce resources. Although scarcity has always existed, it has recently achieved unprecedented levels in the modern history of wealthy nations. Scarcity is not a symptom of a specific period of economic downturn or the consequence of bad administration. Rather, it is a ubiquitous phenomenon of organizations and individuals that have constrained budgets and virtually endless possibilities of demand for spending. Even in a country where scarcity of resources has not been perceived to be a problem, it will likely become evident in the near future, given that the major phenomena bringing budgetary pressures, aging populations and the advent of new technologies, affect all these nations to a significant degree. Having this ongoing phenomenon in mind, we wanted to investigate the practices of decision making in PSRA in health care systems in high-income countries. The intent was not to obtain a complete map or comprehensive database of existing strategies but rather to produce a mosaic or “photo album” of current practices. We relied on an online qualitative survey addressed to academics and decision makers who work or have worked with the topic in the selected countries. Our findings can be briefly summarized as follows: • Resource allocation in health care has largely been based on historical patterns and through ad hoc decisions. 130 • According to participants, the most common PSRA strategy is HTA. In our opinion, HTA should not be understood as a tool for setting priorities and allocating a fixed budget but rather as a value assessment framework that appraises the value for money of a technology in relation to others. • Several initiatives of PSRA have been tested and published, with special emphasis on PBMA, but there is limited evidence that health organizations have continuously used them in a systematic way across countries. • Countries that have established a national policy on priority setting, indicating principles or criteria to base decisions on, have not provided pragmatic guidance to decision makers who still want advice on practical procedures to allocate resources efficiently and fairly. • There is wide understanding that decisions must be based on multiple criteria. Yet, this approach is inconsistently applied in practice. • A point of increasing convergence is the reliance of multiple types of evidence to judge the value of investment options, perhaps identifying pragmatism as a leading principle in PSRA. • Public engagement is not a reality in most places. Decision makers and researchers do not know how and when to involve the general public in the decision-making processes. • Disinvestment frameworks are very rare and the topic itself has only just begun to appear with any regularity. • There seems to be complete absence of debate about what would constitute an appeals mechanism within this context. 131 Overall, our work represents a first attempt to apply a systematic approach based on qualitative descriptive studies to provide a big picture of existing practices of decision making in PSRA. Despite the limitations and difficulties previously discussed, the methodology employed provided an insightful representation of the status quo in this realm. We observed that the debate on priority setting has grown considerably in most developed countries, which positively feeds back the scientific research in the field. Nonetheless, this phenomenon has not translated into the implementation of explicit and formal approaches to allocate fixed budgets in a systematic and widespread manner. In spite of the paramount growth of HTA (which in our view, although possesses a terrific relevance for value assessment of individual technologies, should not be taken as a PSRA approach), most health care systems have advanced very little in this field. Even countries with legally defined policies on PSRA were unable to offer practical tools and operational knowledge to decision makers. Our findings suggest an agenda for the research field on PSRA, which includes the development/improvement of decision-making frameworks that • have well-defined steps, • are based on multiple criteria, • are capable of assessing the opportunity costs involved, • focus on achieving higher value and not on adoption, • engage involved stakeholders and the general public, • make good use and appraisal of all evidence available, and • emphasize transparency, legitimacy, and fairness. 132 Health economics has to show its value by offering practical tools to manage health care systems. Whereas academics keep pursuing a perfect technocratic solution, budgets are largely allocated without any reasonable consideration of opportunity costs and through obscure and ad hoc procedures. Although much has been achieved in the field, there is a need for more pragmatic approaches to support decision making. 133 Bibliography 1.  Tragakes, Ellie, Vienonen, Mikko. Key issues in rationing and priority-setting for health care services. WHO Regional Office for Europe Health Care Systems; 1998. 2.  Klein R, Day P, Redmayne S. Managing scarcity: Priority-setting and rationing in the National Health Service. Buckingham ; Philadelphia: Open University Press; 1996. 3.  Canadian Institute for Health Information. Health Care in Canada, 2011—A Focus on Seniors and Aging. Canadian Institute for Health Information: Ottawa; 2011. 4.  Statistics Canada. Population projections, children and seniors (graph). An aging population, 2016. [cited 2018 Jul 4]. Available from: https://www150.statcan.gc.ca/n1/pub/11-402-x/2010000/chap/pop/pop02-eng.htm 5.  Government of Canada SC. Census in Brief: A portrait of the population aged 85 and older in 2016 in Canada, Census year 2016 [Internet]. 2016 [cited 2018 Jul 4]. Available from: http://www12.statcan.gc.ca/census-recensement/2016/as-sa/98-200-x/2016004/98-200-x2016004-eng.cfm 6.  Canadian Institute for Health Information [Internet]. Health Spending. 2017 [cited 2018 Jul 4]. Available from: https://www.cihi.ca/en/health-spending 7.  Organisation for EconomicCo-operation and Development [Internet]. Health resources - Health spending - OECD Data. [cited 2018 Jul 4]. Available from: http://data.oecd.org/healthres/health-spending.htm 8.  Mitton C, Donaldson C. Priority-setting Toolkit : Guide to the use of economics in healthcare decision making (1) [Internet]. London, GB: John Wiley & Sons, Incorporated; 2009 [cited 2017 Apr 18]. Available from: http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10158782 9.  Smith A. The wealth of nations. CreateSpace Independent Publishing Platform; 2016. 10.  Arrow KJ. Uncertainty and the welfare economics of medical care. Am Econ Rev. 1963;53(5):941–73. 11.  Porter ME. What Is Value in Health Care? N Engl J Med. 2010 Dec 23;363(26):2477–81. 12.  Hargreaves Heap S, Lyons B, Hollis M, Sugden R, Weale A. The theory of choice: A critical guide. Oxford, U.K ; Cambridge, Mass: Wiley-Blackwell; 1992. 13.  Blaug M. Economic theory in retrospect. 5th ed. Cambridge ; New York: Cambridge University Press; 1997. 14.  Hurley JE. Health economics. 1st ed. Toronto: McGraw-Hill Ryerson; 2010. 134 15.  Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. 4th ed. Oxford, United Kingdom ; New York, NY, USA: Oxford University Press; 2015. 16.  Hutton J. Cost-benefit analysis in health care expenditure decision-making. Health Econ. 1992 Dec 1;1(4):213. 17.  Elvik R. Cost-benefit analysis of ambulance and rescue helicopters in Norway: reflections on assigning a monetary value to saving a human life. Appl Health Econ Health Policy. 2002;1(2):55–63. 18.  Fawsitt CG, Bourke J, Murphy A, McElroy B, Lutomski JE, Murphy R, et al. A cost-benefit analysis of two alternative models of maternity care in Ireland. Appl Health Econ Health Policy. 2017 Dec 1;15(6):785–94. 19.  Phelps CE, Mushlin AI. On the (near) equivalence of cost-effectiveness and cost-benefit analyses. Int J Technol Assess Health Care. 1991;7(1):12–21. 20.  Johannesson M. The relationship between cost-effectiveness analysis and cost-benefit analysis. Soc Sci Med. 1995 Aug 1;41(4):483–9. 21.  Bala MV, Zarkin GA, Mauskopf JA. Conditions for the near equivalence of cost-effectiveness and cost-benefit analyses. Value Health. 2002 Jul 1;5(4):338–46. 22.  Dolan P, Edlin R. Is it really possible to build a bridge between cost-benefit analysis and cost-effectiveness analysis? J Health Econ. 2002 Sep 1;21(5):827–43. 23.  Donaldson C. The (near) equivalence of cost-effectiveness and cost-benefit analyses. PharmacoEconomics. 1998 Apr 1;13(4):389–96. 24.  Seixas BV. Welfarism and extra-welfarism: a critical overview. Cad Saúde Pública [Internet]. 2017 [cited 2017 Sep 7];33(8). Available from: http://www.scielo.br/scielo.php?script=sci_abstract&pid=S0102-311X2017000803001&lng=en&nrm=iso&tlng=en 25.  Durkheim É. The rules of sociological methods. 8th ed. New York: Free Press of Glencoe; 1964. 26.  Mooney GH. Challenging health economics. Oxford ; New York: Oxford University Press; 2009.  27.  Kaldor N. Welfare propositions of economics and interpersonal comparisons of utility. Econ J. 1939;49(195):549–52. 28.  Blaug M. Economic theory in retrospect. Cambridge: Cambridge University; 1996. 29.  Rice TH. The Economics of Health Reconsidered. Chicago, Ill: Health Administration Press; 1998. 135 30.  Sen A. Inequality reexamined. New York : Cambridge, Mass: Russell Sage Foundation ; Harvard University Press; 1992. 207 p. 31.  Brouwer WBF, Culyer AJ, van Exel NJA, Rutten FFH. Welfarism vs. extra-welfarism. J Health Econ. 2008 Mar;27(2):325–38. 32.  Culyer AJ. The normative economics of health care finance and provision. Oxf Rev Econ Policy. 1989 Mar 1;5(1):34–58. 33.  Richardson J. Is the incorporation of equity considerations into economic evaluation really so simple? A comment on Cookson, Drummond and Weatherly. Health Econ Policy Law. 2009 Apr;4(2):247–54. 34.  Williams A. Intergenerational equity: an exploration of the “fair innings” argument. Health Econ. 1997 Apr;6(2):117–32. 35.  Norman R, Hall J, Street D, Viney R. Efficiency and Equity: A Stated Preference Approach. Health Econ. 2013 May 1;22(5):568–81. 36.  Keeney RL, Raïffa H. Decisions with multiple objectives: preferences and value tradeoffs. Wiley; 1976. 616 p. 37.  Regier DA, Peacock S. Theoretical Foundations of MCDA. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, editors. Multi-Criteria Decision Analysis to Support Healthcare Decisions [Internet]. Cham: Springer International Publishing; 2017 [cited 2018 May 1]. p. 9–28. Available from: http://link.springer.com/10.1007/978-3-319-47540-0_2 38.  Baltussen R, Niessen L. Priority-setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc. 2006 Aug 21;4:14. 39.  Marsh K, Dolan P, Kempster J, Lugon M. Prioritizing investments in public health: a multi-criteria decision analysis. J Public Health. 2013 Sep 1;35(3):460–6. 40.  Peacock S, Mitton C, Bate A, McCoy B, Donaldson C. Overcoming barriers to priority-setting using interdisciplinary methods. Health Policy Amst Neth. 2009;92(2–3):124–32. 41.  Belton V, Stewart TJ. An Integrated approach to MCDA. In: Multiple criteria decision analysis [Internet]. Springer, Boston, MA; 2002 [cited 2018 May 4]. p. 331–43. Available from: http://link.springer.com/chapter/10.1007/978-1-4615-1495-4_11 42.  Dolan JG. Multi-criteria clinical decision support. Patient Patient-Centered Outcomes Res. 2010 Dec 1;3(4):229–48. 43.  Von Neumann J, Morgenstern O. Theory of games and economic behavior. Princeton: Princeton University Press; 1944. 625 p. 44.  Mitton C, Donaldson C. Twenty-five years of programme budgeting and marginal analysis in the health sector, 1974-1999. J Health Serv Res Policy. 2001;6(4):239–48. 136 45.  Mitton C, Peacock S, Donaldson C, Bate A. Using PBMA in health care priority-setting: description, challenges and experience. Appl Health Econ Health Policy. 2003;2(3):121–7. 46.  Peacock SJ, Mitton C, Ruta D, Donaldson C, Bate A, Hedden L. Priority-setting in healthcare: towards guidelines for the program budgeting and marginal analysis framework. Expert Rev Pharmacoecon Outcomes Res. 2010;10(5):539–52. 47.  Gibson J, DuBois-Wing CM and G. Priority-setting in Ontario’s LHINs: Ethics and Economics in Action [Internet]. Healthcare Quarterly. 2011 [cited 2017 May 29]. Available from: http://www.longwoods.com/content/22649 48.  Gibson J, Mitton C, Martin D, Donaldson C, Singer P. Ethics and economics: does programme budgeting and marginal analysis contribute to fair priority-setting? J Health Serv Res Policy Lond. 2006 Jan;11(1):32–7. 49.  Gibson JL, Singer DKM and PA. Evidence, Economics and Ethics: Resource Allocation in Health Services Organizations [Internet]. Healthcare Quarterly. 2005 [cited 2017 May 29]. Available from: http://www.longwoods.com/content/17099 50.  Gibson JL, Martin DK, Singer PA. Priority-setting in hospitals: Fairness, inclusiveness, and the problem of institutional power differences. Soc Sci Med. 2005 Dec 1;61(11):2355–62. 51.  Rowe G, Frewer LJ. A typology of public engagement mechanisms. Sci Technol Hum Values. 2005 Apr 1;30(2):251–90. 52.  Peacock SJ, Richardson JRJ, Carter R, Edwards D. Priority-setting in health care using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Soc Sci Med. 2007 Feb 1;64(4):897–910. 53.  Mullen PM. Quantifying priorities in healthcare: transparency or illusion? Health Serv Manage Res. 2004 Feb;17(1):47–58. 54.  Garattini L, Padula A. Multiple criteria decision analysis in health technology assessment for drugs: Just another illusion? Appl Health Econ Health Policy. 2018 Feb 1;16(1):1–4. 55.  Campillo-Artero C, Puig-Junoy J, Culyer AJ. Does MCDA trump CEA? Appl Health Econ Health Policy. 2018 Apr 1;16(2):147–51. 56.  Airoldi M. Essays on healthcare priority-setting for population health [Internet] [phd]. The London School of Economics and Political Science (LSE); 2014 [cited 2018 Apr 22]. Available from: http://etheses.lse.ac.uk/916/ 57.  Marsh K, Lanitis T, Neasham D, Orfanos P, Caro J. Assessing the value of healthcare interventions using multi-criteria decision analysis: A review of the literature. PharmacoEconomics Auckl. 2014 Apr;32(4):345–65. 58.  Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, et al. Multiple criteria decision analysis for health care decision making--emerging good practices: Report 2 of the 137 ISPOR MCDA Emerging Good Practices Task Force. Value Health J Int Soc Pharmacoeconomics Outcomes Res. 2016 Apr;19(2):125–37. 59.  McDonald J, Ollerenshaw A. Priority-setting in primary health care: a framework for local catchments. Rural Remote Health. 2011;11(2):1714. 60.  Gallego G, Taylor SJ, Brien JE. Priority-setting for high cost medications (HCMs) in public hospitals in Australia: A case study. Health Policy. 2007 Nov 1;84(1):58–66. 61.  Mentzakis E, Paolucci F, Rubicko G. Priority-setting in the Austrian healthcare system: results from a discrete choice experiment and implications for mental health. J Ment Health Policy Econ. 2014 Jun;17(2):61–73. 62.  Cornelissen E, Mitton C, Davidson A, Reid C, Hole R, Visockas A-M, et al. Fit for purpose? Introducing a rational priority-setting approach into a community care setting. J Health Organ Manag. 2016 Jun 20;30(4):690–710. 63.  Stafinski T, Menon D, McCabe C, Philippon DJ. To fund or not to fund. PharmacoEconomics. 2011 Sep 1;29(9):771–80. 64.  Urquhart B, Mitton C, Peacock S. Introducing priority-setting and resource allocation in home and community care programs. J Health Serv Res Policy. 2008 Jan;13 Suppl 1:41–5. 65.  Mitton C, Dionne F, Donaldson C. Managing healthcare budgets in times of austerity: the role of program budgeting and marginal analysis. Appl Health Econ Health Policy. 2014;12(2):95–102. 66.  Dionne F, Mitton C, Smith N, Donaldson C. Evaluation of the impact of program budgeting and marginal analysis in Vancouver Island Health Authority. J Health Serv Res Policy. 2009;14(4):234–42. 67.  Mitton C, Dionne F, Damji R, Campbell D, Bryan S. Difficult decisions in times of constraint: Criteria based resource allocation in the Vancouver Coastal Health Authority [Internet]. University of British Columbia; 2011 [cited 2017 Nov 5]. Available from: https://open.library.ubc.ca/cIRcle/collections/facultyresearchandpublications/52383/items/1.0223748 68.  Greenberg D, Siebzehner MI, Pliskin JS. The process of updating the National List of Health Services in Israel: Is it legitimate? Is it fair? Int J Technol Assess Health Care Camb. 2009 Jul;25(3):255–61. 69.  Choe YJ, Han OP, Cho H, Bae G-R, Chun B-C, Kim J-H, et al. Prioritization of the introduction of new vaccines to the national immunization program in the Republic of Korea. Vaccine. 2014 Oct 21;32(46):6049–53. 70.  Jeonghoon Ahn, Gaeun Kim, Hae Sun Suh, Sang Moo Lee. Social values and healthcare priority-setting in Korea. J Health Organ Manag. 2012 Jun 15;26(3):343–50. 138 71.  Ashton T, Tenbensel T, Cumming J, Barnett P. Decentralizing resource allocation: Early experiences with district health boards in New Zealand. J Health Serv Res Policy. 2008 Apr;13(2):109–15. 72.  Defechereux T, Paolucci F, Mirelman A, Youngkong S, Botten G, Hagen TP, et al. Health care priority-setting in Norway a multicriteria decision analysis. BMC Health Serv Res. 2012 Feb 15;12:39. 73.  Waldau S. Bottom-up priority-setting revised. A second evaluation of an institutional intervention in a Swedish health care organisation. Health Policy. 2015 Sep;119(9):1226–36. 74.  Waldau S, Lindholm L, Wiechel AH. Priority-setting in practice: Participants opinions on vertical and horizontal priority-setting for reallocation. Health Policy. 2010 Aug 1;96(3):245–54. 75.  Goodwin E, Frew EJ. Using programme budgeting and marginal analysis (PBMA) to set priorities: Reflections from a qualitative assessment in an English Primary Care Trust. Soc Sci Med. 2013 Dec 1;98:162–8. 76.  Bate A, Donaldson C, Ray H. Resource allocation in orthopaedics: economic evaluation to priority-setting. Clin Orthop. 2007 Apr;457:49–56. 77.  Airoldi M. Disinvestments in practice: overcoming resistance to change through a sociotechnical approach with local stakeholders. J Health Polit Policy Law. 2013 Dec;38(6):1149–71. 78.  Airoldi M, Bevan G, Morton A, Oliveira M, Smith J. Requisite models for strategic commissioning: the example of type 1 diabetes. Health Care Manag Sci N Y. 2008 Jun;11(2):89–110. 79.  Wilson E, Sussex J, Macleod C, Fordham R. Prioritizing health technologies in a Primary Care Trust. J Health Serv Res Policy. 2007 Apr;12(2):80–5. 80.  Canham-Chervak M, Hooper TI, Brennan FH, Craig SC, Girasek DC, Schaefer RA, et al. A systematic process to prioritize prevention activities: Sustaining progress toward the reduction of military injuries. Am J Prev Med. 2010 Jan 1;38(1, Supplement):S11–8. 81.  Polisena J, Clifford T, Elshaug AG, Mitton C, Russell E, Skidmore B. Case studies that illustrate disinvestment and resource allocation decision-making processes in health care: A systematic review. Int J Technol Assess Health Care Camb. 2013 Apr;29(2):174–84. 82.  Barasa EW, Molyneux S, English M, Cleary S. Setting healthcare priorities in hospitals: a review of empirical studies. Health Policy Plan. 2015 Apr;30(3):386–96. 83.  Hipgrave DB, Alderman KB, Anderson I, Soto EJ. Health sector priority-setting at meso-level in lower and middle income countries: Lessons learned, available options and suggested steps. Soc Sci Med. 2014 Feb 1;102:190–200. 139 84.  Wiseman V, Mitton C, Doyle-Waters MM, Drake T, Conteh L, Newall AT, et al. Using economic evidence to set healthcare priorities in low-income and lower-middle-income countries: A systematic review of methodological frameworks. Health Econ. 2016 Feb 1;25:140–61. 85.  Cromwell I, Peacock SJ, Mitton C. ‘Real-world’ health care priority-setting using explicit decision criteria: a systematic review of the literature. BMC Health Serv Res. 2015;15:164. 86.  Edwards RT, Charles JM, Thomas S, Bishop J, Cohen D, Groves S, et al. A national Programme Budgeting and Marginal Analysis (PBMA) of health improvement spending across Wales: disinvestment and reinvestment across the life course. BMC Public Health. 2014 Aug 12;14:837. 87.  Mooney G. Priority-setting in mental health services. Appl Health Econ Health Policy. 2002;1(2):65–74. 88.  Haas M, Viney R, Kristensen E, Pain C, Foulds K. Using programme budgeting and marginal analysis to assist population based strategic planning for coronary heart disease. Health Policy Amst Neth. 2001;55(3):173–86. 89.  Angelis A, Kanavos P. Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: The advance value framework. Soc Sci Med. 2017 Sep 1;188:137–56. 90.  Goetghebeur M, Wagner M, Khoury H, Levitt R, Erickson L, Rindress D. PMC50 evidence and value: impact on decision making—the EVIDEM framework and potential applications. Value Health. 2008 May 1;11(3):A183–4. 91.  Airoldi M, Morton A, Smith JAE, Bevan G. STAR—people-powered prioritization: A 21st-century solution to allocation headaches. Med Decis Making. 2014 Nov;34(8):965–75. 92.  Kapiriri L, Razavi D. How have systematic priority-setting approaches influenced policy making? A synthesis of the current literature. Health Policy. 2017 Sep 1;121(9):937–46. 93. Tsourapas A, Frew E. Evaluating ‘success’ in programme budgeting and marginal analysis: A literature review. J Health Serv Res Policy. 2011 Jul 1;16(3):177–83. 94.  Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. 2000 Aug 1;23(4):334–40. 95.  Sandelowski M. What’s in a name? Qualitative description revisited. Res Nurs Health. 2010 Feb 1;33(1):77–84. 96. Alasuutari P. Researching culture: Qualitative method and cultural studies. Thousand Oaks, CA: Sage Publications; 1995. 97. Kvale S. Interviews: an introduction to qualitative research interviewing. Thousand Oaks, CA: Sage Publications; 1996. 140 98. Maxwell J. Understanding and validity in qualitative research. Harv Educ Rev. 1992 Sep 1;62(3):279–301. 99. Agar M. We have met the other and we’re all nonlinear: Ethnography as a nonlinear dynamic system. Complexity. 2004 Nov 1;10(2):16–24. 100. Patton MQ, Patton MQ. Qualitative research & evaluation methods. 3rd ed. Thousand Oaks, CA: Sage Publications; 2002. 101. Morgan DL. Qualitative content analysis: A guide to paths not taken. Qual Health Res. 1993 Feb 1;3(1):112–21. 102. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005 Nov 1;15(9):1277–88. 103. Sandelowski M, Barroso J. Handbook for synthesizing qualitative research. New York, NY: Springer Pub. Co; 2007. 284 p. 104. Ragin CC. The comparative method: moving beyond qualitative and quantitative strategies. Berkeley: University of California Press; 1987. 185 p. 105. Geertz C. The interpretation of cultures; selected essays. New York: Basic Books; 1973. 106. International Society for Pharmacoeconomics and Outcomes Research [Internet]. Pharmaceutical HTA and reimbursement processes - Denmark. [cited 2018 Jun 3]. Available from: https://www.ispor.org/HTARoadMaps/Denmark.asp 107. Wester G, Bringedal B. The Norwegian National Council for Priority-setting in Health Care: decisions and justifications 1. Health Econ Policy Law. 2018 Apr;13(2):118–36. 108. Heintz E, Arnberg K, Levin L-Å, Liliemark J, Davidson T. The impact of health economic evaluations in Sweden. Z Für Evidenz Fortbild Qual Im Gesundheitswesen. 2014 Jan 1;108(7):375–82. 109. Browman GP, Manns B, Hagen N, Chambers CR, Simon A, Sinclair S. 6-STEPPPs: A modular tool to facilitate clinician participation in fair decisions for funding new cancer drugs. J Oncol Pract. 2008 Jan;4(1):2–7. 110. Waldau S. Creating organisational capacity for priority-setting in health care: Using a bottom-up approach to implement a top-down policy decision. DIVA [Internet]. 2010 [cited 2018 May 23]; Available from: http://urn.kb.se/resolve?urn=urn:nbn:se:umu:diva-36316 111. CADTH. Guidelines for the economic evaluation of health technologies: Canada. 4th ed. Ottawa; 2017. 112. Mitton C, Donaldson C, Shellian B, Pagenkopf C. Priority-setting in a Canadian surgical department: a case study using program budgeting and marginal analysis. Can J Surg J Can Chir. 2003;46(1):23–9. 141 113. Mitton C, Donaldson C. Setting priorities in Canadian regional health authorities: a survey of key decision makers. Health Policy Amst Neth. 2002;60(1):39–58. 114. Mitton C, Patten S, Donaldson C. Listening to the decision makers: sustainability of PBMA in Alberta. Appl Health Econ Health Policy. 2004;3(3):143–51. 115. Smith N, Mitton C, Hiltz M-A, Campbell M, Dowling L, Magee JF, et al. A qualitative evaluation of program budgeting and marginal analysis in a Canadian pediatric tertiary care institution. Appl Health Econ Health Policy. 2016;14(5):559–68. 116. Mitton CR, Donaldson C. Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis (PBMA). Health Policy Amst Neth. 2003;64(3):335–48. 117. Mitton C, Patten S, Donaldson C, Waldner H. Priority-setting in health authorities: moving beyond the barriers. The Calgary experience. Healthc Q Tor Ont. 2005;8(3):49–55. 118. Norwegian Health Care Service. [Internet]. nyemetoder.no. [cited 2018 Jun 6]. Available from: https://nyemetoder.no/english  142 Appendices Appendix A Literature Review Search Strategy Database: MEDLINE (OVID)  Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations & and Ovid MEDLINE(R) <1946 to Present>  Search Name: 2017 Res M10  Date: Nov.18, 2017  Search Strategy: -------------------------------------------------------------- 1     (framework or frameworks).tw,kw. (213264) 2     (tool or tools).tw,kw. (588532) 3     case stud$.mp. (89455) 4     (approach or approaches).tw,kw. (1447206) 5     or/1-4 [Frameworks] (2153708) 6     *resource allocation/ (3486) 7     *health care rationing/ (6485) 8     *health priorities/ (5178) 9     or/6-8 (13437) 10     5 and 9 (2152) 11     comment/ or editorial/ or letter/ or news/ (1858162) 12     10 not 11 (2083) 13     limit 12 to yr="2007 -Current" (1077) 14     limit 13 to English language (998) 15     commissioning.mp. (3703) 16     5 and 15 (923) 17     comment/ or editorial/ or letter/ or news/ (1858162) 18     16 not 17 (911) 19     limit 18 to yr="2007 -Current" (723) 20     limit 19 to English language (718) 21     20 not 14 (707)  143  Appendix B Inclusion and Exclusion Criteria of Systematic Literature Review     144 Appendix C Extraction Tool  IDENTIFICATION 1. In what country did the implementation take place? 2. What level or part of the system did the priority-setting implementation take place? (e.g., national, provincial, regional, hospital, community care, etc.) 3. What was the scope or context of the implementation? (e.g., drugs, vaccines, disease area, across disease areas, across sectors, etc.) DECISION-MAKING 4. Who establishes the strategic guidance for the organization and how specific is it (i.e., how much room is there for interpretation)? 5. How does the organization establish priorities and make decisions on where to increase or reduce spending? • Is there any formal process or framework that is used for that purpose or is it done on a case by case basis?  • How are requests for funding initiated? Who do they go to? • Is current spending typically reviewed as part of the process? • Are the organization’s priority setting decisions subject to review by external stakeholder(s)? 6. What stakeholders were involved in the decision-making process? (e.g., researchers, policy makers, public members, patients, clinicians) 7. What types of evidence/information are taken into account? (e.g., epidemiological evidence, clinical evidence, economic evidence, expert opinion, patient reported outcomes) EVALUATION 8. Was there a reported discrepancy between “recommendations” and actions taken? 9. Was the implementation formally evaluated and if so what were the findings of the evaluation (specifically, were health outcomes impact assessed)? How successful was the implementation? What are the key lessons learned? 10. What was the level of political involvement? Were notions of equity explicitly considered?  145 Appendix D Summary of the 23 Papers Selected in the Systematic Review  Authors Date Country Level of exercise Area of application Primary decision maker Framework Peacock et al.(52) 2007 Australia Regional Mental health services Executive team PBMA McDonald et al.(59) 2011 Australia Regional Primary care services Executive team Organization-wide priority-setting process with explicit criteria Galego et al.(60) 2007 Australia Hospital Drugs Executive team Organization-wide priority-setting process with explicit criteria Mentzaskis et al.(61) 2014 Austria National General health services Policy makers Organization-wide priority-setting process with explicit criteria (specifically, DCE application) Stafinski et al.(63) 2011 Canada National Health technologies Policy makers HTA set within a broader process for decision making Urquhart et al.(64) 2008 Canada Regional Home and community care Executive team PBMA/A4R Dionne et al.(66) 2009 Canada Regional General health services Executive team PBMA Mitton et al.(67) 2011 Canada Regional Primary care, community care and public health Executive team PBMA Cornelissen et al.(62) 2016 Canada Regional Community care Executive team PBMA Greenberg et al. 2009 Israel National Health technologies Policy makers HTA set within a broader process for decision making Ahn et al. 2012 Korea National Health technologies Policy makers HTA set within a broader process for decision making 146 Authors Date Country Level of exercise Area of application Primary decision maker Framework Choe et al. 2014 Korea National Vaccines Policy makers Ministry level process with explicit steps and criteria Ashton et al. 2008 New Zealand Regional General health services Executive team Organization-wide priority-setting process with explicit criteria Defechereux  et al. 2012 Norway National General health services Policy makers Organization-wide priority-setting process with explicit criteria Waldau et al. 2015 Sweden Regional General health services Executive team Organization-wide priority-setting using A4R conditions Waldau et al. 2010 Sweden Regional General health services Executive team Organization-wide priority-setting using A4R conditions Goodwin et al. 2013 UK Regional General health services Executive team PBMA Bate et al. 2007 UK Regional Orthopedic surgery Executive team PBMA Airoldi et al. 2013 UK Regional Mental health services Executive team Organization-wide priority-setting process with explicit criteria Marsh et al. 2013 UK National Preventative health interventions Policy makers Organization-wide priority-setting process with explicit criteria Airoldi et al. 2008 UK National Diabetes Policy makers Modeling based on an explicit set of criteria Wilson et al. 2007 UK Regional General health services Executive team PBMA Canham-Chervak et al. 2010 US National Military injuries Executive team Organization-wide priority-setting process with explicit criteria    147 Appendix E Initial Pool of Countries to be Sampled Country GDP per capita (USD) Population Australia 49,755.32 24,210,809 Austria 44,757.63 8,731,471 Belgium 41,271.48 11,338,476 Canada 42,348.95 36,264,604 Denmark 53,578.76 5,728,010 Finland 43,433.03 5,495,303 France 36,857.12 66,892,205 Germany 42,161.32 82,487,842 Hong Kong SAR, China 43,740.99 7,336,600 Ireland 64,175.44 4,749,777 Israel 37,180.53 8,546,000 Italy 30,668.98 60,627,498 Japan 38,972.34 126,994,511 Korea, Rep. 27,538.81 51,245,707 Kuwait 27,359.23 4052584 Netherlands 45,637.89 17,030,314 New Zealand 39,412.49 4,693,200 Norway 70,867.94 5,236,151 Portugal 19,871.72 10,325,452 Saudi Arabia 20,028.65 32,275,687 Singapore 52,962.49 5,607,283 Spain 26,616.49 46,484,533 Sweden 51,844.76 9,923,085 Switzerland 79,887.52 8,372,413 United Arab Emirates 37,622.21 9,269,612 United Kingdom 40,412.03 65,595,565 United States 57,638.16 323,127,513  148  Appendix F List of Organizations’ Websites Used in the Grey Literature Search to Identify Potential Participants  Organization Official Website Address MSAC www.msac.gov.au HealthPACT https://www.health.qld.gov.au/healthpact/ PBAC https://pbac.pbs.gov.au/ Norwegian Council for Quality Improvement and Priority Setting in Health Care http://www.prioritering.no/ SHTG http://www.healthcareimprovementscotland.org/ Osteba http://www.osakidetza.euskadi.eus/ Avalia-T http://www.sergas.es SBU http://www.sbu.se NICE www.nice.org.uk HTAi www.htai.org INAHTA htttp://www.inahta.org ISPOR www.ispor.org EUnetHTA http://www.eunethta.eu IHEA www.healtheconomics.org AHRQ http://www.ahrq.gov/ CADTH www.cadth.org     149 Appendix G Survey Instrument (Introductory Message, Consent Form, Questionnaire) First of all, thanks for being part of this study!  Our research team is composed by: - Dr. Craig Mitton (principal investigator), Professor, School of Population and Public Health, University of British Columbia; - Mr. Brayan Seixas (study coordinator), MSc Student, Research Assistant, School of Population and Public Health, University of British Columbia; - Dr. Dean Regier (co-investigator), Professor, School of Population and Public Health, University of British Columbia.   In the present survey, we are interested in understanding the strategies and rationales commonly used for priority setting and resource allocation in the context of health care systems in high-income countries. Your participation will be of great importance in comprehending the processes taking place within your national/regional/local health system. This is a qualitative survey containing closed-ended and open-ended questions and it will take approximately 20 minutes of your time.  Please keep the following definitions in mind as you answer each question: • “Priority setting” refers to the process of determining preferences for allocating the resources available. Given that virtually every organization operates with a limited budget, priorities are normally established in order to decide where the financial resources should be placed. This process of setting priorities for investment is what we refer to as “priority setting”. • “Disinvestment” refers to the process of partial or total withdrawal of services. The reduction or elimination of covered technologies and procedures can be due to a variety of reasons, such as budget cutbacks, client dissatisfaction, realization that a given health technology is unsafe, low value, inefficient or obsolete, etc. • The questions here often ask you about “the organization you are most familiar with”, which refers to the level of governance or institution you are mainly involved with, either as a staff member or as an external researcher.   Completion of this survey is voluntary. You may discontinue at any time by exiting the survey page. Survey responses will not be anonymous, but they will be kept in strict confidence by the research team at the School of Population and Public Health, University of British Columbia, Vancouver.  If you consent to participate in this survey, please continue. If you have any concerns or complaints about your rights as a research participant and/or your experiences while participating in this study, contact the Research Participant Complaint Line in the UBC Office of Research Ethics at [omitted phone number] or if long distance email [omitted email address] or call toll free [omitted phone number]. o I consent, begin the study o I do not consent, I do not wish to participate  150  Questions  Personal Information o Name: ______________________________________________________________ o Country (current place of work): ___________________________________________ o Email address: _________________________________________________________    1) Which job title best designates your main role? o Decision maker o Academic / Researcher    2) What is the organizational context of priority setting within the health care system you are most familiar with?  a) In terms of level of governance: ▢ National level ▢ State or Provincial level ▢ Regional level (e.g., health authority, health district, health region) ▢ Single organizational level (e.g., hospital, community organization, residential care facility) ▢ Other. Please specify: ________________________________________   151  b) And in terms of scope or context? (Check ALL that apply.) ▢ Drugs ▢ Non-drug health technologies ▢ Work force / Human resources ▢ Specific fields of care (e.g., primary care, cancer, public health). Please name them:  ____________________________________________________ ▢ Other(s). Please specify: ___________________________________________    3) How does the organization you are most familiar with establish priorities to invest in? Is there any formal process or framework that is used for that purpose? What is the underlying rationale or principles in decision making (even if there is no formal process)? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________    4) Conversely, how are priorities established for disinvestment? Is there a formal process for that purpose and if so, is it linked to the investment priority-setting process? On what basis/rationale are decisions for disinvestment made? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________      152  5.a) Which stakeholders are commonly involved in the process of decision-making in priority setting?  Check ALL that apply. ▢ Politicians ▢ Managers / Administrators ▢ Health professionals ▢ Patients ▢ Other(s). Please specify: _____________________________________    5.b) How are these stakeholders involved? How is their input incorporated into the process of decision-making? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________    6) Does the general public participate in the process in any way? If so, is there direct involvement somehow or are there specific strategies to elicit public preferences? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________    153 7.a) What types of evidence/information are taken into account?        Check ALL that apply. ▢ Epidemiological evidence ▢ Clinical evidence ▢ Economic evidence ▢ Expert opinions ▢ Other(s). Please specify: ________________________________________    b) And how specifically is evidence integrated into the process of decision-making? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________    8) What are the facilitators for the development and implementation of formal and explicit processes of priority setting you can identify in the context you are involved with or study (e.g., strong leadership, culture to learn, openness to change, etc.)?  ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________    9) What are the barriers for the development and implementation of formal and explicit processes of priority setting you can identify in the context you are involved or study (e.g., discontinuity of personnel, misalignment of incentives, lack of trust between stakeholders, etc.)? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________    154 10) Is there any mechanism for appealing decisions once made? o Yes o No o Uncertain    10.a) If yes, please explain this appeals process: ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________    11) Are the current practices perceived to be fair?  Yes No Prefer not to answer Do not know By involved stakeholders  o  o  o  o  By the general public o  o  o  o  In your opinion o  o  o  o      12) Overall, how would you rate the current priority-setting process in terms of performance?  Very poor Poor Fair Good Very good Overall performance  o  o  o  o  o        13) In your view, what are the main strengths within the current priority-setting processes? ________________________________________________________________ ________________________________________________________________ 155 ________________________________________________________________ ________________________________________________________________    14) What specific areas could be improved within the current priority-setting processes? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________   Do you know other researchers or decision makers in your country that may also be able to contribute to this project?  Could you please list below their names and institutional affiliation (and/or email address), as long as this is publicly available information? Otherwise, please feel free to share the survey link with these potential participants. o 1.  ________________________________________________ o 2.  ________________________________________________ o 3.  ________________________________________________ o 4.  ________________________________________________ o 5.  ________________________________________________     156 Appendix H Standard Invitation Message Dear Dr. (Participant Name),  On behalf of Dr. Craig Mitton, professor at the University of British Columbia, I am writing to invite you to participate in an international research project that aims to map the existing practices of priority setting and resource allocation in health care systems of high-income countries.   In order to describe the strategies employed for priority setting within the context of health care systems, we are conducting a survey with key decision makers and health researchers from several countries. The reason why you are being invited is that our search processes identified you as an important name on priority setting in your national/regional context.   Thus, we would like to invite you to participate in our study considering your background and experience. To access the survey, please click here OR copy and paste the following URL into your internet browser: https://ubc.ca1.qualtrics.com/jfe/form/SV_8vHonlmcATTD9Sl  This is expected to take 20 minutes of your time. If you prefer, you may also answer the questionnaire by phone or Skype. In this case, all you need to do is to reply to this email and we will arrange a time to connect.  We very much appreciate your willingness to participate in this international survey. Your responses will not be attributable to you individually. All findings will be publicized in a way that no specific individual or organization can be identified. At the end of the study, all participants will receive by email a brief summary of the project findings.  Participation in the study is voluntary.  If you choose not to participate, please disregard this message. If you know of others in your country that may also be able to contribute to this project, please feel free to forward this message to them or reply with their contact details.  Sincerely, Brayan Seixas Study Coordinator (MSc Student / Research Assistant) School of Population and Public Health University of British Columbia Vancouver, BC – Canada  To remove your contact from our list, please follow this link: ${l://OptOutLink?d=opt-out-link}  157 Appendix I Snowball Sampling Invitation Message Dear Dr. (Participant Name),  On behalf of Dr. Craig Mitton, professor at the University of British Columbia, I am writing to invite you to participate in an international research project that aims to map the existing practices of priority setting and resource allocation in health care systems of high-income countries.   In order to describe the strategies employed for priority setting within the context of health care systems, we are conducting a survey with key decision makers and health researchers from several countries. The reason why you are being invited is that, as part of our snowball sampling process, you were referred by Dr. (Referee Name) as a key potential participant in our study.  Thus, we would like to invite you to participate in our study considering your background and experience. To access the survey, please click here OR copy and paste the following URL into your internet browser: https://ubc.ca1.qualtrics.com/jfe/form/SV_8vHonlmcATTD9Sl  This is expected to take 20 minutes of your time. If you prefer, you may also answer the questionnaire by phone or Skype. In this case, all you need to do is to reply to this email and we will arrange a time to connect.  We very much appreciate your willingness to participate in this international survey. Your responses will not be attributable to you individually. All findings will be publicized in a way that no specific individual or organization can be identified. At the end of the study, all participants will receive by email a brief summary of the project findings.  Participation in the study is voluntary.  If you choose not to participate, please disregard this message. If you know of others in your country that may also be able to contribute to this project, please feel free to forward this message to them or reply with their contact details.  Sincerely, Brayan Seixas Study Coordinator (MSc Student / Research Assistant) School of Population and Public Health University of British Columbia Vancouver, BC – Canada  To remove your contact from our list, please follow this link: ${l://OptOutLink?d=opt-out-link}  


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items