Open Collections

UBC Faculty Research and Publications

‘Real-world’ health care priority setting using explicit decision criteria: a systematic review of the… Cromwell, Ian; Peacock, Stuart J; Mitton, Craig Apr 17, 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-12913_2015_Article_814.pdf [ 932.73kB ]
JSON: 52383-1.0221337.json
JSON-LD: 52383-1.0221337-ld.json
RDF/XML (Pretty): 52383-1.0221337-rdf.xml
RDF/JSON: 52383-1.0221337-rdf.json
Turtle: 52383-1.0221337-turtle.txt
N-Triples: 52383-1.0221337-rdf-ntriples.txt
Original Record: 52383-1.0221337-source.json
Full Text

Full Text

RESEARCH ARTICLE‘Real-world’ health care prexplicit decision criteria: athe demand for programs, services, and technologies thatimprove human health, and the need for fiscal restraintmakers have an ethical obligation to allocate those re-sources in a way that is fair, transparent, and accountableCromwell et al. BMC Health Services Research  (2015) 15:164 DOI 10.1186/s12913-015-0814-3of these decisions [2-6] and must be considered as well.Vancouver, CanadaFull list of author information is available at the end of the articleand the reality of a finite resource pool. All health care de-cision making involves making choices between many at-tractive alternatives, and saying ‘no’ to some things thatmight be desirable and valuable.[1]. Further, health care decisions should be made accord-ing to the available evidence relating to a number ofpossible decision making criteria, including effectiveness,cost-effectiveness, equity considerations, feasibility, afford-ability etc. of the proposed program, service, or technology.But allocation decisions are also influenced by a numberof factors other than medical and health economic evi-dence – pragmatic issues of organizational structure andpolitical realities are legitimate and important components* Correspondence: icromwell@bccrc.ca1Canadian Centre for Applied Research in Cancer Control, British ColumbiaCancer Agency, Vancouver, Canada2Department of Cancer Control Research, British Columbia Cancer Agency,Background: Health care decision making requires making resource allocation decisions among programs, services,and technologies that all compete for a finite resource pool. Methods of priority setting that use explicitly definedcriteria can aid health care decision makers in arriving at funding decisions in a transparent and systematic way.The purpose of this paper is to review the published literature and examine the use of criteria-based methods in‘real-world’ health care allocation decisions.Methods: A systematic review of the published literature was conducted to find examples of ‘real-world’ prioritysetting exercises that used explicit criteria to guide decision-making.Results: We found thirty-three examples in the peer-reviewed and grey literature, using a variety of methods andcriteria. Program effectiveness, equity, affordability, cost-effectiveness, and the number of beneficiaries emergedas the most frequently-used decision criteria. The relative importance of criteria in the ‘real-world’ trials differedfrom the frequency in preference elicitation exercises. Neither the decision-making method used, nor the relativeeconomic strength of the country in which the exercise took place, appeared to have a strong effect on the typeof criteria chosen.Conclusions: Health care decisions are made based on criteria related both to the health need of the populationand the organizational context of the decision. Following issues related to effectiveness and affordability, ethicalissues such as equity and accessibility are commonly identified as important criteria in health care resourceallocation decisions.Keywords: Programme budgeting and marginal analysis, Multi-criteria decision analysis, Health care priority setting,Health priorities, Health care rationing, Health care decision making, Literature review, Literature synthesisBackgroundHealth care decision making requires the balancing ofBecause health care resources are, in many jurisdictions,provided through public subsidy, health care decisionthe literatureIan Cromwell1,2*, Stuart J Peacock1,2,3 and Craig Mitton3,4Abstract© 2015 Cromwell et al.; licensee BioMed CentCommons Attribution License (http://creativecreproduction in any medium, provided the orDedication waiver (http://creativecommons.orunless otherwise stated.Open Accessiority setting usingsystematic review ofral. This is an Open Access article distributed under the terms of the, which permits unrestricted use, distribution, andiginal work is properly credited. The Creative Commons Public Domaing/publicdomain/zero/1.0/) applies to the data made available in this article,Similar to PBMA, MCDA is conducted with the inputCromwell et al. BMC Health Services Research  (2015) 15:164 Page 2 of 11Two popular proposed methods for guiding the settingof health care resource allocation priorities are ProgrammeBudgeting and Marginal Analysis (PBMA) and Multi-Criteria Decision Analysis (MCDA). PBMA involves thelisting of all relevant activities and their resource re-quirements, the evaluation of the effectiveness of theseactivities according to a set of explicit criteria, and theapplication of that evaluation to the available budget[7]. The PBMA process can be summarized by askingfive questions [8]:1. What resources are available in total?2. In what ways are these resources currently spent?3. What are the main candidates for more resourcesand what would be their level of effectiveness?4. Are there any areas of care which could be providedto the same level of effectiveness but with fewerresources to fund candidates from (3) (i.e.,addressing technical efficiency)?5. Are there areas of care which, despite beingeffective, should receive fewer resources because aproposal from (3) is more effective (per dollar spent)(i.e., addressing allocative efficiency)?PBMA exercises are commonly conducted by an advis-ory panel of expert stakeholders, and should be accompan-ied by an evaluation of the outputs to ensure allegiancewith the priorities and needs of the organization [9].MCDA involves the numerical quantification of themerit of competing options, according to explicit decisioncriteria [10]. The primary aim of MCDA is to developmodels of decision maker objectives and their value trade-offs so that options under consideration can be comparedwith each other in a consistent and transparent manner[11]. A key principle is that decisions between differentoptions (for example different interventions) shouldbe consistent with stakeholders’ objectives. MCDA istransparent in that it shows that decisions are the lo-gical implications of those objectives. In MCDA, objec-tives are deemed to be within the discretion of thedecision-makers. That is, they are not predetermined bysome underlying theory from economics or ethics (suchas utilitarianism).MCDA then typically consists of two overarchingstages [10,11]. First, problem structuring involves gener-ating a set of alternatives and a set of criteria againstwhich the alternatives are to be evaluated and compared.In order to structure the problem, the first questions toask are ‘what priority setting objectives do decision-makers wish to pursue?’And, ‘what locally relevant criteriado decision-makers use when deciding between alterna-tive interventions?’ Objectives are the principles that de-termine priority setting policies (e.g. improving populationhealth) whereas criteria are the standards that alternativeof decision makers and relevant stakeholders in the ultim-ate resource allocation decision [10]. In addition, MCDAcan be employed within the broader PBMA approach asthe mechanism for benefit measurement to inform alloca-tion or re-allocation recommendations. As such, these ap-proaches in our view are best viewed as complementary.In a time when organizations are adopting explicitcriteria-based decision methods and deciding whichmethod is the best fit for their organization [12,13], it isimportant to examine the criteria that have been used inprevious decisions. While reviews of the priority-settingliterature have been conducted [2,14], these studies havefocused on hypothetical exercises and stated preferencesrather than cases where decision-making bodies havehad to make decisions under the actual constraints ofbudgetary and political realities.The purpose of this paper is to summarize the avail-able literature on health care decision-making where ex-plicit criteria-based methods like PBMA or MCDA wereused (i.e., a set list of factors were weighed against eachother according to some underlying framework), in orderto examine the criteria used by decision makers in ‘realworld’, rather than hypothetical, settings. The added valueof this paper is that our review, based on significant ex-perience in this field over the past 12 years, focused on‘real-world’ priority setting using explicit decision criteria.These examples may be interpreted as a better predictorof future health care priority setting than decisions madein the abstract.MethodsA systematic review of the literature was conducted, basedon English language search terms used in a previouslypublished review of this literature [15]. Relevant subjectheadings were abstracted from the articles included in thatreview: “Decision Making, organizational”; “Health Carerationing”; “Health Priorities”; “Budgets”; and “Commu-nity Health Planning”.To broaden the scope of our search to ensure inclusionof exercises using popular decision-making methods, theinterventions are judged by (e.g. health outcomes fromdifferent treatments). Second, model building entails con-structing some form of model which represents decision-makers’ objectives and their value judgements. There arethen two key considerations to be addressed in this typeof model [10,11]: the methods used to describe decision-makers preferences and elicit importance weights fordecision-making criteria; and, the type of aggregationmodel used to combine criteria scores (see Peacock 2009for more detail).terms “Program (me) Budget”, “Marginal Analysis”, and“Multi-Criteria Decision Analysis” as well as theirrespective acronyms (“PBMA”, “MCDA”) were also addedto the search terms.The relevant search terms were used to search theMEDLINE, ECONLit, and PAIS databases. The searchterms were also entered into the Google™ search engine toinvestigate the presence of ‘grey literature’ (i.e., non peer-reviewed publications). Retrieval was limited to docu-ments published between January 1st, 2000 and July 31st,2013, to reflect the relevant time period since the previousliterature review.Articles were identified as potentially eligible based ona review of their abstracts. Potentially eligible items under-went closer examination based on exclusion criteria. Itemswere excluded if they did not meet the following descrip-tion: a description of a priority setting exercise in which afunding decision was made based on a set of explicitly-defined criteria. Articles published in languages other thanEnglish were not included in this review.The reference lists of all items (i.e., excluded and in-cluded) were manually searched for potentially-eligiblewas undertaken. Because this research did not involvehuman participants, no Research Ethics approval wassought or gained.ResultsA total of sixty-five articles were identified as potentiallyeligible from the indexed databases. An additional fifty-two were identified from the manual reference search and25 from web search, for a total of 142 potentially eligibleitems. Manual search from the grey literature did not yieldany additional unique items. After applying the exclusioncriteria, thirty-five articles were identified as eligible. Twopeer-reviewed items were duplicates of items from thegrey literature and were deemed redundant, bringing thetotal number of included items to thirty-three.Study characteristicsCharacteristics of the included studies, including the set-ting, method of decision-making used, and the result ofCromwell et al. BMC Health Services Research  (2015) 15:164 Page 3 of 11items. The reference-identified items underwent the samescrutiny and application of exclusion criteria. The searchstrategy is diagrammed in Figure 1. One reviewer (IC)conducted the review and data abstraction from all stud-ies. Two other reviewers (SJP and CM) verified the accur-acy of the search process and the application of theexclusion criteria. Disagreements about eligibility were re-solved through consensus. Because studies included in thisexercise were descriptive rather than quantitative, andbecause of the nature of the research question, no for-mal assessment of the quality or bias of included itemsFigure 1 Search strategy and results.the exercise (e.g., a change in policy, a list of priorities,etc.), are presented in Table 1. Most exercises were con-ducted in North American or European or European-descended (“Western”) countries, primarily Canada (10)and the United Kingdom (7). There was equal represen-tation of studies using an MCDA framework (15) asusing PBMA (15). Some studies did not explicitly statethe decision-making method, or used a synonymousterm (e.g., “Decision Science”), and were classified asPBMA or MCDA based on the characteristics of themethod used (e.g., generic vs. algorithmic methods ofTable 1 Summary of included itemsStudy Country Setting Weighting method Method Type of decision[22] Australia Hospital Mixture of several differentmethods (ratio, rating scale)PBMA Increased resource allocation forhighly-ranked programs[23] Nepal National level Discrete Choice Experiment MCDA Ranking of 34 possible interventions[24] UK Primary Care Trust Allocation of points method PBMA Prioritizing 4 programs for diabetescare[25] New Zealand Public Health System No weights described PBMA 5 investments, 5 disinvestments[26] Norway Norwegian Health Ministry Discrete Choice Experiment MCDA Ranking of 21 different alternativesamong 5 health domains[27] Canada Health Authority Weights, method notdescribedPBMA 18 investments, 13 disinvestments,$4.5 m reallocation[18] Canada Health Authority 40 points could be allocatedto any of 40 itemsPBMA $16 m reallocated, $1 m releasedthrough service reduction[28] Canada Not specified Weights, method not clear MCDA Creation of priorities list[29] New Zealand Health Authority No weights described PBMA Summary of decisions[30] Canada Municipal District No weights described PBMA Program alternatives prioritized[31] USA Health Authority Percentages (allocation?Ratio?)MCDAa Ranking of 47 programs funded bythe region[32] UK 2 Primary Care Trusts Allocation of points method PBMA 66 proposals approved that metcriteria out of 134 submitted[33] Ghana National level Discrete Choice Experiment MCDA Ranking of 11 health programs[34] Canada Provincial level Discrete Choice Experiment MCDA Development of decision tool[35] UK Primary Care Trust Mix of ratio (for main criteria)and points allocation (forsub-criteria)PBMA £3.37 m disinvested, £2 m used fordefecit reduction[36] Taiwan National Health Insurance Grey incidence mathematicalexpressionMCDA Access to care optimization[37] Korea Hospital Goal programmingmulticriteria decisionmodellingMCDA Staffing and other logisticoptimization for hospital resourceallocation to meet goals[38] Tanzania National Ministry of Health No weights described MCDA Prioritization of 9 programs[39] UK Department of (Public)HealthDiscrete Choice Experiment MCDA Ranking of 14 different preventativehealth measures[40] South Africa Department of Health Rating Scale MCDA Evaluation of LBC as cervical cancerscreening tool[41] Canada Health Authority 40 points could be allocatedto any of 40 itemsPBMA $40 m in resources released, used fordefecit and reinvestment[42] Canada Health Authority Allocation of points method MCDA 9 alternative programs ranked[43] Canada Health Authority Allocation of points method PBMA 44 disinvestments, $4.9 million incost reduction[44] Canada University faculty of medicine Allocation of points method PBMA 55 disinvestments, $2.7 million incost reduction[45] UK Health Authority No weights described MCDA Construction of optimization model;mapping of disinvestments[46] Canada Surgical Department inHospitalNo weights described MCDA Evaluation of 53 health technologies[47] Canada Surgical Services in HealthRegionNo weights described MCDA Development of decision tool[48] UK Primary Care Trust Allocation of points method PBMA Ranking of 7 programs with PBMA,then with ad hoc approach[49] Canada Health Authority No weights described PBMA Additional funding of $200,000Cromwell et al. BMC Health Services Research  (2015) 15:164 Page 4 of 11ofofhotslogCromwell et al. BMC Health Services Research  (2015) 15:164 Page 5 of 11criteria weighting and decision ranking, inclusion ofbudgetary information, etc.).The most common outcome of MCDA exercises was aranked list of alternatives, rather than an explicit fundingdecision of the type that was more common among ex-ercises that used PBMA. Allocation of points, wherebyan agreed-upon number of points can be assigned todifferent categories, was a common method of assign-ing weights to decision criteria. Many MCDA exercisesused Discrete Choice experiments to elicit criteriaweights. A number of studies did not describe anyweighting method, and may have simply considered allcriteria equally important.Choice of decision-making criteriaThe decision-making criteria used in each included itemwere identified and extracted. Where possible, criteriawith identical/similar definitions across different studieswere collapsed into a single criterion (by IC, decisionsreviewed by SJP and CM), to make direct comparisonbetween exercises more possible. A total of seventy-twounique criteria were identified among the includeditems. These criteria are listed in Additional file 1: Ap-pendix A. The most common criteria were: The effectiveness of the program (21 items) Budgetary impact/Affordability (16 items) Reducing inequalities between groups/”Equity”(14 items) Number of people likely to benefit from program/Table 1 Summary of included items (Continued)[50] UK Primary Care Trust Allocation[51] UK Primary Care Trust Allocation[52] Thailand National level Discrete C[53] Thailand National level No weigha – This paper describes its methodology as “decision science”, but the methodointervention (13 items) Ability to access the program/intervention (13 items) Cost-effectiveness or other health economicsevidence (12 items) Quality of the available evidence (10 items)It is important to note that, because of variations in theway in which criteria were described, it may be possiblethat some criteria are ‘redundant’, insofar as some maysimply be more specific iterations of others. For example,the criteria of “poverty reduction” and “age/risk of targetgroup” could both be collapsed into the “Equity” criterion.A conservative approach was taken to combining criteriain this way, preferring to list criteria individually in caseswhere there was ambiguity about whether or not termswere truly synonymous. Similarly, the classification of agiven criterion into a domain was prone to subjectivity –criteria were not always well-described in the text ofthe exercise. We again used a conservative approachfor this classification process, and resolved ambiguitiesthrough consensus.It should also be noted that many studies listed broadcriteria that contained a number of sub-criteria withinthem (e.g., one study listed an “effectiveness” criterionthat included “number of patients”, “individual benefit”,“magnitude of benefit”, “duration of benefit”, “personalnetworks”, “collective benefits”, “population impact”, and“social capital” as included sub-criteria). In cases wherecriteria weights were not provided for each sub-criterion,the broad criteria were preferred.Domains of decision-making criteriaCriteria used on priority setting exercises were classifiedinto ten descriptive ‘domains’, using a classification sys-tem described by Tanios and colleagues [14]:a. Intervention outcomes and Benefitsb. Type of Health Servicec. Disease Impact (burden)d. Therapeutic Contexte. Economic Impactf. Environmental Impact of the Interventiong. Quality/Uncertainty of Evidencepoints method MCDA Ranking of 4 program alternativespoints method MCDA Ranking of 6 different alternativesice Experiment MCDA Ranking of 40 HIV/AIDS interventionsdescribed MCDA Ranking of 17 possible services forinclusion in national insuranceschemey is very similar to MCDA, as was therefore classified that way.h. Implementation Complexityi. Priorities (fairness)j. Overall ContextA full list of all criteria included in each domain is pro-vided in Additional file 1: Appendix A. A proportionalbreakdown of each domain is provided in Figure 2. Eco-nomic Impact and Intervention Outcomes/Benefits werethe two most frequently-cited domains, followed byOverall Context, Disease Impact (burden), and Priorities(fairness). Only two of the included items included aMiscellaneous category (i.e., a category labeled “miscel-laneous” in the report/manuscript itself ).Domain by type of exercise Domain by country of originFigure 2 Decision criteria by domain.Cromwell et al. BMC Health Services Research  (2015) 15:164 Page 6 of 11It is possible that the types of criteria used may differdue to the differences in the setting and methodologybetween PBMA and MCDA exercises. The frequency ofcriteria use within each domain was calculated for eachtype of approach. Results are presented in Figure 3.Overall, the types of criteria used were similar across theexercise types. Differences between the proportion ofdomain in each exercise were not significant (two-tailedt-test; α = 0.05).Figure 3 Criteria domain frequency by priority-setting method (PBMA vs. MIt is similarly possible that the size of the national economyaffects the types of criteria used in health care decisions.Exercises in our study were grouped based on membershipin the G7 group of countries (Canada, France, Germany,Italy, Japan, the United Kingdom, and the United States).The frequency of criteria use within each domain was cal-culated for exercises conducted in G7 and non-G7 coun-tries. Once again, the frequency of criteria was largelysimilar between the two groups. Exercises performed inCDA).non-G7 countries were more likely to consider criteria inthe Disease Impact (Burden) domain than those in G7countries (two-tailed t test, p = 0.002); conversely, exercisesin G7 countries were more likely to consider criteria in theOverall Context domain (p = 0.006) – other differenceswere not statistically significant. Results are presentedin Figure 4.DiscussionOur search yielded 33 distinct ‘real-world’ priority set-ting exercises conducted using explicit decision-makingcriteria. Decisions were made largely along commoncategories of criteria that included the likely impact of aprogram/intervention, the ability of a program/inter-vention to address inequalities, and the political/organizational realities of the entities that are makingfunding decisions.with the pragmatic constraints all health systems areunder.It is worth noting that fewer than half of the items in-cluded in this review explicitly included cost-effectivenessevidence in their decision-making process. While it is cer-tainly true that costs and outcomes are considered inother criteria (i.e., Affordability, Effectiveness of program),incremental cost-effectiveness is not a component of theseanalyses. Health economic evidence is uniquely suited toprovide valuable information when making decisionsabout scarce resources, but was not explicitly consideredin the majority of the identified exercises. This findingis in line with a growing body of research suggestingthat while decision-makers would like to use health eco-nomic evidence, they may not be able to do so [16,17].Health economists and policy makers must continue towork together to determine the best way of makinghealth economic findings more policy-relevant andusable.Cromwell et al. BMC Health Services Research  (2015) 15:164 Page 7 of 11Summary of main findingsOur review found that while health-specific decisionmaking criteria are of primary importance in prioritysetting exercises, criteria relating to organizational andpolitical considerations (i.e., criteria that fall primarilyinto the “Implementation Complexity” and “OverallContext” domains) are also important elements in deci-sion making. Health care delivery must be guided by theavailable biomedical evidence, but as it is still a humanendeavour, political realities must be considered andweighed. Decision makers should be forthcoming aboutthe need to balance the goal of improving human healthFigure 4 Criteria domain frequency by size of national economy (G7 vs. noIt is valuable to note that, after “effectiveness of theprogram” and “affordability”, issues related to equityand fairness were foremost in the decision-makingprocess. This suggests that, at least where decisioncriteria are made explicit, decision makers are con-cerned with addressing systemic inequalities and pro-viding health care on a ‘by-need’ basis, underliningthe important role of ethics in health care decisionmaking [4,18].We found that, broadly speaking, the studies in thisreview fell equally into two categories: MCDA exercisesand PBMA exercises. It is important to note that a num-ber of studies did not explicitly categorize themselves asn-G7).Cromwell et al. BMC Health Services Research  (2015) 15:164 Page 8 of 11either PBMA or MCDA, for example in one case usingthe term “Decision Science” to describe the method-ology. In these cases, we classified all studies in this re-view as either PBMA or MCDA based on the methodsdescribed in the publication. It is conceivable, therefore,that other reviewers may classify them differently. Ourdiscussion of the contrast between “MCDA exercises”and “PBMA exercises” should be read with this caveat inmind.The primary outcome of the MCDA items included inthis review was the prioritization of alternatives. It is notclear from the manuscripts themselves whether the prior-ities identified through the MCDA process became officialpolicy, rather than simply a set of recommendations.By way of contrast, the PBMA studies often refer ex-plicitly to investment/disinvestment actions taken as aresult of the exercise. It may be valuable to policy-makers to know, when deciding what priority settingapproach they prefer, how the approach translatesinto actual policy decisions.Given that the resource constraints in developing econ-omies are greater than those of countries with largerbudgets, the use of criteria-guided priority setting is ofparticular importance in countries with developing econ-omies [19,20]. MCDA was used in a number of countrieswith developing economies, whereas PBMA was used ex-clusively in countries with developed economies (espe-cially Canada and the UK). The preference for MCDAappears to be due in part to the effort of investigators as-sociated with Dr. Rob Baltussen, who is listed as an authoron four of the seven relevant MCDA exercises. In parallel,application of PBMA has tended to be associated with ahandful of applied researchers working in the UK,Australia and Canada. While the socio-geographic dis-parity seen in our review is likely to be a simple reflec-tion of this authorship ‘clustering’ effect, it may beworthwhile to investigate whether there are barriers orrelevant factors to the use of other decision-makingtechniques in these countries; whether there is a truedifference in the suitability of PBMA or MCDA in par-ticular political/financial resource environments.We looked for differences in the type of criteria usedin PBMA vs. MCDA exercises, as well as in G7 countriesvs. the rest of the world. Care should be taken, however,in extrapolating from these results. The small number ofavailable studies and the lack of a consistent definitionof criteria mean that such comparisons are inherentlydifficult to make. Researchers may wish to investigatethe extent to which decision-makers in wealthy coun-tries face different pressures when allocating health re-sources than those faced by decision-makers in lesswealthy countries. Our findings suggest that both PBMAand MCDA can incorporate the criteria that are mostimportant to the local context, and that the decision-making method should be chosen independently of thechosen criteria, based on the feasibility and applicabilityof a given method to that context.Regardless of the method of priority setting used, theuse of explicit criteria is valuable for all levels of healthcare decision making. Because no health care system canfund all possible alternatives, decision makers have anethical obligation to act as good stewards of finite healthcare resources, and should be accountable to the com-munities they serve.LimitationsOur search criteria were based on a previously-publishedreview of PBMA exercises. As a result, it is possible thatthe terms are biased toward one decision-making methodat the expense of others. We intentionally used MeSHterms and included an additional term that specifiesMCDA in order to counteract any potential bias towardPBMA. We also used manual searches of the referencesection of all papers included in the review, in order toensure that as wide a ‘net’ was used in the search. It isworth noting that while the majority of the items in-cluded in this review were found using generic searchterms (e.g., “Decision Making”, “Health Priorities”, etc.),we did supplement our search to use PBMA and MCDA-specific terms as well. It is possible, therefore, that ourreview is biased toward these methods. Regardless, we be-lieve that this exercise represents a fair encapsulation ofthe status of the literature.We chose to classify our criteria according to apreviously-published conceptual framework [14]; how-ever, our classification was subject to our own interpret-ation – criteria were not listed in consistent language,requiring us to make our own decisions. It is possiblethat our interpretations were biased, and that other raterswould classify a given criterion differently. It is addition-ally possible that an interested reader would classify orcombine the listed criteria differently (e.g., “efficiency” and“appropriate use of resources” may be seen as identicalcriteria, though this review counts them separately). Theconservative approach we used – preferring to list criteriaverbatim to reduce rater bias – leaves a great deal of sub-jectivity up to the reader. Accordingly, we have listed thecriteria used in this analysis, as well as the way in whichthe criteria were assigned to domains, in Additional file 1:Appendix A.The conceptual framework we used also makes theseevaluations vulnerable to the bias of the raters. An alter-native framework has been described by Tromp andBaltussen [21] that classifies criteria according to twobroad categories – ‘health system goals’ and ‘health sys-tem building blocks’ – as well as a number of subcat-egories. There is a great deal of overlap between theTanios et al. framework and the one proposed by TrompCromwell et al. BMC Health Services Research  (2015) 15:164 Page 9 of 11and Baltussen, and there is a level of arbitrariness inchoosing one rather than the other. Perhaps import-antly, the Tromp and Baltussen framework specificallydoes not consider “Quality of Evidence” as an independentcriterion, rather treating it as a component that underliesall criteria. Given that some type of evidence quality criter-ion was present thirteen times in the studies in this review,we feel that the Tanios framework accurately encapsulatesthe terms that health care decision makers have used todescribe their work.It is worth noting that health care decisions are oftenmade on an ad hoc basis rather than using a specificframework like MCDA or PBMA – our review includesall published studies, which likely overrepresents the fre-quency with which these two approaches are used. It isunlikely that our search comprises an ‘exhaustive’ reviewof all criteria-based priority setting exercises in healthcare. As many entities within the UK’s National HealthService (NHS) have adopted PBMA as part of its decision-making process, more exercises are likely either in progressor completed without publicly-available documentation.Similarly, successes of MCDA as a priority settingmethod in developing countries will likely yield furtheruse of the process in future decision-making. While nota comprehensive review of all such exercises, this reviewdoes provide insight into the types of criteria thatdecision-making bodies consider important when allo-cating scarce resources.Further discussionThe authors of this review are reliable advocates of PBMAas a priority setting exercise, and are named as authors onseveral of the studies included in this review. One advan-tage of using PBMA is that budgetary analysis is an intrin-sic component of the priority setting process, whichallows for the important process of disinvestment – thereduction of funding to programs that do not deliver ac-ceptable value for money. As health systems and regionsin various parts of the world face pressure to control, or insome cases reduce the size of their budgets, disinvestmentbecomes an increasingly important component of healthcare priority setting. As described in Table 1, several stud-ies were able to find ways to disinvest as a method of re-allocating scarce funds – all of these studies used PBMA.A review conducted by Guindo et al. [2] explored thecriteria used in resource allocation decision-making; how-ever, the Guindo et al. review included focus groups andother activities not explicitly tied to a particular resourceallocation decision, rather than those decisions made in a‘real-world’ setting. As in our review, the criteria of equity,effectiveness, organizational requirements, and availabilityof cost-effectiveness literature were commonly cited asimportant. An important difference between the two stud-ies, however, is that “stakeholder interests & pressures”was a highly influential criterion in the Guindo review,and not in ours. This may suggest that, though theyidentify it as important, decision-makers are less likely toinclude stakeholder input in actual decision-making exer-cises tied to explicit funding decisions. Of course, theoverlap of papers was not great and thus there could alsobe other reasons for this difference including the settingand/ or individuals involved in the given work. Further in-vestigation should be conducted into the use of stake-holder input into decision-making processes, to see how,and at what level, such preferences are incorporated intohealth care decision making.This review contributes to the scientific literature intwo important ways: first, it includes much of the ‘grey’literature that is not present in previously-published re-views of this field. Because a great deal of health caredecision-making is done outside the context of peer-reviewed journals, it is critical for researchers anddecision-makers alike to be able to draw from as broada set of examples as possible when trying to decidewhat sort of criteria are relevant to their particular con-text of interest. Secondly, by focusing on health care re-source allocation decisions that have been made, asopposed to hypothetical exercises, our review reflects aset of decision-making criteria that is more ‘pragmatic’than the more ‘aspirational’ criteria that may be gener-ated in the abstract.ConclusionsThis review points to the criteria that are most importantto health care decision makers in actual policy-setting en-vironments, and builds on previous reviews that includedhypothetical decision criteria. It is important to recognizethat each priority-setting environment has its own uniquechallenges, and the criteria used will reflect this hetero-geneity. However, this review does suggest some conver-gence in those criteria that are most frequently used in a‘real-world’ setting.Additional fileAdditional file 1: Appendix A. Decision Criteria.AbbreviationsMCDA: Multi-Criteria Decision Analysis; PBMA: Programme Budgeting andMarginal Analysis.Competing interestsThe authors declare that they have no competing interests.Authors’ contributionsIC was responsible for the manual searching of the scientific literature andmade the primary argument for inclusion/exclusion of studies. IC performedthe analysis of the data and was primarily responsible for the preparation ofthe manuscript. SJP conceived of the study, was responsible for screening alleligible studies and determining their eligibility. SJP approved all analyses,provided material input to the interpretation of the findings, and wasCromwell et al. BMC Health Services Research  (2015) 15:164 Page 10 of 11responsible for the preparation of the manuscript. CM was responsible forscreening all eligible studies and determining their eligibility. The searchstrategy used in this study was based on work previously conducted by CMand colleagues. CM approved all analyses, provided material input to theinterpretation of the findings, and was responsible for the preparation of themanuscript. All authors read and approved the final manuscript.Authors’ informationSJP and CM are both longtime health services researchers specializing incriteria-guided decision making, particularly PBMA.AcknowledgmentsThe authors wish to acknowledge the support of their colleagues at ARCCand C2E2.The Canadian Centre for Applied Research in Cancer Control (ARCC) isfunded by the Canadian Cancer Society Research Institute grant #019789.Author details1Canadian Centre for Applied Research in Cancer Control, British ColumbiaCancer Agency, Vancouver, Canada. 2Department of Cancer ControlResearch, British Columbia Cancer Agency, Vancouver, Canada. 3School ofPopulation and Public Health, University of British Columbia, Vancouver,Canada. 4Centre for Clinical Epidemiology and Evaluation, University of BritishColumbia, Vancouver, Canada.Received: 14 January 2014 Accepted: 23 March 2015References1. Peacock S, Ruta D, Mitton C, Donaldson C, Bate A, Murtagh M. Usingeconomics to set pragmatic and ethical priorities. BMJ. 2006;332(7539):482–5.2. Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, et al. Fromefficacy to equity: Literature review of decision criteria for resourceallocation and healthcare decisionmaking. Cost Effective Resour Alloc.2012;10(1):9.3. Smith N, Mitton C, Peacock S, Cornelissen E, MacLeod S. Identifying researchpriorities for health care priority setting: a collaborative effort betweenmanagers and researchers. BMC Health Serv Res. 2009;9:165.4. Kapiriri L, Norheim OF, Martin DK. Fairness and accountability forreasonableness. Do the views of priority setting decision makers differacross health systems and levels of decision making? Soc Sci Med.2009;68(4):766–73.5. Mitton C, Donaldson C. Health care priority setting: principles, practice andchallenges. Cost Effective Resour Alloc. 2004;2(1):3.6. Patten S, Mitton C, Donaldson C. From the trenches: views fromdecision-makers on health services priority setting. Health ServManage Res. 2005;18(2):100–8.7. Peacock SJ, Richardson JR, Carter R, Edwards D. Priority setting in healthcare using multi-attribute utility theory and programme budgeting andmarginal analysis (PBMA). Soc Sci Med. 2007;64(4):897–910.8. Mitton C, Donaldson C. Tools of the trade: a comparative analysis ofapproaches to priority setting in healthcare. Health Serv Manage Res.2003;16(2):96–105.9. Mitton C, Donaldson C. Priority setting toolkit: a guide to the use ofeconomics in healthcare decision making. London, UK: BMJ Books; 2004.10. Baltussen R, Niessen L. Priority setting of health interventions: the need formulti-criteria decision analysis. Cost Effective Resour Alloc. 2006;4:14.11. Peacock S, Mitton C, Bate A, McCoy B, Donaldson C. Overcoming barriersto priority setting using interdisciplinary methods. Health Policy.2009;92(2–3):124–32.12. Devlin NJ, Sussex J. Incorporating multiple criteria in HTA: methods andprocesses. Whitehall, London, UK: Office of Health Economics; 2011.13. Thokala P, Duenas A. Multiple criteria decision analysis for healthtechnology assessment. Value Health. 2012;15(8):1172–81.14. Tanios N, Wagner M, Tony M, Baltussen R, Til Jv, Rindress D, et al. Whichcriteria are considered in healthcare decisions? Insights from aninternational survey of policy and clinical decision makers. Int J TechnolAssess Health Care. 2013; e-pub ahead of print.15. Mitton C, Donaldson C. Twenty-five years of programme budgeting andmarginal analysis in the health sector, 1974–1999. J Health Serv Res Policy.2001;6(4):239–48.16. Mitton C, Patten S, Donaldson C, Waldner H. Priority-setting in healthauthorities: moving beyond the barriers. The Calgary experience. Healthc Q.2005;8(3):49–55.17. Ruta D, Mitton C, Bate A, Donaldson C. Programme budgeting and marginalanalysis: bridging the divide between doctors and managers. BMJ.2005;330(7506):1501–3.18. Gibson J, Mitton C, Martin D, Donaldson C, Singer P. Ethics and economics:does programme budgeting and marginal analysis contribute to fair prioritysetting? J Health Serv Res Policy. 2006;11(1):32–7.19. Youngkong S, Kapiriri L, Baltussen R. Setting priorities for healthinterventions in developing countries: a review of empirical studies. TropMed Int Health. 2009;14(8):930–9.20. Kapiriri L, Martin DK. Successful priority setting in low and middle incomecountries: a framework for evaluation. Health Care Anal. 2010;18(2):129–47.21. Tromp N, Baltussen R. Mapping of multiple criteria for priority setting of healthinterventions: an aid for decision makers. BMC Health Serv Res. 2012;12:454.22. Astley J, Wake-Dyster W. Evidence-based priority setting. Aust Health Rev.2001;24(2):32–9.23. Baltussen R, ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, NiessenLW. Priority setting using multiple criteria: should a lung health programmebe implemented in Nepal? Health Policy Plan. 2007;22(3):178–85.24. Baughan S, Fergunson B: Road testing Programme Budgeting and MarginalAnalysis (PBMA) in three English Regions: Hull Teaching Primary Care TrustDiabetes pilot project. In. Edited by England PH. Yorkshire, UK; 2008.25. Bohmer P, Pain C, Watt A, Abernethy P, Sceats J. Maximising health gainwithin available resources in the New Zealand public health system. HealthPolicy. 2001;55(1):37–50.26. Defechereux T, Paolucci F, Mirelman A, Youngkong S, Botten G, Hagen TP,et al. Health care priority setting in Norway a multicriteria decision analysis.BMC Health Serv Res. 2012;12:39.27. Dionne F, Mitton C, Smith N, Donaldson C. Evaluation of the impact ofprogram budgeting and marginal analysis in Vancouver Island HealthAuthority. J Health Serv Res Policy. 2009;14(4):234–42.28. Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C.Combining multicriteria decision analysis, ethics and health technologyassessment: applying the EVIDEM decision-making framework to growthhormone for Turner syndrome patients. Cost Effective Resour Alloc. 2010;8:4.29. Grocott R. Applying Programme Budgeting Marginal Analysis in the healthsector: 12 years of experience. Expert Rev Pharmacoecon Outcomes Res.2009;9(2):181–7.30. Halma L, Mitton C, Donaldson C, West B. Case study on priority setting inrural Southern Alberta: keeping the house from blowing in. Can J RuralMed. 2004;9(1):26–36.31. Honore PA, Fos PJ, Smith T, Riley M, Kramarz K. Decision science: a scientificapproach to enhance public health budgeting. J Public Health Manag Pract.2010;16(2):98–103.32. Iqbal Z, Pryce A, Afza M. Rationalizing rationing in health care: experience oftwo primary care trusts. J Public Health (Oxf). 2006;28(2):125–32.33. Jehu-Appiah C, Baltussen R, Acquah C, Aikins M, d’Almeida SA, Bosu WK,et al. Balancing equity and efficiency in health priorities in Ghana: the use ofmulticriteria decision analysis. Value Health. 2008;11(7):1081–7.34. Johnson-Masotti AP, Eva K. A decision-making framework for theprioritization of health technologies. In: Beach CM, Chaykowsi RP, Shortt S,St-Hillaire F, Sweetman A, editors. Health services restructuring in Canada:new evidence and new directions. Kingson, Ontario, Canada: Queen’sUniversity Policy Studies; 2006.35. Kemp L, Fordham R: Road Testing Programme Budgeting and MarginalAnalysis (PBMA) in three English Regions: Norfolk Mental Health PBMA PilotProject. In. Edited by England PH. Norfolk, UK: University of East AngliaNorwich; 2007.36. Kreng VB, Yang CT. The equality of resource allocation in health careunder the National Health Insurance System in Taiwan. Health Policy.2011;100(2–3):203–10.37. Lee CW, Kwak NK. Strategic enterprise resource planning in a health-caresystem using a multicriteria decision-making model. J Med Syst.2011;35(2):265–75.38. Makundi E, Kapiriri L, Norheim OF. Combining evidence and values inpriority setting: testing the balance sheet method in a low-income country.BMC Health Serv Res. 2007;7:152.39. Marsh K, Dolan P, Kempster J, Lugon M. Prioritizing investments in publichealth: a multi-criteria decision analysis. J Public Health (Oxf) 2012.40. Miot J, Wagner M, Khoury H, Rindress D, Goetghebeur MM. Field testing ofa multicriteria decision analysis (MCDA) framework for coverage of ascreening test for cervical cancer in South Africa. Cost Effective Resour Alloc.2012;10(1):2.41. Mitton C, Patten S, Waldner H, Donaldson C. Priority setting in healthauthorities: a novel approach to a historical activity. Soc Sci Med.2003;57(9):1653–63.42. Mitton C, Mackenzie J, Cranston L, Teng F. Priority setting in the ProvincialHealth Services Authority: case study for the 2005/06 planning cycle.Healthcare Policy. 2006;2(1):91–106.43. Mitton C, Dionne F, Damji R, Campbell D, Bryan S. Difficult decisions intimes of constraint: criteria based resource allocation in the VancouverCoastal Health Authority. BMC Health Serv Res. 2011;11:169.44. Mitton C, Levy A, Gorsky D, Macneil C, Dionne F, Marrie T. Allocating limitedresources in a time of fiscal constraints: a priority setting case study fromDalhousie university faculty of medicine. Acad Med. 2013;88(7):939–45.45. Nuti S, Vainieri M, Bonini A. Disinvestment for re-allocation: a process toidentify priorities in healthcare. Health Policy. 2010;95(2–3):137–43.46. Poulin P, Austen L, Kortbeek JB, Lafreniere R. New technologies and surgicalinnovation: five years of a local health technology assessment program in asurgical department. Surg Innov. 2012;19(2):187–99.47. Poulin P, Austen L, Scott CM, Waddell CD, Dixon E, Poulin M, et al.Multi-criteria development and incorporation into decision tools for healthtechnology adoption. J Health Organ Manag. 2013;27(2):246–65.48. Robson A, Bate A, Donaldson C: Road Testing PBMA in three Englishregions: Newcastle CAMHS PBMA pilot project. In. Edited by England H.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionCromwell et al. BMC Health Services Research  (2015) 15:164 Page 11 of 11Newcastle, UK; 2008.49. Urquhart B, Mitton C, Peacock S. Introducing priority setting and resourceallocation in home and community care programs. J Health Serv Res Policy.2008;13 Suppl 1:41–5.50. Wilson EC, Rees J, Fordham RJ. Developing a prioritisation framework in anEnglish Primary Care Trust. Cost Effective Resour Alloc. 2006;4:3.51. Wilson E, Sussex J, Macleod C, Fordham R. Prioritizing health technologiesin a Primary Care Trust. J Health Serv Res Policy. 2007;12(2):80–5.52. Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteriadecision analysis for setting priorities on HIV/AIDS interventions in Thailand.Health Res Policy Syst. 2012;10:6.53. Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y.Multicriteria decision analysis for including health interventions in theuniversal health coverage benefit package in Thailand. Value Health.2012;15(6):961–70.Submit your manuscript at


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items