UBC Faculty Research and Publications

Identifying research priorities for health care priority setting: a collaborative effort between managers… Smith, Neale; Mitton, Craig; Peacock, Stuart; Cornelissen, Evelyn; MacLeod, Stuart Sep 15, 2009

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-12913_2008_Article_1036.pdf [ 177.23kB ]
JSON: 52383-1.0132553.json
JSON-LD: 52383-1.0132553-ld.json
RDF/XML (Pretty): 52383-1.0132553-rdf.xml
RDF/JSON: 52383-1.0132553-rdf.json
Turtle: 52383-1.0132553-turtle.txt
N-Triples: 52383-1.0132553-rdf-ntriples.txt
Original Record: 52383-1.0132553-source.json
Full Text

Full Text

ralssBioMed CentBMC Health Services ResearchOpen AcceResearch articleIdentifying research priorities for health care priority setting: a collaborative effort between managers and researchersNeale Smith1, Craig Mitton*1,2, Stuart Peacock2,3,4, Evelyn Cornelissen5 and Stuart MacLeod6Address: 1Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute, Vancouver, BC, Canada, 2School of Public and Population Health, University of British Columbia, Vancouver, BC, Canada, 3National Centre for Health Economics, Services, Policy and Ethics, British Columbia Cancer Agency, Vancouver, BC and Toronto, ON, Canada, 4British Columbia Cancer Agency, Vancouver, BC, Canada, 5Faculty of Health and Social Development, UBC Okanagan, Kelowna, BC, Canada and 6Provincial Health Services Authority, Vancouver, BC, CanadaEmail: Neale Smith - neale.smith@ubc.ca; Craig Mitton* - craig.mitton@ubc.ca; Stuart Peacock - speacock@bccrc.ca; Evelyn Cornelissen - ecorneli@interchange.ubc.ca; Stuart MacLeod - smacleod@cw.bc.ca* Corresponding author    AbstractBackground: To date there has been relatively little published about how research priorities are set, and evenless about methods by which decision-makers can be engaged in defining a relevant and appropriate researchagenda. We report on a recent effort in British Columbia to have researchers and decision-makers jointlyestablish an agenda for future research into questions of resource allocation.Methods: The researchers enlisted decision-maker partners from each of British Columbia's six healthauthorities. Three forums were held, at which researchers and decision-makers from various levels in the healthauthorities considered possible research areas related to three key focus areas: (1) generation and use of decisioncriteria and measurement of 'benefit' against such criteria; (2) identification of so-called 'disinvestment'opportunities; and (3) evaluation of the effectiveness of priority setting procedures. Detailed notes were takenfrom each forum and synthesized into a set of qualitative themes.Results: Forum participants suggested that future research into healthcare priority setting would benefit fromstudies that were longitudinal, comparative, and/or interdisciplinary. As well, participants identified two broadtheme areas in which specific research projects were deemed desirable. First, future research might usefullyconsider how formal priority setting and resource allocation projects are situated within a larger organizationaland political context. Second, additional research efforts should be devoted to better understanding andimproving the actual implementation of priority setting frameworks, particularly with respect to issues of changemanagement and the resolution of impediments to action on recommendations for resource allocation.Conclusion: We were able to validate the importance of initial areas posed to the group and observedemergence of additional concerns and directions of critical importance to these decision-makers at this time. Itis likely that the results are broadly applicable to other healthcare contexts. The implementation of this researchagenda in British Columbia will depend upon the ability of the researchers and decision-makers to developparticular projects that fit within the constraints of existing funding opportunities. The process of engagementitself had benefits in terms of connecting decision-makers with their peers and sparking increased interest in thePublished: 15 September 2009BMC Health Services Research 2009, 9:165 doi:10.1186/1472-6963-9-165Received: 4 November 2008Accepted: 15 September 2009This article is available from: http://www.biomedcentral.com/1472-6963/9/165© 2009 Smith et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Page 1 of 9(page number not for citation purposes)use and refinement of priority setting frameworks.BMC Health Services Research 2009, 9:165 http://www.biomedcentral.com/1472-6963/9/165BackgroundAs research typically involves an investment of society'slimited resources, there exists at least some obligation toensure that research activity aligns with the interests,needs and values of the larger community. Applied healthservices research, in particular, focuses on practical trans-lation and uptake of research findings [1]. The knowledgetransfer and exchange literature highlights that regularinteraction between researchers and decisions makers isone of the most effective ways to increase the likelihoodthat research knowledge will be used [2-4]. Such interac-tion also enables each group to better understand the con-texts of the other and the constraints under which eachoperates. It is also critical that this interaction occurs overthe entire research cycle, not just in the latter stages whensome notion of 'transfer' is to take place [4,5]. Given this,it would be of great value for researchers and decision-makers to spend more time developing research direc-tions and potential research questions collaboratively.The field of health care priority setting and resource allo-cation is a good example of an applied health servicesresearch field that should produce findings aligned withthe needs of the end users. However, to our knowledgethere has been limited input to date, at least as reported inthe literature, from decision-makers in helping to identifyrelevant research priorities. There are good opportunitiesin British Columbia to make headway in this regard. Cur-rently there are two research groups-based at the Centrefor Clinical Epidemiology and Evaluation, in Vancouver,and at the British Columbia Cancer Agency -- that havehealth care priority setting as their primary research inter-est. These two groups have collaborated with each otherextensively over the last five years, have attracted provin-cial and national level funding, and have strong relation-ships with senior decision-makers in all six of the BChealth authorities. These decision-makers, noting currentfiscal constraints which are mirrored in many other Cana-dian provinces and in other countries, are keen toimprove priority setting and resource allocation practicesand engage in relevant research of practical importance.Our intent in the current project was to engage decision-makers in jointly developing a set of realistic, mutuallyappealing research priorities. This would be accomplishedthrough a series of interactive forums, organized aroundknown challenges in the priority setting field. We reporthere on the design and implementation of these forums,and the follow up work leading to future research initia-tives. Our description of the processes we undertook in BCwill offer insight both into how research priorities mightbe developed in collaborative fashion, and into specificareas which appear today to be substantive priorities - inLomas, Fulop, Gagnon & Allen identify two approaches todeveloping research priorities, the technical and the inter-pretive [1]. Technical approaches involve the use of exist-ing data to drive priority choice, for instance, based on theprevalence of a disease or economic burden of illness.Interpretive approaches, by contrast, employ interactivediscussion among informed stakeholders to generate pri-orities. This may or may not include substantial back-ground data. Our intent in the current project was tofollow the interpretive model. We were able to identify inthe literature a handful of examples of joint agenda settingamong health researchers and other stakeholders. In whatfollows, we briefly describe some of these examples,which were either provincial or national in scope.In the US, pressure from Congress and other stakeholdersin the 1990s led to efforts to make processes for settingresearch priorities for the National Institutes of Healthmore explicit, and to include a wider degree of input fromthe public and other stakeholders [7,8]. Rosenstock,Olenek, & Wagner describe a process used by NationalInstitute of Occupational Safety and Health (NIOSH) toestablish a national research agenda for occupationalhealth and safety [7]. An approach derived from thismodel was subsequently used in the Pacific Northwestregion specific to the needs of agricultural workers andemployers, public health agencies, and researchers [8]. Inaddition, O'Fallon, Wolfle, Brown, Dearry & Oldendescribe sixteen "Town [Hall] Meetings" conducted over anumber of years by the National Institute of Environmen-tal Health Sciences (NIEHS) [9]. They suggest that thesemeetings had community impact through increased edu-cation and outreach, and generated both new researchand public health policy changes.In 1991, the UK established the National Health Service(NHS) Research & Development program with the intentof making the NHS a central agency in supporting appliedhealth research [10,11]. Lomas et al describe efforts by theNHS Service Development and Organization (SDO)branch to consult on research priority development viawhat they term a 'listening model', including both anexpert forum and local focus groups [1]. Noting that con-sumers were not directly involved in this process, otherresearchers subsequently followed this work with morefocused consultation with service users on priorities, spe-cifically related to midwifery and nursing research [12].This involved a series of five focus groups in locationsaround England.A more recent effort in the UK has perhaps the closestresemblance to our own work. The UK seminar series,"Managing Scarcity in the NHS: Building on Theory,Page 2 of 9(page number not for citation purposes)the short- to medium-term -- for the field of health carepriority setting and resource allocation.Learning from Practice" (2005-2007) had a similar objec-tive to our own--engaging government leaders, healthcareBMC Health Services Research 2009, 9:165 http://www.biomedcentral.com/1472-6963/9/165professionals, and academics to discuss resource alloca-tion challenges (though not aiming to identify researcha-ble projects per se). Although we are not aware of peerreviewed publications yet resulting from this series, infor-mal communication with the organizers suggests anumber of key learnings. Terminology (such as "commis-sioning") is understood differently by the major stake-holders and so different approaches have evolved. As well,participants noted tensions between national-level deci-sions and local health system management needs; effortsto strengthen local capacity were seen as desirable. Fur-ther, the systematic use of explicit priority setting frame-works has had some degree of success in centralinstitutions, such as NICE, and so there was researcherand decision-maker interest in local adaptations of suchwork. Perhaps most importantly, all participants realizedthat cost-effectiveness analysis alone would not suffice toadequately deal with the complexities of real-world deci-sion-making.In Canada, major federal funding agencies have collabo-ratively undertaken three rounds of a stakeholder consul-tation exercise known as "Listening for Direction." Thefirst was conducted in 2001 [1]. It consisted of an environ-mental scan, five regional workshops and a nationalworkshop with invited participants, focused on pressingissues that decision-makers expected to encounter in themedium-term (three to five years). The process wasrepeated a second time in 2004 [13], and the most recentiteration, involving eight partner organizations, tookplace in 2007 [14].MethodsFor this project, key contacts in each of British Columbia'ssix health authorities were recruited as research partners.Recruitment was done purposively; many of these indi-viduals already had long standing interests in priority set-ting and resource allocation in their respectiveorganizations. The decision-maker partners were invitedto attend the forums and/or to send additional interestedand appropriate colleagues. Purposeful efforts were madeto engage a varied mix of people from different sectors,positions and job responsibilities (including for exampleprogram managers, finance managers, Vice Presidents andone CEO). Forums were not recorded but detailed noteswere taken. At least two research team members tooknotes at each forum; afterward these notes were combinedand any differences reconciled. Notes captured both thecontent of the discussion as well as contextual informa-tion which was used in subsequent analysis of the forumresults. A summary of the notes, including potentialresearch questions, was circulated to decision maker par-ticipants after each forum; this allowed them to verify thatgroup potential research questions and general commentsinto a set of overarching key themes; a second author pro-vided additional analysis and review of the findings, whileall authors provided comments. Subsequent to theforums, researchers contacted decision-makers directlyand held one-on-one meetings to elaborate on the find-ings and to discuss specific areas for partnering. Thisserved as additional participant validation of the resultspresented here.The team planning forumsThree key topics were identified and ultimately imple-mented as focus areas animating this series of forums:• generation of decision criteria, deployment of crite-ria and measurement of 'benefit' against such criteria(September 2007)• identification of so-called 'disinvestment' opportu-nities in order to release resources from a given budgetto shift into higher value investment areas (January2008)• evaluation of the effectiveness of priority settingfrom the perspective of ethics and economics, notinga high degree of system complexity where each deci-sion can have multiple effects and organizational con-straints play a major role (April 2008)These choices arose out of the researchers and decision-makers' past experiences with priority setting research inhealth care organizations, and our own assessment ofwhat appeared to be gaps in the literature [15,16]. Impor-tantly, these questions served as a starting point and initialguide to structure discussions, but it was expected thatother related and/or unique issues would emerge as partof the discussion process at each forum. For instance,questions related to public engagement emerged at eachof the three forums. An outside researcher with subject-area expertise was recruited to facilitate discussion of eachtopic.For each forum, two presentations relevant to the day'stheme were delivered: one by a researcher with interests inthe given topic and the second by a decision-maker with'on the ground' experience. Following this, there was anopen, facilitated discussion based on the material pre-sented offering decision-makers the opportunity to dis-cuss and share their own concerns. There was asubsequent facilitated discussion explicitly aimed at elicit-ing ideas for potential research. Each forum was evaluatedusing a set of standard questions, noting whether cleargoals and objectives for the day were stated and met,Page 3 of 9(page number not for citation purposes)their opinions were accurately captured. The forum noteswere analyzed qualitatively by the lead author, in order torequesting comments on the quality of the presentations,and finally participant satisfaction with the degree ofBMC Health Services Research 2009, 9:165 http://www.biomedcentral.com/1472-6963/9/165interactivity. Open-ended comments and recommenda-tions for future forums were also solicited.The Forums took place in Vancouver (BC's largest city andmajor transportation hub, which allowed for the easiestaccess for people coming from different sites around theprovince). They were held at a neutral location (i.e., not atthe offices of a health authority or the researchers' institu-tions). No pre-reading or preparation was required. Eachforum was one half day in length (4-5 hours) whichallowed sufficient time to engage in the discussions butalso was respectful of the many other demands on deci-sion-makers' time.ResultsFeatures of priority setting researchForum participants emphasized three important featuresof future design in priority setting research: longitudinalstudies, comparative studies, and inter- or multi-discipli-nary studies.Longitudinal research would examine how priority settingprocesses in a health care organization develop and evolveover time. For instance, such research should considerhow new approaches may be successfully implementedand maintained in the organization, as well as definingwhich factors facilitate or hinder this. Research could alsoinvestigate growth of organizational trust over time withrespect to both leadership and joint or more collaborative,explicit decision making. It is reasonable to ask if formal-ized approaches to priority setting are conducive to suchtrust-building (I:8). [Citations here are from the researchteam's notes from the forums. These should be read as fol-lows: [Forum] I: [page] 8.]Participants welcomed opportunities for comparativeresearch, including opportunities for quasi-experimentaldesigns, such as studies of pre-post implementation of anexplicit approach to decision-making. One participantsuggested that it would be interesting to know if a require-ment that resource re-allocation proposals be submittedfrom interdepartmental or collaborative teams wouldresult in different results then the traditional siloapproach to proposal submission (I:10). Organizationshave distinct cultures--some seek to be on the 'cuttingedge' in implementing new approaches to care and servicedelivery, while others are more content to maintain stand-ard and accepted routines. How might these differencesinfluence priority setting and resource allocation choices?It was noted that there have been a number of Canadianexperiences with formal approaches to priority setting,such as PBMA (Program Budgeting and Marginal Analy-sis), and that there might be lessons drawn from a meth-Interdisciplinary research can be defined as "any study orgroup of studies undertaken by scholars from two or moredistinct scientific disciplines. The research is based upon aconceptual model that links or integrates theoreticalframeworks from those disciplines, uses study design andmethodology that is not limited to any one field, andrequires the use of perspectives and skills of the involveddisciplines throughout multiple phases of the researchprocess" [17]. Multidisciplinarity, by contrast, involvesresearchers who work on a project together, but moreindependently and with less crossing of disciplinaryboundaries [18]. The feeling of many forum participantswas that the best avenues for future research would builda knowledge base, employing concepts and drawing onliteratures beyond health economics. These might includehealthcare ethics, organizational psychology, and the pol-icy sciences. Quantitative methods will have a role, butthere was also a recognized place for research in the estab-lished qualitative traditions such as narrative inquiry, eth-nography and discourse analysis.Future research theme areasSeveral theme areas were also identified for futureresearch on health care priority setting--in other words,specific ideas, topics or projects that might be pursued.These can be grouped under two broad headings,acknowledging that there will be some degree of overlapbetween these categories (See Table 1). First, futureresearch might usefully consider how formal priority set-ting and resource allocation projects are situated within alarger organizational and political context and how result-ing decisions have an impact on different parts of the con-tinuum of care and/or different organizations entirely.Second, we might advantageously devote additionalresearch efforts to better understanding and improvingthe actual implementation of priority setting frameworks.Decisions in broader contextForum participants noted that wherever formal prioritysetting processes have so far been implemented in theirorganizations, it has inevitably been within a larger con-text in which many other decision system initiatives andcycles occur simultaneously. These include strategic plan-ning (II:3), quality improvement (I:10; II:10), capitalplanning (III:7), the annual budget cycle (III:3), and issu-ing of provincial Ministry of Health directives (II:8). Thereis value to decision-makers in knowing where they sit inrelation to the larger network or web of decision proc-esses. Modeling or mapping these connections might beboth interesting and valuable (II:7). Case study and com-parative research might consider how priority settingactivities 'align' with other organizational activities.Page 4 of 9(page number not for citation purposes)odologically rigourous synthesis of these experiences. There are also potential 'ripple' effects that might be inves-tigated--for instance, 'how do or could priority settingBMC Health Services Research 2009, 9:165 http://www.biomedcentral.com/1472-6963/9/165exercises link to, identify, or catalyze knowledge of wherelarger systemic change might be needed within healthauthorities' (I:7)?-- how might 'local solutions ... becomeinstitutional ones' (II:6)? Also, 'a research question mightbe, how do executive level decisions and values translateto the everyday decisions people are making on the frontlines, at the service delivery level' (II:11)?There was some doubt among participants as to whetheror not the full set of outcomes and costs for prioritizationis ever captured. As one example, it was noted that while'the literature calls regularly for reductions in acute carespending to support community based care' (III:5), wemuch less frequently hear about efforts to do this, andeven less frequently are informed about the outcomes andwhether or not any savings or improved outcomes areactually realized. For instance, 'decreasing bed use amonga certain population might be seen as a success and a sav-ing in isolation, but it may simply allow the beds to benever appear' (III:5). A cut in one place might impose anew burden or cost elsewhere in the organization. Theseinter-related effects often go unrecognized: 'people don'tget what it costs the system' (III:7). For instance, it wasperceived that many clinicians do not think about whattheir service costs the system when they provide care. Sohow might priority setting processes ensure that these sys-tem-wide impacts are identified and incorporated (III:6)?As one participant stated, "You may not know until 5-10years down the road if the choices that seemed equivalenttoday really are so". (II:4) Such an observation argues forlongitudinal follow-up and evaluation studies.Finally, there was interest from a number of participantsin determining the relevant strengths and weaknesses ofacross the board versus incremental approaches to theintroduction of new formal priority setting processes. 'Isthere enough of a body of evidence as to whether 'bigbang' implementation or the accumulation of micro-trialsTable 1: Overview of research theme areasTheme Areas for possible investigationPriority setting decisions in a broader context How can formal priority setting processes best align with and complement other decision making processes?What consequential and reactive impacts result in the implementation of priority setting approaches?Can we improve the measurement of costs and benefits to account for the full range of organizational impacts?Can we assess the relative merits of implementing formal priority setting as a small scale pilot or as an organization wide mandate?Priority setting implementation Report on typical criteria used in formal priority setting studies and guidance about how to draft locally relevant measures for assessing spending optionsUnderstand the different ways in which decision-makers understand and apply the concept of disinvestmentExplore the rhetorical and tactical choices made in 'bundling' spending options and how these affect the results of formal priority settingIdentify the personal, social and organizational dimensions of how decision-makers manage conflicting role loyalties in priority settingImprove the quality and accessibility of relevant dataIdentify the skills and capacities needed for effectively using formal priority setting methods, as well as the related education and training requirementsProvide guidance about why, when and how to engage the public in priority setting and resource allocation decisionsPage 5 of 9(page number not for citation purposes)occupied by a different group so that the supposed savingon which re-investment choices are predicated mightis more effective at changing organizational decision mak-ing systems and patterns' (III:9)? What is a good 'entryBMC Health Services Research 2009, 9:165 http://www.biomedcentral.com/1472-6963/9/165point' for introducing an explicit priority setting frame-work in an organization (I:9)? Again, comparativeresearch might profitably examine this in some detail.Priority setting implementationChoosing among options requires decision criteria withadjudicative power; that is, they can distinguish amongdifferent funding proposals on the basis of the featureswhich decision-makers value. Participants identified anumber of issues faced and where better criteria would bebeneficial. In particular there were questions about howto give due consideration to strategic investments (I:5),performance agreements (II:7), long-term or transforma-tive changes (II:4), and non-health outcomes (I:11). Onesuggestion was to study criteria already used in formal pri-ority setting processes with a sort of 'sensitivity analysis'(I:6)--that is, to test how different groups within theorganization (e.g., program managers versus senior execu-tive members) understood and weighted different levelsof a given rating scale employed in determining overallbenefit of specific proposals. Some respondents won-dered if there was any prospect of developing a 'criteriadictionary' (I:10) of measures that have been used for pri-ority setting in other contexts and either formally vali-dated or otherwise found useful.Certain priority setting approaches, such as PBMA, neces-sarily link questions about investment and disinvestmentof services; however, participants wondered what wasknown about the way in which decision-makers under-stand and interpret these two ideas. Do views about disin-vestment as a concept differ from thinking about newinvestment (III:5)? While "everyone is there [when there'sa chance] to spend money" (II:9) comparatively littlethought seems to go into plans for reducing or eliminat-ing spending. Often the participants propose a 'hand-gre-nade' option (II:9)--blowing up a service in a way that is,and is known by all to be, unrealistic (such as stopping allsurgeries for the balance of a fiscal year). In many cases,program managers want the ability to pair investment anddisinvestment proposals - 'if we cut X, we can do Y' (III:3)--and so retain any freed-up resources within their ownprogram. A disinvestment is seen as 'taking somethingaway' from someone--perhaps community groups orother interests acquire a 'sense of entitlement' to existingservices or programs. This issue might be circumventedwhen the question is whether or not to invest in new ini-tiatives which do not (or may not) have an establishedand organized constituency (III:5). The problem may beexacerbated if disinvestment and new investment are notdirected to the same target populations. Many participantswondered about the importance of terminology; forinstance, would a process be more easily accepted if a termOne forum participant noted that "how one bundlestrade-offs is quite fascinating" (II:5). The way in whichoptions are framed or linked can most definitely affect theway in which choices are justified. These effects are mosteasily seen where the resource allocation decisionsinvolve global rather than earmarked or targeted sourcesof money. Research with those doing health care prioritysetting should be focused on helping to understand theextent to which these positions may be pre-planned. Alter-natively, positions evolve during the course of negotia-tions as a response to the understood political influenceand power of the different players in the priority setting'game'. When research into priority setting processes looksonly at what happens around the 'decision table' it maymiss important questions about how different options arereached by individual departments or portfolio managersand how the choices are 'filtered' prior to that point. Onehealth authority representative noted that in their experi-ence, each portfolio was limited to three submissions--butit was unclear what processes narrowed the field down tothese final options (I:9). In other words, 'how do we comeup with the areas considered for investment or disinvest-ment--out of the whole range of things that might be pos-sible candidates' (III:8)? Are the right options on the tableat all? Research efforts to explore these questions wouldrequire deeper and more extensive engagement with deci-sion-makers at all levels of an organization.Some participants noted that those setting priorities hadto manage conflicting role loyalties. 'How could you besitting at the table and not get anything for us?' is some-thing that those who are involved in organization-widepriority setting hear from their colleagues when theyreturn to their home departments (I:9). Executive mem-bers themselves speak to these dilemmas: as one stated, "Iknow I need to defend my program, but how do I fit intothe organization?" (II:6). There are also issues for clini-cians who must balance roles as patient advocates withfunctions as gatekeepers to the system and its resources(II:1). Particularly in smaller sectors or communities, pri-ority setting choices cannot simply be seen in the abstract--decision-makers know that their selections will impactidentifiable individuals, even friends or neighbours (II:8).'It's not just a position but a name' (III:6). There seems tobe considerable scope to study individual stories abouthow decision-makers respond to these pressures andexpectations, and strike a balance with their understand-ing of supposed organization-wide needs.In the eyes of the decision-makers participating in theforums, effective priority setting and resource allocation isoften held back by a lack of good, appropriate, applicable,reliable and valid data (III:6). There seemed to be a gen-Page 6 of 9(page number not for citation purposes)like 'resource re-allocation' was substituted for 'disinvest-ment' (I:5)?eral consensus that more effort in local data collection (aswell as improved local access to data collected for provin-BMC Health Services Research 2009, 9:165 http://www.biomedcentral.com/1472-6963/9/165cial performance reporting purposes) would produce asolid return on investment (III:7).Taking a step back, participants recognized that healthcare managers have not frequently engaged in use of a for-mal, explicit framework for priority setting and resourceallocation. Forum participants wondered if members oftheir own organizations had the skills and capacitiesneeded for this work (I:9)--and wondered if researchcould help to systematically identify what these skills andcapacities might be. Investigation of the sorts of organiza-tional structures and supports that would best assist staffin these efforts was felt to be equally important (I:9; I:11).Health authorities are being asked to make importantresource allocation decisions, and so one might wish toknow if their current organizational forms are "fit for pur-pose" (III:9).Finally, health system managers are under pressure frommany directions to increase the extent of public engage-ment in decision making. This was clearly on the minds ofparticipants who raised various questions around obtain-ing public input such as, 'what is its purpose?' and, 'howshould it be used?' (I:6) i.e., how might the public's viewsand values be integrated with other forms of evidence?Public engagement efforts can be time-consuming andcostly, so decision-makers asked whether evidence basedguidance can be developed and suggest when it is worth-while or necessary. Is there a way to prioritize the deci-sions that are subject to formal public participation efforts(II:3, II:11)? To what extent can or should the public beengaged as questions get to be more narrow and 'techni-cal' in nature (I:6; II:11)? One purpose of public engage-ment may be to test whether or not decision-makerassumptions about what the public wants are in fact accu-rate and valid (II:10).DiscussionIn recent years there has been a good deal of research intohealth care priority setting [19-27]. Some findings areclear. Priority setting is more than a technical exercise; itneeds to be understood as a management process [28].Economic approaches to priority setting should incorpo-rate ethical principles and vice versa [16,29,30]. Bothresearchers and decision-makers need to think broadlyabout what constitutes appropriate and relevant evidence[31,32]. The literature suggests that there are several gapsin knowledge, such as the evaluation of priority settingframeworks relevant to health outcomes and otherinterim outputs (e.g., does formal priority setting helpwith setting useable priorities; are these priorities used tomake decisions; does a formal framework increase the useof evidence?) and in terms of appropriate ways to engageGiven this work and the myriad of challenges faced bydecision-makers, we believe that future research in thisarea should be highly collaborative. The exact questionsmust be shaped as much by decision-maker input asresearcher interest. The current project aimed to develop aset of research priorities in the field of health care prioritysetting. While it is not our intent to suggest that the prior-ities identified are all encompassing, nor necessarily applyin all jurisdictions, our experience in this field both withinand outside of Canada would suggest that the issues thatwere raised should be relevant for most jurisdictions facedwith allocating a finite set of resources.The forum process resulted in a research agenda with thefollowing characteristics: (1) The province-wide nature ofthe project allowed us to identify a comprehensive rangeof issues, including those most relevant for organizationsat various stages of development in formal priority settingand resource allocation work, while recognizing differentgeographies, populations and health needs, organiza-tional structures, service mixes, financial positions, etc;(2) There is considerable potential and desire for compar-ative work, which would allow health regions to shareexperiences and avoid 'reinventing the wheel'; (3) Wehave identified priorities that already have decision-maker buy-in, so it should be somewhat simpler andquicker to promote subsequent dissemination and uptakein British Columbia.Several other general desirable outcomes were obtainedfrom the very process of engaging in these forums. Tobegin with, they demonstrate one successful way of bridg-ing academic and practice worlds. Principles identified inthe knowledge transfer and exchange literature--such astwo-way interaction among decision-makers andresearchers working together to evolve priority settingpractice-were demonstrated [2-4]. We speculate that thiswill result over the longer-term in a vibrant and growingnetwork of BC researchers focused on priority setting, aswell as a more common understanding of formal prioritysetting approaches and application of key principles at thehealth authority level. Such embedded knowledge shouldthen contribute to improvements in routine priority set-ting practice.The process allowed for sharing and networking amongthe health authorities themselves. Health authority per-sonnel who attended were informed during the course ofthe forums about current priority setting research andpractice in BC - their attendance increased their awarenessof frameworks in use and likely promoted more in-depthcontemplation regarding priority setting in the province.This form of research, involving active and reflectivePage 7 of 9(page number not for citation purposes)the community in resource allocation decisions engagement in priority setting exercises, is also conduciveto organizational learning and creation of greater aware-BMC Health Services Research 2009, 9:165 http://www.biomedcentral.com/1472-6963/9/165ness and understanding among decision-makers abouthow choices affect the organization as a whole.Our observations suggest that the challenges, ideas andresearch topics would not have arisen - at least in the formthey did - without direct interaction between researchersand decision-makers. Even as applied health servicesresearchers, working closely with decision-makers inhealth service organizations, we could not predict, norwould we presume to know, the intricacies of priority set-ting at the coal face. We also observed a high level of peerto peer interaction between decision-makers from differ-ent health authorities. This was mentioned in the partici-pant evaluations as a valuable aspect of the project.Furthermore, the expenditure on this team planning exer-cise was relatively modest, amounting to $37,500 total.Would we do anything differently? We offer two sugges-tions.• First, asking decision-makers to give up three half dayswithin 9 months was in hindsight asking too much. If wewere to do this again, we would have one day-long work-shop geared towards a facilitated group discussion tomaximize peer-to-peer and researcher-to-decision-makerinteraction. If more then 20 decision-makers were inter-ested in attending, we would hold separate workshops butensure that all health authorities were represented at eachsession.• Second, in follow-up, we would suggest having a moreformal process in place to engage decision-makers withtransition from an idea to an actual research question and,in due course, to a full research proposal. With the BCgeography this is perhaps difficult, but nonetheless, allo-cating the budget to ensure one-on-one meetings witheach of the health authorities following a primary work-shop would in our view result in greater likelihood ofongoing research collaboration.A few other potential limitations are worth mentioning.Were the 'right' people involved in the forums? Movingresearch forward effectively in practice settings requires agood balance between people who can speak to the tech-nical issues 'at the coal face' of priority setting andresource allocation - those who know the challengingissues and dilemmas firsthand -- and those who are seniorleaders able to devote resources to research. In this regard,our participants represented a good balance. In terms ofprevious experience, some participants had much directpriority setting experience to cite, others had relatively lessor none. Most participants seemed actively engaged,though not all and not consistently. The fact that manyespecially given the many competing demands on deci-sion-makers' time. A conscious choice was made not toinclude personnel from the Provincial Ministry of Health;the focus was squarely placed on regional decision mak-ing.We did not seek consensus, where all participants neces-sarily agreed upon particular research priorities shared inall health regions. Rather, the directions reported herereflect the whole range of topics raised; some persons andregions may be more interested in some of these than oth-ers. We have not reported here every possible researchquestion that was raised during forum discussions;instead we have tried to group them into broad themeareas, with the attendant risk of omitting details whichmight be potentially very important to individual partici-pants. Finally, the health authorities in BC have differentpopulations and geographies. We wanted to be sure thatthe final agenda reflects this range of interests. Thus, wemust ask whether or not some participants dominated thediscussion and the outcomes. In careful review of theforum notes, most individuals did contribute; less vocalparticipants were called out for their perspective. How-ever, it is realistic to suggest that some participants had adeeper understanding and greater engagement with thematters being discussed and their specific views may haverisen to the top more readily.Participants had several opportunities to validate the find-ings of this research; thus we are confident that the prior-ities represent their immediate needs and interests. Thedepth of that interest, however, will be shown by whetheror not successful research collaborations are subsequentlypursued. Of course, research proposals will necessarily befitted to or constrained by available funding streams.Finally, there is frequent turnover among decision makersin the health care sector (already including some of ourpartners in this project). If the priorities we have identifiedare truly those of the health delivery organizations, ratherthan of the particular participants, then they should sur-vive such developments intact. Again, time will tell.ConclusionThese forums have given us insight into what decision-makers see as important, and have uncovered numerousareas where, jointly, research questions can be posed. Wewere able to validate the importance of our starting pointas well as to observe the emergence of additional concernsand directions of critical importance to these decision-makers at this time. While some, indeed probably most,of the research priorities are likely relevant elsewhere, wewould advocate for others in different contexts to under-take their own research priority generating exercise. Pro-Page 8 of 9(page number not for citation purposes)participants returned for subsequent forums speaks totheir engagement and sustained interest in this work,viding an environment where researchers and decision-makers can interact, debate and collaboratively generate aBMC Health Services Research 2009, 9:165 http://www.biomedcentral.com/1472-6963/9/165set of research directions should be seen as a positive steptowards the goal of a more efficient and sustainable healthcare system.Competing interestsThe authors declare that they have no competing interests.Authors' contributionsCM, SP and SM were project Investigators and conceivedand prepared the successful funding application. Allauthors attended the three Team Planning Forums. NSand EC conducted the data analysis. NS prepared the ini-tial manuscript draft. All authors reviewed, commentedupon and approved the manuscript.AcknowledgementsThe research reported here was supported by a Michael Smith Foundation for Health Research Team Planning Grant. Both Craig Mitton and Stuart Peacock are Michael Smith Foundation for Health Research Scholars. Eve-lyn Cornelissen is funded by the Western Regional Training Centre (a joint funding initiative of CIHR, CHSRF, and AHRMR). The authors would like to acknowledge the contributions of Francois Dionne, Cam Donaldson, Jenni-fer Gibson, Steven Lewis, Bonnie McCoy, Zahra Musa, Pawlos Teckle, Kim van der Hoek and the other Team Planning forum participants.References1. Lomas J, Fulop N, Gagnon D, Allen P: On being a good listener:setting priorities for applied health services research.  MilbankQ 2003, 81:363-388.2. Elliott H, Popay J: How are policy makers using evidence? Mod-els of research utilisation and local NHS policy making.  J Epi-demiol Commun H 2000, 54:461-468.3. Lavis J, Robertson D, Woodside JM, et al.: How can researchorganizations more effectively transfer research knowledgeto decision makers?  Milbank Q 2003, 81:221-248.4. Lomas J: Using linkage and exchange to move research intopolicy at a Canadian foundation.  Health Affair 2000, 19:236-240.5. Vingilis E, Hartford K, Schrecker T, Mitchell B, Lent B, Bishop J: Inte-grating knowledge generation with knowledge diffusion andutilization.  Can J Public Health 2003, 94:468-471.6. Dresser R: Public advocacy and allocation of federal funds forbiomedical research.  Milbank Q 1999, 77:257-274.7. Rosenstock L, Olenec C, Wagner G: The National OccupationalResearch Agenda: a model of broad stakeholder input intopriority setting.  Am J Public Health 1998, 88:353-356.8. Fenske RA, Hidy A, Morris SL, Harrington MJ, Keifer MC: Healthand safety hazards in Northwest agriculture: setting an occu-pational research agenda.  Am J Ind Med 2002, 42(Supp):62-67.9. O'Fallon LR, Wolfle GM, Brown D, Dearry A, Olden K: Strategiesfor setting a national research agenda that is responsive tocommunity needs.  Environ Health Persp 2003, 111:1855-1860.10. Black N: A national strategy for research and development:lessons from England.  Annu Rev Publ Health 1997, 18:485-505.11. Shergold M, Grant J: Freedom and need: the evolution of publicstrategy for biomedical and health research in England.Health Res Policy Syst 2008, 6:2.12. Smith E, Ross FM, Mackenzie A, Masterson A: Developing a serv-ice-user framework to shape priorities for nursing and mid-wifery research.  J Res Nurs 2005, 10:107-118.13. Dault M, Lomas J, Barer M, on behalf of the Listening for Direction IIpartners: Listening for Direction II: National Consultation onHealth Services and Policy Issues for 2004-2007.  2004 [http://www.chsrf.ca/other_documents/listening/pdf/LfD_II_Final_Report_e.pdf]. Ottawa, Ontario, Canadian Health Serv-ices Research Foundationwww.chsrf.ca/other_documents/listening/documents/LfDIII-FINAL_ENG.pdf]. Ottawa, Ontario, Canadian Health ServicesResearch Foundation15. Mitton C, Donaldson C: Priority Setting Toolkit: A Guide to the Use of Eco-nomics in Healthcare Decision Making BMJ Books, London; 2004. 16. Peacock S, Ruta D, Mitton C, Donaldson C, Bate A, Murtagh M:Using economics to set pragmatic and ethical priorities.  BMJ2006, 332:482-485.17. Aboelela SW, Larson E, Bakken S, Carrasquillo O, Formicola A, GliedSA, Haas J, Gennie KM: Defining interdisciplinary research:Conclusions from a critical review of the literature.  HealthServ Res 2007, 42:329-346.18. Nair KM, Dolovich L, Brazil K, Raina P: It's all about relationships:A qualitative study of health researchers' perspectives ofconducting interdisciplinary health research.  BMC Health ServRes 2008, 8:110.19. Donaldson C: Economics, public health and health care pur-chasing: reinventing the wheel?  Health Policy 1995, 33:79-90.20. Ham C: Priority setting in health care: learning from interna-tional experience.  Health Policy 1997, 42:49-66.21. Gibson JL, Martin DK, Singer PA: Setting priorities in health careorganizations: criteria, processes, and parameters of suc-cess.  BMC Health Serv Res 2004, 4:25.22. Bruni RA, Laupacis A, Martin DK: Public engagement in settingpriorities in health care.  CMAJ 2008, 179:15-18.23. Mitton C, Patten S, Waldner H, Donaldson C: Priority setting inhealth authorities: a novel approach to a historical activity.Soc Sci Med 2003, 57:1653-1663.24. Makundi E, Kapiriri L, Norheim OF: Combining evidence and val-ues in priority setting: testing the balance sheet method in alow-income country.  BMC Health Serv Res 2007, 7:152.25. Peacock SJ, Richardson JR, Carter R, Edwards D: Priority setting inhealth care using multi-attribute utility theory and pro-gramme budgeting and marginal analysis (PBMA).  Soc SciMed 2007, 64:897-910.26. Segal L, Mortimer D: A population-based model for priority set-ting across the care continuum and across modalities.  CostEff Resour Alloc 2006, 4:6. doi:10.1186/1478-7547-4-627. Wilson ECF, Peacock SJ, Ruta D: Priority setting in practice:what is the best way to compare costs and benefits?  HealthEcon 2008, 18(4):467-78.28. Donaldson C, Bate A, Mitton C, Peacock S, Ruta D: Priority settingin the public sector: Turning economics into a managementprocess.  In Managing Improvement in Public Service Delivery: Progressand Challenges Edited by: Hartley J, et al. London, Cambridge Univer-sity Press; 2007. 29. Gibson J, Martin D, Singer P: Evidence, economics and ethics:resource allocation in health services organizations.  Health-care Q 2005, 8:50-59.30. Gibson J, Mitton C, Martin D, Donaldson C, Singer P: Ethics andeconomics: does programme budgeting and marginal analy-sis contribute to fair priority setting?  J Health Serv Res Policy2006, 11:32-37.31. Armstrong K, Mitton C, Carleton B, Shoveller J: Drug formularydecision-making in two regional health authorities in BritishColumbia, Canada.  Health Policy 2008, 88:308-316.32. Ruta DA, Donaldson C, Gilray I: Economics, public health andhealth care purchasing: the Tayside experience of pro-gramme budgeting and marginal analysis.  J Health Serv Res Pol-icy 1996, 1:185-193.Pre-publication historyThe pre-publication history for this paper can be accessedhere:http://www.biomedcentral.com/1472-6963/9/165/prepubPage 9 of 9(page number not for citation purposes)14. Law S, Flood C, Gagnon D, on behalf of the Listening for Direction IIIpartners: Listening for Direction III: National Consultation on Health Serv-ices and Policy Issues, 2007-2010, Final Report 2008 [http://


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items