UBC Faculty Research and Publications

Indicators and measurement tools for health system integration: a knowledge synthesis protocol Oelke, Nelly D; Suter, Esther; da Silva Lima, Maria A D; Van Vliet-Brown, Cheryl Jul 29, 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-13643_2015_Article_90.pdf [ 436.07kB ]
JSON: 52383-1.0221455.json
JSON-LD: 52383-1.0221455-ld.json
RDF/XML (Pretty): 52383-1.0221455-rdf.xml
RDF/JSON: 52383-1.0221455-rdf.json
Turtle: 52383-1.0221455-turtle.txt
N-Triples: 52383-1.0221455-rdf-ntriples.txt
Original Record: 52383-1.0221455-source.json
Full Text

Full Text

PROTOCOLe:imtegration. Theh system inte-ons given theespite the lackidentified thatOelke et al. Systematic Reviews  (2015) 4:99 DOI 10.1186/s13643-015-0090-7(3) geographic coverage and rostering, (4) standardized caredelivery through interprofessional teams, (5) performanceWay, Kelowna, British Columbia V1V 1V7, CanadaFull list of author information is available at the end of the articlecollectively support health system integration. The princi-ples include (1) comprehensive services, (2) patient focus,* Correspondence: nelly.oelke@ubc.ca1School of Nursing, University of British Columbia, Okanagan 3333 Universitycare reform [1]. Given our aging population and higherrates of chronic disease, there has been a shift fromacute, episodic care to a greater focus on integrated careacross the continuum [1, 2]. However, delivery of healthcare services continues to be very fragmented [3]. Two[4] focused on models for health system inresults showed there was no definitive healtgration model appropriate for all organizaticomplexity of health care service delivery. Dof a definitive model, 10 key principles wereBackgroundIntegration in health care is a key component of healthof the authors (NO/ES) previously conducted a CanadianInstitute of Health Research-funded knowledge synthesisAbstractBackground: Health system integration is a key component of health system reform with the goal of improvingoutcomes for patients, providers, and the health system. Although health systems continue to strive for betterintegration, current delivery of health services continues to be fragmented. A key gap in the literature is the lackof information on what successful integration looks like and how to measure achievement towards an integratedsystem. This multi-site study protocol builds on a prior knowledge synthesis completed by two of the primaryinvestigators which identified 10 key principles that collectively support health system integration. The aim isto answer two research questions: What are appropriate indicators for each of the 10 key integration principlesdeveloped in our previous knowledge synthesis and what measurement tools are used to measure theseindicators? To enhance generalizability of the findings, a partnership between Canada and Brazil was createdas health system integration is a priority in both countries and they share similar contexts.Methods/design: This knowledge synthesis will follow an iterative scoping review process with emerging informationfrom knowledge-user engagement leading to the refinement of research questions and study selection. This paperdescribes the methods for each phase of the study. Research questions were developed with stakeholder input. Indicatoridentification and prioritization will utilize a modified Delphi method and patient/user focus groups. Based on priorityindicators, a search of the literature will be completed and studies screened for inclusion. Quality appraisal of relevantstudies will be completed prior to data extraction. Results will be used to develop recommendations and key messagesto be presented through integrated and end-of-grant knowledge translation strategies with researchers andknowledge-users from the three jurisdictions.Discussion: This project will directly benefit policy and decision-makers by providing an easy accessible set ofindicators and tools to measure health system integration across different contexts and cultures. Being ableto evaluate the success of integration strategies and initiatives will lead to better health system design andimproved health outcomes for patients.Keywords: Delphi, Focus groups, Health systems, Integration, Knowledge synthesis, Systematic reviewIndicators and measuremhealth system integrationsynthesis protocolNelly D. Oelke1*, Esther Suter2, Maria Alice Dias da Silva L© 2015 Oelke et al. This is an Open Access art(http://creativecommons.org/licenses/by/4.0),provided the original work is properly creditedcreativecommons.org/publicdomain/zero/1.0/Open Accessnt tools fora knowledgea3 and Cheryl Van Vliet-Brown1icle distributed under the terms of the Creative Commons Attribution Licensewhich permits unrestricted use, distribution, and reproduction in any medium,. The Creative Commons Public Domain Dedication waiver (http://) applies to the data made available in this article, unless otherwise stated.Oelke et al. Systematic Reviews  (2015) 4:99 Page 2 of 8management, (6) information technology, (7) organizationalculture and leadership, (8) physician integration, (9) govern-ance structure, and (10) financial management [5].Integration initiatives are being implemented at alllevels of the health care system [1]. Health system inte-gration has enduring relevance provincially, nationally,and internationally. Many jurisdictions continue to grap-ple with the development of integration strategies andhow to measure integration. For example, in BritishColumbia (BC), the 10 key principles have been used bythe BC Ministry of Health, Interior Health and theMichael Smith Foundation for Health Research to guidehealth care reform and evaluation of integrated commu-nity health services. In Porto Alegre, Brazil, a workinggroup was established to create networks to increase pri-mary health care integration [6]. Internationally, theWorld Health Organization (WHO) hosted a technicalmeeting, “WHO Strategy on People-Centered and Inte-grated Health Services” to discuss and review indicesand measures [7].Health systems are consistently striving to deliver inte-grated health services as integrated models have the po-tential to positively impact patient, provider, and systemoutcomes. More specifically, integration has the poten-tial to improve quality of care [1, 8] and decreaseutilization of resources [2, 8]. Despite health care organi-zations’ efforts to achieve integration, there is little infor-mation on what successful integration looks like andhow to measure achievement towards an integrated sys-tem [9]. An important gap that emerged in the priorknowledge synthesis was the lack of indicators and toolsto measure integration. Some organizations have devel-oped balanced scorecards around particular integrationcomponents [10]. Others have used the Clinical Micro-system Assessment Tool [11], which allows a snapshotof where an organization lies along a continuum of inte-gration. Strandberg-Larsen and Krasnik [1] argue that“methods to measure integrated health care delivery areclearly emerging” (p. 4); however, the few tools that existare not easy to find as literature on integration is dis-persed. Likewise, it is unclear if the tools that cur-rently exist cover all 10 principles of integration asidentified in our previous knowledge synthesis andthere is no inventory of indicators and tools that canbe accessed by decision-makers to develop evaluationand performance monitoring plans. This hampers ourcollective ability to monitor the effectiveness of inte-gration strategies and limits our ability to improvehealth system integration [12].There are five reasons why this knowledge synthesis issignificant and urgently needed: (1) Current systemscontinue to be fragmented. (2) Many health systems in-clude goals focusing on integrated health care. (3) Mosthealth systems are implementing integration initiativesdesigned to improve quality of care and efficiency whilereducing costs and resource utilization. (4) There is alack of understanding by health systems of their achieve-ment towards integrated health care. (5) There is a lackof tools available to measure health system integration.This knowledge synthesis will address gaps in themeasurement of integration in health systems. Our re-search questions include the following: (1) What are ap-propriate indicators for each of the 10 key integrationprinciples developed in our previous knowledge synthe-sis? And (2) what measurement tools are used to meas-ure these indicators? This project will directly benefitpolicy and decision-makers by providing an easily ac-cessible set of indicators and tools to measure healthsystem integration across different contexts and cultures.Being able to evaluate the success of integration strat-egies and initiatives will lead to better health system de-sign and improved health outcomes for patients. Toenhance the global applicability of the proposed work,we have developed a partnership between Brazil andCanada. This partnership was specifically chosen as bothcountries have publicly funded health systems, compar-able funding priorities, and similar geography of largeurban centres and rural communities. Furthermore,health system integration is a priority in both countries.Methods/designThis knowledge synthesis will follow processes for scop-ing reviews recommended by Levac et al. [13]. Theystress the need for an iterative process rather than alinear process with emerging information leading torefinement of research questions and study selection.The Levac et al. [13] framework was selected based on itsapplicability and relevance given the policy context ofthe questions, the need for an iterative process, andknowledge-user engagement. This methodological frame-work outlines six stages for rigorous scoping reviews: (1)identifying the research question; (2) identifying relevantstudies; (3) selecting studies; (4) charting the data; (5)collating, summarizing, and reporting results; and (6) con-sulting [13]. For this knowledge synthesis, an additionalstage termed identifying indicators was added followingidentifying the research question. As this review does notaddress the effects of interventions and/or strategies toprevent, diagnose, treat, and/or monitor health conditions,for which there is a health-related outcome, it is ineligiblefor PROSPERO registration.Identifying the research questionIdentifying the research question requires considerationof the scope of inquiry and the purpose of the review[13]. The scope of inquiry for this review is to identifyand validate relevant integration indicators associatedwith the 10 key principles and measurement tools toOelke et al. Systematic Reviews  (2015) 4:99 Page 3 of 8monitor progress towards integrated health systems. Thepurpose is to enable evaluation of the success of integra-tion strategies and initiatives, which will lead to betterhealth system design and improved health outcomes forpatients. Knowledge-users were consulted on the issuesof integrated health systems and the focus of our know-ledge synthesis. Research questions were developed andshared with our knowledge-users who confirmed the im-portance of these questions. The research questions tobe addressed by the scoping review include the follow-ing: What are appropriate indicators for each of the 10key integration principles and what measurement toolsare used to measure these indicators?Identifying indicatorsAs part of the knowledge synthesis and prior to con-ducting our systematic review on existing tools to meas-ure health system integration, a better understanding ofthe indicators for successful integration is needed. Twoapproaches to prioritize and identify indicators will beutilized. A Delphi survey with key stakeholders will beused to identify appropriate indicators and priorities forthe same. The second approach, focus groups withpatients or users of the health system, will determinepatient and user prioritization of the principles the indi-cators measure. Drawing on previous work and a scan ofthe literature, research team members generated a pre-liminary list of possible indicators for each of the 10 keyprinciples (see Table 1 for sample indicators). Researchteam members including researchers and knowledge-users confirmed the preliminary list of indicators.Delphi surveyA modified Delphi method will be used to obtain con-sensus from a panel of integration experts, policy anddecision-makers, and providers on the most relevant in-dicators for each key integration principle. This ap-proach allows expert perspectives and judgments to becollected without the need for face-to-face or virtualmeetings, thus reducing costs and logistical details [14].It also reduces socially desirable responding as ratingsare anonymous to the rest of the panel [15]. Panel mem-bers include individuals from Canada and Brazil, as wellas international experts on the topic area. The surveywill be translated into Portuguese to enable key stake-holders from Brazil to participate.For each round of the Delphi process, the panel mem-bers (n = 30–35) will receive a survey with a list of theprinciples and indicators by email. They will be asked torate the fit and importance of each indicator to its re-spective key principle using a 5-point Likert-type scale.Within each of the key principles, participants will alsobe asked to prioritize the indicators. Results will be com-piled and used to inform subsequent rounds of thesurvey [16, 17] until consensus is attained. The deter-mination of consensus is open to interpretation; how-ever, it generally falls between 70 and 80 % [15]. In thisstudy, a consensus level of 75 % agreement has been se-lected. Indicators will be deleted when they are consid-ered not relevant demonstrated by 75 % of participantsrating at 4 or 5 on appropriateness and importance andrated lower than 3 or 4 for priority. An indicator will beaccepted once 75 % consensus is obtained on all threeratings.Round 1 Panel members will receive the survey with acomplete list of preliminary indicators. They will beasked to add any relevant missing indicators. Results willbe compiled and indicators suggested by the panel mem-bers added. We anticipate there may be a substantialnumber of additional indicators suggested in round 1.First, these will be themed by a research assistant andreviewed by the primary investigator (PI) coordinatingthe study. Those indicators that are essentially thesame will be combined. Second, the list of existing in-dicators, themed indicators, and other additional indi-cators will be circulated to the research team. All teammembers will be asked to rate the appropriateness,importance, and priority as per the Delphi surveyscale. They will also be asked to provide rationale fortheir ratings. These results will be compiled and ana-lysed. A sub-committee of the research team includingPIs from each jurisdiction and a co-investigator withexpertise in Delphi processes will discuss these resultsand decide on the indicators to be included in the sec-ond version of the Delphi survey.Round 2 The round 2 survey will be sent out to all par-ticipants who were invited to participate in the firstround except those who indicated they were not able toor did not wish to participate. In this round, there willbe no opportunity for the addition of new indicators.Participants will only be asked to rate the appropriate-ness, importance, and priority of each indicator. Thesurvey will then be revised to only include those indica-tors that are relevant and those where there is noagreement.Subsequent rounds The revised survey will be sent outto all participants invited as per rounds 1 and 2. Round3 and subsequent rounds will be analysed in the samemanner as round 2. We anticipate three to four roundswill be needed to achieve consensus among the panelmembers. Each of the indicators will be assessed indi-vidually for agreement. If there is 75 % agreement on fitand importance, they will be included in the final list. Ifthere is no 75 % agreement across Delphi participantsfor a specific indicator, they will not be included in theOelke et al. Systematic Reviews  (2015) 4:99 Page 4 of 8Table 1 Sample indicators for each key principlefinal list. We will aim to settle on two or three indicatorsper key principle. This final set of indicators will providethe foundation for the systematic literature review.Key principle Description of the principle1. Comprehensiveservices across the carecontinuum• Cooperation between health and social careorganizations• Access to care continuum with multiple points oaccess• Emphasis on wellness, health promotion, and prcare2. Patient focus • Patient-centred philosophy; focusing on patients• Patient engagement and participation• Population-based needs for assessment; focus ondefined population3. Geographic coverageand rostering• Maximize patient accessibility and minimize dupof services• Roster: responsibility for identified population; rigpatient to choose and exit4. Standardized caredelivery throughinterprofessional teams• Interprofessional teams across the continuum of• Provider-developed, evidence-based care guideliand protocols to enforce one standard of care, regof where patients are treated5. Performancemanagement• Committed to quality of services, evaluation, andcontinuous care• Diagnosis, treatment, and care interventions linkeclinical outcomes6. Information systems • State-of-the-art information systems to collect, trand report activities• Efficient information systems that enhance communand information flow across the continuum of care7. Organizational cultureand leadership• Organizational support with demonstration ofcommitment• Leaders with vision who are able to instil a stroncohesive culture8. Physician integration • Physicians are the gateway to integrated healthdelivery systems• Pivotal in the creation and maintenance of asingle-point-of-entry or universal electronic patienrecord• Engage physicians in leading role, participation oBoard to promote buy-in9. Governance structure • Strong, focused, diverse governance representedcomprehensive membership from all stakeholder• Organizational structure that promotes coordinaacross settings and levels of care10. Financialmanagement• Aligning service funding to ensure equitable fundistribution for different services or levels of servic• Funding mechanisms must promote interprofessteamwork and health promotion• Sufficient funding to ensure adequate resourcessustainable change11. Overall integrationFocus groupsOne patient or user focus group consisting of four toeight individuals will be held in each of the regions (BCSample indicatorsfimary• Coordinated transitions in care across services [23]• Shared programs across sectors/services [24]• Third next available appointment [25]• Emergency department average LOS registration to discharge;registration to admission (QPSD 23) [26]• Measure wait time for referral to treatment by providertype (QPSD 20) [26]• Proportion of patients with health outcomes which areavoidable given the current state of medical knowledgeand access to appropriate care [27]• Tobacco screening [28]’ needs • Involvement in care planning for chronic disease/complexcare [29]• Evidence of a population-based needs assessment [30, 31]licationht of• Existence of primary care network structures (e.g. familyhealth teams, primary care networks, GP Divisions, inner cityprimary health care clinics) [30]carenesardless• Team effectiveness [32]• Using a shared clinical pathway across care sectors (e.g.diabetes care, asthma care) [33]d to• Performance measurement indicators and tools are in placeand being used regularly [34]• Clinical outcomes being measured [35]ack,ication• Shared information systems across care sectors [36, 37]g,• Extent to which organizational goals and objectives arealigned across care sectors [36]caretn• Physician integration within care teams and across caresectors [10, 36, 38]• Practitioner payment models that support integration [37]by agroupstion• Existence of interagency agreements, service delivery teamcoalitions [39]• Governance model that includes representation ofcommunities served [30]• Evidence of governance in monitoring and evaluation ofhealth system [40]dingesionalfor• Extent to which financial management is coordinated acrosscare units and sectors [36]• Degree of integration within the health system and acrosssectors [41, 42]Oelke et al. Systematic Reviews  (2015) 4:99 Page 5 of 8and Alberta, Canada, and Rio Grande do Sul, Brazil) togain an understanding of patient perspectives on theprinciples of integration. This methodology was selectedas it allows for the expansion of knowledge through pur-poseful interaction of group members to generate con-textually grounded opinions and beliefs about a topic[18]. Each focus group will be facilitated by a member ofthe research team who is familiar with the 10 key princi-ples of integration using a set procedure and interviewguide. A second team member will take field notes andaudio record the session. Participants will be providedwith a list of the 10 key principles of integration andtheir descriptions. The handout will also have a columnwhere they will be asked to prioritize the principles atthe end of the session with #1 being a high priority and#10 being the lowest priority. Priority ratings for theprinciples will be analysed using descriptive statistics.Focus group recordings and field notes will be tran-scribed verbatim and coded using NVivo10™ software. Astandardized coding framework built on the principlesand indicators will be developed for use by all teamscoding the data. A research assistant/PI team will codeand conduct a thematic analysis of the data in Brazil,and another such team will analyse the two focus groupscompleted in Canada. They will then hold a virtualmeeting to discuss the similarities and differences inthemes. Key sections of data from Brazil will be trans-lated into English, and a final round of analysis of allfocus group data will then be conducted. These resultswill create an in-depth understanding of their perspec-tives on integration and which principles patients andusers see as important to an integrated health system.To our knowledge, there is little research on patients’perceptions on integration. This unique perspective willinfluence recommendations for which indicators shouldbe prioritized for implementation or further research.Identifying relevant studiesThe research librarian team member will assist with identi-fication of search terms together with a sub-committee ofthe research team including researchers, research librarian,and research assistants. The initial search will focus onsearch terms relating to each of the indicators and will bereduced by including terms relating to health system inte-gration and tools/tool development. Search terms for eachindicator will be reviewed by the sub-committee andsample searches conducted prior to the final searches beingexecuted by our librarian. Material about health system in-tegration and related indicators may also be found insources outside the traditional research literature. Thesearch strategy will encompass both the peer-reviewed andtargeted grey literature published from 1995 to 2015. Thesearch for relevant literature will include [19] Health Sci-ences, Education and Management/Business bibliographicdatabases (Medline including the Cochrane Library,EMBASE, PsycINFO, CINAHL, ABI Inform, and Busi-ness Source Premier), websites of relevant governmentagencies and research organizations (e.g. Institute forHealth care Improvement), scanning reference lists ofincluded studies, contacting key authors to identifyadditional papers focusing on measurement tools, Webof Science citation searching, and consulting with expertsto highlight key papers.Selecting studiesAbstracts will be downloaded into ProQuest RefWorks™bibliographic management software program, and dupli-cates will be removed. An initial set of inclusion and exclu-sion criteria have been developed and reviewed by theresearch team. Inclusion criteria include quantitative, quali-tative, and mixed-methods study designs, published in Eng-lish or Portuguese languages, and published within the last20 years (1995–2015) when integration in health systemsbecame a more common topic of discussion. Priority willbe given to randomized control trials and other quantitativestudies that specifically discuss the development or use of atool. In instances where there are no tools in the quantita-tive literature for an indicator, qualitative studies looking atresearch to support tool development will be included.Articles will be excluded if they were published prior to1995, are from non-health care settings, or are of a theoret-ical, editorial, or commentary nature. The same approachwill be used for the identification of other sources (e.g. web-sites) where reports, papers, and abstracts focused on meas-urement tools related to indicators will be downloaded intobibliographic software.All research team members involved in reviewing ab-stracts will pre-test the criteria using 20 randomly se-lected abstracts. This will allow establishment of inter-rater reliability. Criteria will be refined if needed, andpre-testing repeated until the mean inter-rater reliabilitykappa of all pairs is satisfactory (kappa >0.8). Subse-quently, pairs of raters from the team will independentlyuse the criteria to screen each abstract from the peer-reviewed literature for relevance. Disagreements will beresolved by a third reader. Full-text articles for abstractsmeeting relevancy criteria will be retrieved. Similarly, ab-stracts or executive summaries of grey literature reportswill then be rated by pairs, and relevant full-text reportswill be retrieved.Pairs of raters will then independently screen for in-clusion of full-text articles and reports for review. Aswith the abstract screening process, criteria for full-textinclusion will be developed and tested. If disagreementoccurs, a third reader will review the article in question.Integration is a broad term, and to ensure important ar-ticles are not missed, a fairly high number of abstractsand full-text articles will be screened. We anticipate onlyOelke et al. Systematic Reviews  (2015) 4:99 Page 6 of 8a smaller number of these articles will meet the rele-vancy criteria. The bibliographies of full-text studiesmeeting inclusion criteria will be scanned to identifyadditional articles of possible relevance which will thenundergo the same selection process. Ratings and selec-tion of Portuguese abstracts and articles will be con-ducted by research team members in Brazil and willfollow the same procedures as outlined above.Appraisal of study qualityA quality appraisal tool [20] was adapted, tested, and im-plemented successfully by one of our primary re-searchers in a previous knowledge synthesis [21]. Allselected studies will be independently assessed by tworeviewers using the tool prior to data extraction. Thisstep will be critical to ensure selection of only high-quality studies discussing measurement of the indica-tor(s) in enough detail to enable replication.Charting the dataData extractionPeer-reviewed articles and grey literature reports con-sidered relevant will be thematically grouped by indi-cator to facilitate extraction of information. Prior tothe commencement of charting of the data, an extrac-tion template will be developed. The extraction cat-egories to be included are author, year of publication,country of publication, integration principle, indicator,study type, sample population including health carecontext, name and description of tool, components ofthe tool, and limitations. This data extraction templatewill be tested by the reviewers on a small set ofarticles to determine the usefulness of the categoriesand identify any gaps in the template. Furthermore,consistency across data extraction will be determined,and inconsistencies will be discussed and resolvedprior to moving along to further data extraction. Datawill be extracted by a single reviewer with systematicaudits completed to ensure accuracy and quality of ex-tracted data.Collating, summarizing, and reporting resultsFrom the extraction template, a listing of relevant toolsavailable for each of the indicators will be compiledincluding key components of the tool (e.g. validity andreliability testing, type of tool). From these summaries, anarrative analysis of the studies will be developedaddressing overall strengths and limitations of the know-ledge base, the quantity of studies/articles for eachindicator, measures and methods used, the quality ofexisting measures, questions addressed, and evidencegaps. Qualitative thematic analysis [13] will be usedwhere appropriate for this synthesis.Draft reports will be reviewed by the project team.The revised report will be circulated to the full researchteam. They will be invited to assess whether the summa-rized information in preliminary form has captured theindicators of interest as well as impressions about thevalidity of conclusions. They will also be asked to high-light the findings most immediately useful, help developrecommendations and key messages, and make sugges-tions for further formatting and communication. Feed-back from the research team will inform revisions to thereport. The final systematic review report will include asingle page of key messages and summary of the policycontext of the review, a three-page executive summary,the full report with appendices, and one or two add-itional user-friendly communication tools as suggestedby knowledge-users.ConsultationThis knowledge synthesis uses an integrated knowledgetranslation (KT) approach [22]. Throughout the develop-ment of the proposal and initial implementation ofthe research, we have been working directly withknowledge-users (decision-makers and policy-makers)in all of the research processes. We have includedknowledge-users from each jurisdiction on our researchteam. These integrated KT approaches will ensure therelevance of the research and facilitate the disseminationand uptake of research results.An end-of-grant KT event will be held for knowledgedissemination and exchange with researchers andknowledge-users from the three jurisdictions (Alberta,BC, and Brazil). Knowledge-user team members will becritical in identifying about 50 provincial, national, andinternational stakeholders to participate. The objectivesof the meeting are to present the results from the sys-tematic review for stakeholder discussion and validation,discuss implications of results within local contexts andhow they will be used by different stakeholders, andidentify outstanding questions. The meeting will behosted through a blended format using face-to-face andinternet technology to allow for broad participation andto reduce costs. Stakeholders will all be connectedthrough the internet (e.g. Web-ex). Similar to videocon-ferencing, we can simultaneously reach all stakeholdersfor an overall presentation online, break-out for smallinteractive sessions, and reconnect as a larger group.Our team has successfully hosted a number of these dis-tributed events with close to 100 participants. The mixbetween face-to-face and internet participation is a cost-effective way to enable networking amongst partnersand other key stakeholders while at the same time creat-ing synergies across the jurisdictions. All participantswill receive a copy of the systematic review and a writtenreport of the event proceedings.Oelke et al. Systematic Reviews  (2015) 4:99 Page 7 of 8Ethical considerationsThe research protocol was submitted to Ethics Boards atall three hosting sites: the University of British ColumbiaOkanagan Behavioural Research Ethics Board, Universityof Calgary Conjoint Health Research Ethics Board andResearch Services Office, and Universidade Federal doRio Grande do Sul Research Ethics Committee. Ethicsapproval has been received from all three sites. Partici-pants in the Delphi study and focus group participantswill be presented with a specific consent form outliningthe research objectives, research implications, and mea-sures to ensure confidentiality along with a list of the re-search team members and their contact information.They will be informed participation in the study is vol-untary. Delphi study participants will be informed theirconsent is implied by electronic submission of the com-pleted survey.Project statusAt the time of submission of this paper, the secondround of the Delphi process has been started. Focusgroups in BC and Brazil have been completed.DiscussionThe outputs of this knowledge synthesis are a list of in-dicators reflective of health system integration as estab-lished through panel consensus, a collection of detailedmeasurement tools for capturing each of the indicators,and a final report outlining the advantages and chal-lenges with each indicator and measurement tool and itsapplication for evaluating integration. This study willhelp stakeholders and policy-makers working in variousjurisdictions on health system integration to measure thesuccess of different strategies through appropriate indica-tors and tools. This will ultimately lead to better design ofhealth care systems and better health outcomes.A number of potential challenges could affect progresson this knowledge synthesis. First, given the nature ofthe concepts under study, a substantial number of po-tential indicators may be generated for consideration.Use of the modified Delphi technique will mitigate thisissue, as panel members will be iteratively choosing indi-cators considered most important to the research ques-tions. This will ensure a focus on only those indicatorsthat are measurable, relevant, and meaningful. Second,the literature searches may result in a vast quantity ofliterature to examine. Past team experience with know-ledge syntheses has facilitated the development of an ef-ficient method of screening abstracts and rating full-textarticles that allows for rapid movement through the pre-liminary stages and focus on extraction of relevant infor-mation. Third, the team has also developed effectivestrategies to conduct international research. Establishedworking relationships currently exist with the Brazilianuniversity. Language issues will be mitigated as all re-search team members are fluent in the English language.AbbreviationsBC: British Columbia; KT: knowledge translation; PI: primary investigator;WHO: World Health Organization.Competing interestsThe authors declare that they have no competing interests.Authors’ contributionsNDO is the principal investigator on this knowledge synthesis. The originalproposal for the knowledge synthesis was developed by NDO, ES, and MAL,and all are co-principal investigators on the research. NDO and CVB draftedthe protocol manuscript. NDO, ES, MAL, and CVB revised the protocol. Allauthors have read and approved the final manuscript.Authors’ informationNDO, PhD, RN, is an assistant professor at the School of Nursing, Universityof British Columbia, Okanagan, and an adjunct assistant professor atCommunity Health Sciences, University of Calgary. NDO has expertise inhealth services integration, primary health care, and knowledge synthesismethods.ES is the Director at Workforce Research and Evaluation, Alberta HealthServices, and an adjunct assistant professor at the Faculty of Social Work,University of Calgary. ES has expertise in knowledge synthesis methods,integration, and interprofessional teams.MAL is a full professor at the School of Nursing, Universidade Federal do RioGrande do Sul. MAL has expertise in health care utilization and transitions incare.CVB, BSc, BSN, RN, is a PhD student at the University of British Columbia,Okanagan.AcknowledgementsThis study (KRS 138203) is funded by the Canadian Institutes for HealthResearch. Funding for publishing this manuscript is provided by the Provost’sOffice, University of British Columbia, Okanagan.Author details1School of Nursing, University of British Columbia, Okanagan 3333 UniversityWay, Kelowna, British Columbia V1V 1V7, Canada. 2Workforce Research &Evaluation, Alberta Health Services, 10301 Southport Lane SW, Calgary,Alberta T2W 1S7, Canada. 3Escola de Enfermagem, Universidade Federal doRio Grande do Sul, Rua São Manoel, 963, Porto Alegre, Brazil.Received: 2 April 2015 Accepted: 10 July 2015References1. Strandberg-Larsen M, Krasnik A. Measurement of integrated health caredelivery: a systematic review of methods and future research directions.Int J Integr Care. 2009;9:1568–4156.2. Strandberg-Larsen M. Measuring integrated care: an internationalcomparative study. Dan Med Bull. 2011;58:1–22.3. Shortell SM, McCurdy RK. Integrated health systems. In: Rouse WB, CorteseDA, editors. Engineering the system of health care delivery. Amsterdam: IOSPress; 2010. p. 369–82.4. Suter E, Oelke ND, Adair CE, Waddell C, Armitage, GD, Huebner LA. Healthsystems integration definitions, processes and impact: a research synthesis.Alberta Health Services. 2007. http://www.albertahealthservices.ca/Researchers/if-res-hswru-hs-integration-report-2007.pdf. Accessed 27Mar 2015.5. Suter E, Oelke ND, Adair CE, Armitage GD. Ten key principles for successfulhealth systems integration [Special issue]. Healthc Q. 2009; doi:10.12927/hcq.2009.21092.6. Prefeitura de Porto Alegre Secretaria Municipal De Saúde. Plano MunicipalDe Saúde 2014 – 2017 Porto Alegre 2013. http://lproweb.procempa.com.br/pmpa/prefpoa/sms/usu_doc/pms_2014_2017.pdf. Accessed 27 Mar 2015.7. World Health Organization. WHO global strategy on people-centred andintegrated health services. http://www.who.int/servicedeliverysafety/areas/people-centred-care/en/ (2006). Accessed 24 Mar 2015.systems integration study. Hosp Health Serv Adm. 1993;38:467–89.37. Morrisey MA, Alexander J, Burns LR, Johnson V. The effects of managed careon physician and clinical integration in hospitals. Med Care. 1999;doi:10.1097/00005650-199904000-00005.38. Burns L, Walston SL, Alexander JA, Zuckerman HS, Andersen RM, Torrens PR.Just how integrated are integrated delivery systems? Results from a nationalsurvey. Health Care Manage R. 2001;26:20–39.39. Cocozza JJ, Steadman HK, Dennis DL, Blasinsky M, Randolph FL, Johnsen M,et al. Successful systems integration strategies: the ACCESS program forpersons who are homeless and mentally ill. Adm Policy Ment Health. 2000;doi:10.1023/A:1021390225515.40. Baker GR. The challenges of making care safer: leadership and systemtransformation [Special issue]. Healthc Q. 2012; doi:10.12927/hcq.2012.22848.41. Ahgren B, Axelsson R. Evaluating integrated health care: a model formeasurement. Int J Integr Care. 2005;5:1–9.42. Browne G, Roberts J, Garni A, Byrne C, Kertyzia J, Loney P. Conceptualizingand validating the human services integration measure. Int J Integr Care.2004;4:3–12.Oelke et al. Systematic Reviews  (2015) 4:99 Page 8 of 88. Hebert J, Veil A. Monitoring the degree of implementation of an integrateddelivery system. Int J Integr Care. 2004;4:1568–4156.9. VanDeusen LC, Meterko M, Lowcock S, Donaldson-Parlier R, Blakely M,Davies M, et al. Monitoring the progress of system integration. Qual ManagHealth Care. 2002;10:1–11.10. Devers KJ, Shortell SM, Gillies RR, Anderson DA, Mitchell JB, Erickson KM.Implementing organized delivery systems: an integration scorecard. HealthCare Manage Rev. 1994;19:7–20.11. Mohr JJ. Clinical Microsystem Assessment Tool. 2001. http://clinicalmicrosystem.org/wp-content/uploads/2014/07/microsystem_assessment.pdf. Accessed 27Mar 2015.12. de Jong I, Jackson C. An evaluation approach for a new paradigm - healthcare integration. J Eval Clin Pract. 2001; doi:10.1046/j.1365-2753.2001.00285.x.13. Levac D, Colquhoun H, O’Brien KK. Scoping reviews: advancing themethodology. Implement Sci. 2010; doi:10.1186/1748-5908-5-69.14. Thompson M. Considering the implication of variations within Delphiresearch. Fam Pract. 2009; doi:10.1093/fampra/camp051.15. Hsu CC, Sandford BA. The Delphi technique: making sense of consensus.Pract Assessment Res Eval. 2007;12:10.16. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi surveytechnique. J Adv Nurs. 2000; doi:10.1046/j.1365-2648.2000.t01-1-01567.x.17. Greenberg J. Behavior in organizations. 10th ed. Upper Saddle River, NJ:Pearson Education/Prentice Hall; 2011.18. Jayasekara RS. Focus groups in nursing research: methodologicalperspectives. Nurs Outlook. 2012; doi:10.1016/j.outlook.2012.02.001.19. Centre for Reviews and Dissemination, University of York. SystematicReviews: CRD’s guidance for undertaking reviews in health care. York, UK:York Publishing Services Ltd; 2009.20. Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De Aguero L, Truman BI,et al. Developing an evidence-based guide to community preventiveservice—methods [Special issue]. Am J Prev Med. 2000;18:35–43.21. Suter E Deutschlander S. Can interprofessional collaboration provide healthhuman resources solutions? 2010. http://www.albertahealthservices.ca/Researchers/if-res-hswru-knowledge-synthesis-report-2010.pdf. Accessed 24Mar 2015.22. Canadian Institutes of Health Research. Guide to knowledge translationplanning at CIHR: integrated and end-of-grant approaches. 2012. http://www.cihr-irsc.gc.ca/e/documents/kt_lm_ktplan-en.pdf. Accessed 8 Mar 2015.23. Coleman E, Smith J, Frank J, Eilertsen T, Thiare J, Kramer A. Developmentand testing of a measure designed to assess the quality of care transitions.Int J Integr Care. 2002;2:1568–4156.24. Fletcher BW, Lehman W, Wexler HK, Melnick G, Taxman FS, Young DW.Measuring collaboration and integration activities in criminal justice andsubstance abuse treatment agencies [Special issue]. Drug Alcohol Depen.2009; doi:10.1016/j.drugalcdep.2009.01.001.25. Institute for Health Care Improvement. Third next available appointment.2015. http://www.ihi.org/resources/Pages/Measures/ThirdNextAvailableAppointment.aspx. Accessed 27 Mar 2015.26. Alberta Health Services. Alberta health services board quality and patientsafety dashboard: background paper. 2010. http://www.albertahealthservices.ca/patientsfamilies/if-pf-quality-and-patient-safety-dashboard-background-paper.pdf. Accessed 27 Mar 2015.27. Schang L, Waibel S, Thomson S. Measuring care coordination: health systemand patient perspectives. Report prepared for the Main Association of AustrianSocial Security Institutions. 2013. http://eprints.lse.ac.uk/59573/1/__lse.ac.uk_storage_LIBRARY_Secondary_libfile_shared_repository_Content_Schang%2C%20L_Schang_Measuring%20care%20coordination_2014.pdf. Accessed 27Mar 2015.28. Solberg L, Maciosek M, Edwards N, Khanchandani H, Goodman M. Repeatedtobacco-use screening and intervention in clinical practice: health impact andcost effectiveness. Am J Prev Med. 2006; doi:10.1016/j.amepre.2006.03.013.29. Osborn R, Squires D. International perspectives on patient engagement:results from the 2011 commonwealth fund survey. J Ambul Care Manage.2012; doi:10.1097/JAC.0b013e31824a579b.30. Spenceley SM, Andres C, Lapins J, Wedel R, Gelber T, Halma LM. Accountabilityby design: moving primary care reform ahead in Alberta. Sch Public Policy.2013;6:1–44.31. Stevens A, Gillam S. Needs assessment: from theory to practice. BMJ.1998;316:1448–52.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistribution32. Reilly S, Challis D, Burns A, Hughes J. Does integration really make adifference? A comparison of old age psychiatry services in England andNorthern Ireland. Int J Geriatr Psychiatry. 2013; doi:10.1002/gps.942.33. Delon S, Mackinnon B. Alberta’s systems approach to chronic diseasemanagement and prevention utilizing the expanded chronic care model[Special issue]. Healthc Q. 2009; doi:10.12927/hcq.2009.21106.34. Hébert RR, Veil AA. Monitoring the degree of implementation of anintegrated delivery system. Int J Integr Care. 2004;4:5–12.35. Newhouse RP, Mills ME, Johantgen M, Pronovost PJ. Is there a relationshipbetween service integration and differentiation and patient outcomes? Int JIntegr Care. 2003;3:15–28.36. Anderson DA, Gillies RR, Mitchell JB, Morgan KL, Shortell SM.Conceptualizing and measuring integration: findings from the healthSubmit your manuscript at www.biomedcentral.com/submit


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items