UBC Faculty Research and Publications

Network meta-analysis incorporating randomized controlled trials and non-randomized comparative cohort… Cameron, Chris; Fireman, Bruce; Hutton, Brian; Clifford, Tammy; Coyle, Doug; Wells, George; Dormuth, Colin R; Platt, Robert; Toh, Sengwee Nov 5, 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-13643_2015_Article_133.pdf [ 1.93MB ]
JSON: 52383-1.0308637.json
JSON-LD: 52383-1.0308637-ld.json
RDF/XML (Pretty): 52383-1.0308637-rdf.xml
RDF/JSON: 52383-1.0308637-rdf.json
Turtle: 52383-1.0308637-turtle.txt
N-Triples: 52383-1.0308637-rdf-ntriples.txt
Original Record: 52383-1.0308637-source.json
Full Text

Full Text

COMMENTARY Open AccessNetwork meta-analysis incorporatingrandomized controlled trials and non-randomized comparative cohort studies forassessing the safety and effectiveness ofmedical treatments: challenges andopportunitiesChris Cameron1,2,8*, Bruce Fireman3, Brian Hutton1,4, Tammy Clifford1,5, Doug Coyle1, George Wells1,Colin R. Dormuth6, Robert Platt7 and Sengwee Toh2AbstractNetwork meta-analysis is increasingly used to allow comparison of multiple treatment alternatives simultaneously,some of which may not have been compared directly in primary research studies. The majority of networkmeta-analyses published to date have incorporated data from randomized controlled trials (RCTs) only; however,inclusion of non-randomized studies may sometimes be considered. Non-randomized studies can complementRCTs or address some of their limitations, such as short follow-up time, small sample size, highly selected population,high cost, and ethical restrictions. In this paper, we discuss the challenges and opportunities of incorporating bothRCTs and non-randomized comparative cohort studies into network meta-analysis for assessing the safety andeffectiveness of medical treatments. Non-randomized studies with inadequate control of biases such asconfounding may threaten the validity of the entire network meta-analysis. Therefore, identification and inclusion ofnon-randomized studies must balance their strengths with their limitations. Inclusion of both RCTs and non-randomizedstudies in network meta-analysis will likely increase in the future due to the growing need to assess multipletreatments simultaneously, the availability of higher quality non-randomized data and more valid methods, andthe increased use of progressive licensing and product listing agreements requiring collection of data over thelife cycle of medical products. Inappropriate inclusion of non-randomized studies could perpetuate the biasesthat are unknown, unmeasured, or uncontrolled. However, thoughtful integration of randomized and non-randomized studies may offer opportunities to provide more timely, comprehensive, and generalizable evidenceabout the comparative safety and effectiveness of medical treatments.Keywords: Network meta-analysis, Randomized controlled trials, Observational studies, Pharmacoepidemiology,Comparative effectiveness research, Distributed research networks* Correspondence: ccame056@uottawa.ca1School of Epidemiology, Public Health and Preventive Medicine, Universityof Ottawa, 451 Smyth Road, Suite RGN 3105, Ottawa, ON K1H 8 M5, Canada2Department of Population Medicine, Harvard Medical School and HarvardPilgrim Health Care Institute, 133 Brookline Avenue, 6th Floor, Boston, MA02215, USAFull list of author information is available at the end of the article© 2015 Cameron et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.Cameron et al. Systematic Reviews  (2015) 4:147 DOI 10.1186/s13643-015-0133-0BackgroundMany medical conditions exist for which there are mul-tiple treatment options. Meta-analysis is a widely usedapproach for aggregating results from multiple studiesto provide more robust evidence on the safety and ef-fectiveness of various treatments [1]. However, evidencebased on pair-wise meta-analysis only considers twotreatments at a time. Accordingly, new meta-analyticmethods have emerged to permit simultaneous compari-son of multiple treatment options across studies thatcompare two or more treatments. These methods aremost commonly referred to as network meta-analysis(NMA) [2, 3].Although earlier NMAs only included randomized con-trolled trials (RCTs) [4], recent NMAs have begun to con-sider both RCTs and non-randomized studies [5–9]. Inthis paper, we describe NMA involving both RCTs andnon-randomized comparative cohort studies—defined ascohort studies that compare two or more treatment alter-natives (which may include usual care or no treatment)using observational data. We discuss some of the promisesand challenges, highlight the potential application ofNMA in multi-center distributed data networks, and offerinsights on opportunities for improving the application ofthis methodology.Introduction to network meta-analysisA network meta-analysis (sometimes called mixed or mul-tiple treatments meta-analysis) is a method for comparingmore than two interventions, some of which may not havebeen compared directly head-to-head in the same study(Fig. 1) [2, 3, 9–13]. The key assumption underlying anyNMA is exchangeability of the studies [2, 3, 14]. That is,all studies measure the same underlying relative treatmenteffects, and any observed differences are due to chance.Stated another way, all treatments included in the NMAcould have been included in the same study, and treat-ments are genuinely competing interventions [2, 3, 14].For example, in Fig. 1, AC trials do not have B arms andAB trials do not have treatment C arms; however, theFig. 1 Network meta-analysis and assessment of the exchangeability assumption. Panel a presents a network meta-analysis assessing whether theexchangeability assumption holds for studies comparing treatments c versus a and treatments b versus a. Panel b presents a table comparingthe patient and study characteristics for these two studies. Panel c assesses and plots the baseline risk of the common comparator (treatment A)for both studies and the combined result using a box plot. We have compared patient and study characteristics at the pair-wise comparison level(e.g., a versus b) although they can also be conducted at the treatment level (e.g., a, b, and c)Cameron et al. Systematic Reviews  (2015) 4:147 Page 2 of 8assumption underlying a NMA is that if an AB trial wouldhave included a C arm, it would measure the same under-lying relative effect for AC as the AC trials included in thenetwork.To assess exchangeability, one can collect informationabout the studies and carefully consider whether theyappear similar enough to be compared based on inspec-tion of this information (Fig. 1) [2, 3, 14]. Although thisapproach is intuitive, it can sometimes be subjective.Another way to assess exchangeability is to compare theevent rate in the common treatment arm(s) [2, 3, 14].Similar event rates may provide some reassurance thatthe populations are comparable. However, even if therates differ, the exchangeability assumption may stillhold if the populations do not differ in characteristicsthat are modifiers of the treatment effect.Lack of exchangeability in NMA can produce discrep-ancy in the treatment effect estimated from direct (solidlines in panel A of Fig. 1) and indirect evidence (dashedlines in panel a of Fig. 1), sometimes also known as in-consistency [15]. There are various statistical methods toevaluate inconsistency when closed loops are available(i.e., both direct and indirect evidence are available toallow a comparison), although issues such as low statis-tical power may limit the applicability of some of thesemethods [15].Rationale and caveats for including non-randomized stud-ies in NMAWith a sufficiently large sample, well-designed RCTs areexpected to achieve high internal validity by balancingall measured and unmeasured prognostic factors acrossintervention groups through random allocation [11, 16].However, RCTs are not without their limitations. Theyoften have short follow-up time, small sample size,highly selected population, high cost, and ethical con-straints to study certain treatments or populations.Well-designed, high-quality non-randomized studies cancomplement RCTs or address some of their limitations(Table 1) [17–20]. These studies may have longer follow-up time, larger sample size, and more generalizable pop-ulations who receive various treatments in real-worldsettings.When considering the inclusion of both RCTs and non-randomized studies in NMA, the quality of evidence under-pinning a network should be carefully assessed for eachpair-wise comparison in the network. Non-randomizedstudies are vulnerable to several biases, including con-founding which occurs when treatment groups differ intheir underlying risk for the outcome [21–23]. Studies thatdo not appropriately account for confounding factors maytherefore produce biased effect estimates (Fig. 2) [24].Therefore, the inclusion of non-randomized studies inNMA requires careful consideration of the validity of thestudies. The Grading of Recommendations Assessment,Development, and Evaluation (GRADE) working grouphas developed a framework for assessing the quality ofevidence from non-randomized studies in the contextof NMA [25]. Other guidelines, such as the STrength-ening the Reporting of OBservational studies in Epi-demiology (STROBE) guidelines [26] and the guidelinesfor good pharmacoepidemiology practices [27], alsooffer useful guidance to assess the quality of non-randomized studies. It is still important to carefully as-sess potential treatment effect modifiers even in high-quality non-randomized studies.Another important issue to consider is whether thenon-randomized studies address the same researchquestions or estimate the same treatment effects as theRCTs. The most commonly used analytic approach inRCTs is the intention-to-treat approach, which esti-mates the effect of treatment initiation. Other analysesthat can be done in RCTs or non-randomized studiesinclude as-treated analysis (which compares the treat-ments that the patients actually receive), per-protocolanalysis (which includes only patients who adhere tothe trial protocol), and other analyses such as inverseprobability weighting that appropriately account fortime-varying confounding [28]. Depending on analyticmethods used, non-randomized studies that comparethe same treatment alternatives may produce treatmenteffects that are valid but different from that estimatedin the RCT [28–31].Table 1 Advantages and disadvantages of incorporating bothrandomized controlled trials and non-randomized comparativecohort studies in network meta-analysisAdvantages• Non-randomized studies can complement randomized controlled trialsor address some of their limitations, such as short follow-up time, smallsample size, highly selected population, high cost, and ethicalrestrictions.• Incorporating both types of data allows assessments of multipletreatments simultaneously, including treatments that may not havebeen studied in randomized controlled trials.• Incorporating both types of data allows larger sample size and morediverse populations, thereby improving the generalizability of thefindings.• Incorporating non-randomized studies might improve network densityand connect disconnected networks.Disadvantages• Including low-quality, non-randomized comparative cohort studiescould perpetuate the biases that are unknown, unmeasured, oruncontrolled.• There is a greater risk of violating the exchangeability assumption ofnetwork meta-analysis, especially if broad populations are considered.• The analysis may be more complex, time- and resource-intensive, andless understood than network meta-analysis that only includesrandomized controlled trials.Cameron et al. Systematic Reviews  (2015) 4:147 Page 3 of 8Network meta-analysis of RCTs and non-randomizedstudiesThere are various approaches for combining RCTs andnon-randomized studies in NMA [9, 13, 32, 33]. Naïvepooling of all randomized and non-randomized study-level data, using either frequentist or Bayesian NMAmethods, is the simplest approach and does not differen-tiate between two study designs [13].Another way to include non-randomized studies inNMA is to use them as prior information or in the formof a hierarchical model that allows for bias adjustment[13]. When incorporating them as prior information,abcFig. 2 Potential bias resulting in network meta-analyses incorporating both randomized controlled trials and non-randomized comparative cohortstudies. a Potential for confounding—randomized versus non-randomized studies. b Indirect estimate from randomized controlled trial. c Indirectestimate from non-randomized studyCameron et al. Systematic Reviews  (2015) 4:147 Page 4 of 8non-randomized studies are analyzed separately andresults are then used as prior information for the RCTmodel. The potential biases associated with non-randomized data can be modeled by adjusting the priordistribution. To downweigh the non-randomized infor-mation, the variance parameter can be inflated; to adjustfor overestimation or underestimation of the treatmenteffect, the mean of the prior information can be shifted.Another approach—a Bayesian hierarchical model—isgenerally considered the most flexible [9, 13, 32, 33]. ABayesian hierarchical model is a statistical model thatestimates the parameters of the posterior distributionusing the Bayesian method [9, 13, 32, 33]. In the model,a study-design level (e.g., RCT, non-randomized study)is introduced [9, 13, 32, 33]. This approach allows forbias adjustments discussed above as well as direct com-parison of study design-specific estimates to overall es-timates. For example, evidence from individual studiesof the same design can first be combined to producestudy-design level estimates; the study-design level esti-mates can then be combined to obtain overall estimates[9, 13, 32, 33]. It also gives an estimate of consistencybetween study designs. There is limited published re-search in this area, especially the latter two approaches.Furthermore, there is a lack of consensus on whatdegree of bias adjustment to apply to non-randomizedstudies.Figure 3 presents scenarios that may occur when com-bining RCTs and non-randomized studies in NMA. Insome cases (e.g., drug B versus drug A), findings fromnon-randomized studies align with those reported inRCTs. In other situations (drug D versus drug C), thefindings reported in the non-randomized studies do notalign with those reported in RCTs. Investigators anddecision makers are generally more likely to have confi-dence in estimates in the scenario where findings fromboth study designs are consistent compared with thescenario where there are discrepancies. However, thediscrepancies may yield insight regarding biases in thenon-randomized studies (e.g., residual confounding), effectmodification by specific patient characteristic, or differ-ences in various treatment effects (e.g., intention-to-treateffects and as-treated effects) that may not have been no-ticed had both study designs not been considered.Incorporation of both RCTs and non-randomizedstudies into NMA typically requires considerably moretime, effort, and costs compared to including only RCTs.The decision to include non-randomized studies shouldcarefully consider the expected additional benefits giventhe additional time, effort, and costs. Restricting theFig. 3 Combining and comparing findings from network meta-analysis using randomized controlled trials and non-randomized comparative cohortstudies. We assume for this example a network which consists of four treatments, namely A, B, C, and D. NMA network meta-analysis, NRS nonrandomized studies, RCT randomized controlled trialsCameron et al. Systematic Reviews  (2015) 4:147 Page 5 of 8analysis to specific types of non-randomized design oranalysis (i.e., propensity score matching) may some-times reduce time, effort, and costs to conduct NMAbut may introduce bias due to exclusion of otherwiseeligible studies.Network meta-analysis of non-randomized studies inlarge distributed data networksOver the past number of years, we have seen an increasein the development of distributed data networks to assistin conducting non-randomized studies. In the USA, theMini-Sentinel program [34] has developed a distributednetwork of 18 data partners with information from over178 million individuals [35], while the Canadian Net-work for Observational Drug Effect Studies (CNODES)[36] includes health and prescription records of over 40million people from eight jurisdictions in Canada andabroad. Other examples of distributed networks includethe “Exploring and Understanding Adverse DrugReactions by integrative mining of clinical records andbiomedical knowledge” (EU-ADR) project in Europe[37] and the Asian Pharmacoepidemiology Network(AsPEN) [38]. These networks permit comparativesafety and effectiveness assessment of medical prod-ucts across multiple databases without creation of acentral data warehouse [34, 36, 39].Both pair-wise meta-analysis and NMA are well-suited for distributed data networks. Traditionally, non-randomized studies for meta-analysis are identified bysystematic review of published and unpublished studies.However, these studies often include a broad array ofstudies with different study questions, study designs,analytic methods, and completeness of information.Combining such heterogeneous information in meta-analysis can sometimes be problematic and challenging.On the other hand, the studies performed in distributeddata networks often use common protocols, data models,or both, which improves the comparability of analysisperformed at each site [34, 36, 39]. Both CNODES andMini-Sentinel have used pair-wise meta-analysis to com-bine data across data sources [36, 40–43]. NMA is well-suited for incorporating data from these networks whenthe study compares multiple treatment options, as in aMini-Sentinel assessment of anti-hyperglycemic agentsand acute myocardial infarction [44].Further, access to data from large distributed data net-works may allow more detailed assessment and adjust-ment for heterogeneity and inconsistency. Larger samplesizes derived from these networks will allow detailed as-sessment of the benefits and harms of treatments insub-populations that may have been understudied inRCTs. Further, access to patient-level data will facilitatethe conduct of meta-regression analyses to adjust fordifferences in characteristics between studies. This maybe particularly important, because even if the estimatefrom a non-randomized study is unbiased, the popula-tion may differ from those studied in RCTs.Currently, data from most distributed data networksare only available to those involved in the networks; fu-ture work is needed investigating the advantages anddisadvantages of making de-identified or summary-leveldata from these networks more accessible for analysis byothers.Discussion and conclusionsThe interest in and need for incorporating both RCTsand non-randomized studies in NMA will likely in-crease in the future due to the growing need to assessmultiple treatments simultaneously, improvement inthe quality and validity of non-randomized data andanalytic methods, and the global movement towardsprogressive licensing [45] and product listing agree-ments [46] where information on a medical product ismonitored throughout its life cycle for regulatory andreimbursement purposes. Incorporating both types ofdata in NMA may improve precision, allow for a widerarray of treatments to be considered (i.e., expand net-work or connect otherwise “disconnected network”),and provide real-world and more generalizable evi-dence on the risks and benefits of medical treatments.However, the inclusion of low-quality, non-randomizedstudies with inadequate control for biases may threatenthe validity of the NMA findings. More studies areneeded to compare the validity of different approachesthat combine RCTs and non-randomized studies inNMA. Although the inclusion of both types of data inNMA poses several methodological challenges, it alsooffer promises to provide more timely, comprehensive,and generalizable evidence on the comparative safetyand effectiveness of medical treatments.AbbreviationsCNODES: Canadian Network for Observational Drug Effect Studies;NMA: network meta-analysis; RCTs: randomized controlled trials.Competing interestsCC is now a Director at Cornerstone Research Group Inc., a health careconsultancy which consults for the pharmaceutical, biotech, and medicaldevice industry. He was not employed by Cornerstone Research Group Inc.when this manuscript was initially drafted.Authors’ contributionsCC wrote the first draft, incorporated suggested revisions/edits by co-authors,approved the final version, and agreed to be accountable for all aspects of thework in ensuring that questions related to the accuracy or integrity of any partof the work are appropriately investigated and resolved. ST aided in the design,edited early versions of the manuscript, approved the final version, and agreedto be accountable for all aspects of the work in ensuring that questionsrelated to the accuracy or integrity of any part of the work are appropriatelyinvestigated and resolved. CC, BF, TC, DC, BH, GW, CD, and RP revised themanuscript, approved the final version, and agreed to be accountable for allCameron et al. Systematic Reviews  (2015) 4:147 Page 6 of 8aspects of the work in ensuring that questions related to the accuracy orintegrity of any part of the work are appropriately investigated and resolved.AcknowledgementsCC is a recipient of a Vanier Canada Graduate Scholarship through CIHR(funding reference number—CGV 121171). He also received a CIHR CanadaGraduate Scholarship—Michael Smith Foreign Study Supplement (fundingreference number—FFS 134035) and University of Ottawa student mobilitybursary to study at the Department of Population Medicine, Harvard MedicalSchool and Harvard Pilgrim Health Care Institute under the supervision of ST.CC is also a trainee on the CIHR Drug Safety and Effectiveness NetworkMeta-Analysis team grant (funding reference number—116573). BH is fundedby a New Investigator award from the Canadian Institutes of Health Researchand the Drug Safety and Effectiveness Network.Author details1School of Epidemiology, Public Health and Preventive Medicine, Universityof Ottawa, 451 Smyth Road, Suite RGN 3105, Ottawa, ON K1H 8 M5, Canada.2Department of Population Medicine, Harvard Medical School and HarvardPilgrim Health Care Institute, 133 Brookline Avenue, 6th Floor, Boston, MA02215, USA. 3Division of Research, Kaiser Permanente Northern California,2000 Broadway, Oakland, CA 94612, USA. 4Ottawa Hospital Research Institute,Center for Practice Changing Research Building, Ottawa Hospital—GeneralCampus, PO Box 201B, Ottawa, ON K1H 8 L6, Canada. 5Canadian Agency forDrugs and Technologies in Health, 865 Carling Ave., Suite 600, Ottawa, ONK1S 5S8, Canada. 6Department of Anesthesiology, Pharmacology andTherapeutics, University of British Columbia, Vancouver, BC V6T 1Z3, Canada.7Department of Epidemiology and Biostatistics, McGill University, 4060 SteCatherine W #300, Montréal, Québec H3Z 2Z3, Canada. 8Evidence SynthesisGroup, Cornerstone Research Group Inc., 3228 South Service Road,Burlington, ON L7N 3H8, Canada.Received: 21 May 2015 Accepted: 13 October 2015References1. Sutton a J, Abrams KR. Bayesian methods in meta-analysis and evidencesynthesis. Stat Methods Med Res. 2001;10:277–303.2. Caldwell DM, Ades a E, Higgins JPT. Simultaneous comparison of multipletreatments: combining direct and indirect evidence. BMJ. 2005;331:897–900.3. Dias S, Sutton AJ, Ades a E, Welton NJ. Evidence synthesis for decisionmaking 2: a generalized linear modeling framework for pairwise andnetwork meta-analysis of randomized controlled trials. Med Decis Making.2013;33:607–17.4. Nikolakopoulou A, Chaimani A, Veroniki AA, Vasiliadis HS, Schmid CH, SalantiG. Characteristics of networks of interventions: a description of a databaseof 186 published networks. PloS ONE. 2014;9:1–10.5. Hutton B, Joseph L, Fergusson D, Mazer CD, Shapiro S, Tinmouth A. Risks ofharms using antifibrinolytics in cardiac surgery: systematic review andnetwork meta-analysis of randomised and observational studies. BMJ.2012;345:e5798–8.6. Vlaar PJ, Mahmoud KD, Holmes DR, van Valkenhoef G, Hillege HL, van derHorst ICC, et al. Culprit vessel only versus multivessel and stagedpercutaneous coronary intervention for multivessel disease in patientspresenting with ST-segment elevation myocardial infarction: a pairwise andnetwork meta-analysis. J Am Coll Cardiol. 2011;58:692–703. Elsevier Inc.7. Stegeman BH, de Bastos M, Rosendaal FR, van Hylckama Vlieg A,Helmerhorst FM, Stijnen T, et al. Different combined oral contraceptives andthe risk of venous thrombosis: systematic review and network meta-analysis.BMJ. 2013;347:f5298.8. Robertson C, Close A, Fraser C, Gurung T, Jia X, Sharma P, et al. Relativeeffectiveness of robot-assisted and standard laparoscopic prostatectomy asalternatives to open radical prostatectomy for treatment of localisedprostate cancer: a systematic review and mixed treatment comparisonmeta-analysis. BJU Int. 2013;112:798–812.9. Verde PE, Ohmann C. Combining randomized and non-randomizedevidence in clinical research: a review of methods and applications. ResSynth Methods. 2015;6:45–62.10. Ioannidis JP a. Integration of evidence from multiple meta-analyses: aprimer on umbrella reviews, treatment networks and multiple treatmentsmeta-analyses. CMAJ. 2009;181:488–93.11. Jansen JP, Naci H. Is network meta-analysis as valid as standard pairwisemeta-analysis? It all depends on the distribution of effect modifiers. BMCMed. 2013;11:159.12. Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, et al.The PRISMA Extension statement for reporting of systematic reviewsincorporating network meta-analyses of health care interventions: checklistand explanations. Ann Intern Med. 2015;162:777.13. Schmitz S, Adams R, Walsh C. Incorporating data from various trial designs intoa mixed treatment comparison model. Stat Med. 2013;32(17):2935–2949.14. Van Valkenhoef G, Tervonen T, Zwinkels T, de Brock B, Hillege H. ADDIS: adecision support system for evidence-based medicine. Decis Support Syst.2013;55:459–75.15. Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades AE. Evidencesynthesis for decision making 4: inconsistency in networks ofevidence based on randomized controlled trials. Med Decis Making.2013;33:641–56.16. Freemantle N, Marston L, Walters K. Making inferences on treatment effectsfrom real world data: propensity scores, confounding by indication, andother perils for the unwary in observational research. BMJ. 2013;6409:1–5.17. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observationalstudies, and the hierarchy of research designs. N Engl J Med.2000;342:1887–92.18. Vandenbroucke J, Psaty B. Benefits and risks of drug treatments: how tocombine the best evidence on benefits with the best data about adverseeffects. JAMA. 2008;300:2417–9.19. Sørensen HT, Lash TL, Rothman KJ. Beyond randomized controlled trials: acritical comparison of trials with nonrandomized studies. Hepatology.2006;44:1075–82.20. Ioannidis JP, Haidich a B, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, etal. Comparison of evidence of treatment effects in randomized andnonrandomized studies. JAMA. 2001;286:821–30.21. Rosenbaum P, Rubin D. The central role of the propensity score inobservational studies for causal effects. Biometrika. 1983;70(1):41–55.22. Walker AM. Confounding by indication. Epidemiology. 1996;7:335–6.23. Grimes D a, Schulz KF. Bias and causal associations in observationalresearch. Lancet. 2002;359:248–52.24. Deeks JJ, Dinnes J, D’Amico R, Sowden a J, Sakarovitch C, Song F, et al.Evaluating non-randomised intervention studies. Health Technol Assess.2003;7:iii – x, 1–173.25. Puhan M a, Schunemann HJ, Murad MH, Li T, Brignardello-Petersen R,Singh J a, et al. A GRADE Working Group approach for rating the qualityof treatment effect estimates from network meta-analysis. BMJ.2014;349:g5630–0.26. Von Elm E, Altman D, Pocock D, Gotzsche, Peter; Vandenbroucke J.Strengthening the reporting of observational studies in epidemiology(STROBE) statement: guidelines for reporting observational studies. BMJ BrMed. 2007;335:20–2.27. International Society for Pharmacoepidemiology. Guidelines for goodpharmacoepidemiology practices (GPP). Pharmacoepidemiol Drug Saf.2008;17:200–8.28. Hernán M, Hernández-Díaz S. Beyond the intention-to-treat in comparativeeffectiveness research. Clin Trials. 2012;9:48–55.29. Hernán M a, Alonso A, Logan R, Grodstein F, Michels KB, Willett WC, et al.Observational studies analyzed like randomized experiments. Epidemiology.2008;19:766–79.30. Kurth T, Walker AM, Glynn RJ, Chan KA, Gaziano JM, Berger K, et al. Resultsof multivariable logistic regression, propensity matching, propensityadjustment, and propensity-based weighting under conditions ofnonuniform effect. Am J Epidemiol. 2006;163:262–70.31. Toh S, Manson JE. An analytic framework for aligning observational andrandomized trial data: application to postmenopausal hormone therapy andcoronary heart disease. Stat Biosci. 2013;5:344–60.32. McCarron CE, Pullenayegum EM, Thabane L, Goeree R, Tarride J-E. Theimportance of adjusting for potential confounders in Bayesian hierarchicalmodels synthesising evidence from randomised and non-randomisedstudies: an application comparing treatments for abdominal aorticaneurysms. BMC Med Res Methodol. 2010;10:64.33. McCarron CE, Pullenayegum EM, Thabane L, Goeree R, Tarride JE. Bayesianhierarchical models combining different study types and adjusting forcovariate imbalances: a simulation study to assess model performance. PLoSOne. 2011;6(10):e25635. doi:10.1371/journal.pone.0025635.Cameron et al. Systematic Reviews  (2015) 4:147 Page 7 of 834. Behrman RE, Benner JS, Brown JS, McClellan M, Woodcock J, Platt R.Developing the Sentinel System—a national resource for evidencedevelopment. N Engl J Med. 2011;364:498–9.35. Mini-Sentinel Data Core. Mini-Sentinel Distributed Database SummaryReport—Year 4. 2014.36. Suissa S, Henry D, Caetano P, Dormuth CR, Ernst P, Hemmelgarn B, et al.CNODES: the Canadian network for observational drug effect studies. OpenMed. 2012;6:134–40.37. Coloma PM, Schuemie MJ, Trifirò G, Gini R, Herings R, Hippisley-Cox J, et al.Combining electronic healthcare databases in Europe to allow for large-scale drug safety monitoring: the EU-ADR Project. Pharmacoepidemiol DrugSaf. 2011;20:1–11.38. Andersen M, Bergman U, Choi N-K, Gerhard T, Huang C, Jalbert J, et al. TheAsian Pharmacoepidemiology Network (AsPEN): promoting multi-nationalcollaboration for pharmacoepidemiologic research in Asia.Pharmacoepidemiol Drug Saf. 2013;22:700–4.39. Brown J, Holmes J, Shah K, Hall K. Distributed health data networks: apractical and preferred approach to multi-institutional evaluations ofcomparative effectiveness, safety, and quality of care. Med Care. 2010;48:45–51.40. Toh S, Reichman ME, Houstoun M, Southworth MR. Comparative risk forangioedema associated with the use of drugs that target the renin-angiotensin-aldosterone system. PloS ONE. 2014;172:1582–9.41. Filion KB, Chateau D, Targownik LE, Gershon A, Durand M, Tamim H, et al.Proton pump inhibitors and the risk of hospitalisation for community-acquired pneumonia: replicated cohort studies with meta-analysis. Gut.2014;63:552–8.42. Dormuth C, Hemmelgarn B. Use of high potency statins and rates ofadmission for acute kidney injury: multicenter, retrospective observationalanalysis of administrative databases. BMJ. 2013;880:1–10.43. Dormuth CR, Filion KB, Paterson JM, James MT, Teare GF, Raymond CB, et al.Higher potency statins and the risk of new diabetes: multicentre,observational study of administrative databases. BMJ. 2014;348:g3244–4.44. Fireman B, Toh S, Butler M. A protocol for active surveillance of acutemyocardial infarction in association with the use of a new antidiabeticpharmaceutical agent. Drug Saf. 2012;21:282–90.45. Eichler H-G, Oye K, Baird LG, Abadie E, Brown J, Drum CL, et al. Adaptivelicensing: taking the next step in the evolution of drug approval. ClinPharmacol Ther. 2012;91:426–37.46. Morgan SG, Thomson PA, Daw JR, Friesen MK. Inter-jurisdictionalcooperation on pharmaceutical product listing agreements: views fromCanadian provinces. BMC Health Serv Res. 2013;13:34.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionSubmit your manuscript at www.biomedcentral.com/submitCameron et al. Systematic Reviews  (2015) 4:147 Page 8 of 8


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items