RESEARCH ARTICLE Open AccessMethods for evaluating adverse drug eventcy departmentdefinitive when assigning preventability ratings.reviewWoo et al. BMC Medical Research Methodology (2018) 18:160 https://doi.org/10.1186/s12874-018-0617-4Research Institute, 828 West 10th Ave, Vancouver, BC V5Z 1M9, CanadaFull list of author information is available at the end of the article2Department of Emergency Medicine, University of British Columbia, 855West 12th Avenue, Vancouver, BC V5Z 1M9, Canada3Centre for Clinical Epidemiology and Evaluation, Vancouver CoastalKeywords: Adverse drug event, Adverse drug reaction, Preventability, Preventable, Explicit review, Implicit* Correspondence: chohl@mail.ubc.caConclusion: There was good agreement between all three methods of determining the preventability of adverse drugevents. However, clinicians found the algorithmic approach constraining, and preferred a best practice-basedassessment method.patientsStephanie A. Woo1, Amber Cragg2,3, Maeve E. Wickham3,4, David Peddie3,5, Ellen Balka3,5, Frank Scheuermeyer2,Diane Villanyi6 and Corinne M. Hohl2,3,7*AbstractBackground: There is a high degree of variability in assessing the preventability of adverse drug events, limitingthe ability to compare rates of preventable adverse drug events across different studies. We compared threemethods for determining preventability of adverse drug events in emergency department patients and exploredtheir strengths and weaknesses.Methods: This mixed-methods study enrolled emergency department patients diagnosed with at least one adversedrug event from three prior prospective studies. A clinical pharmacist and physician reviewed the medical andresearch records of all patients, and independently rated each event’s preventability using a “best practice-based”approach, an “error-based” approach, and an “algorithm-based” approach. Raters discussed discordant ratings untilreaching consensus. We assessed the inter-rater agreement between clinicians using the same assessment method,and between different assessment methods using Cohen’s kappa with 95% confidence intervals (95% CI). Qualitativeresearchers observed discussions, took field notes, and reviewed free text comments made by clinicians in a “comment”box in the data collection form. We developed a coding structure and iteratively analyzed qualitative data for emergingthemes regarding the application of each preventability assessment method using NVivo.Results: Among 1356 adverse drug events, a best practice-based approach rated 64.1% (95% CI: 61.5–66.6%) of eventsas preventable, an error-based approach rated 64.3% (95% CI: 61.8–66.9%) of events as preventable, and an algorithm-based approach rated 68.8% (95% CI: 66.1–71.1%) of events as preventable. When applying the same method, the inter-rater agreement between clinicians was 0.53 (95% CI: 0.48–0.59), 0.55 (95%CI: 0.50–0.60) and 0.55 (95% CI: 0.49–0.55) forthe best practice-, error-, and algorithm-based approaches, respectively. The inter-rater agreement between differentassessment methods using consensus ratings for each ranged between 0.88 (95% CI 0.85–0.91) and 0.99(95% CI 0.98–1.00). Compared to a best practice-based assessment, clinicians believed the algorithm-based assessmentwas too rigid. It did not account for the complexities of and variations in clinical practice, and frequently was toopreventability in emergen© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.Woo et al. BMC Medical Research Methodology (2018) 18:160 Page 2 of 8BackgroundAdverse drug events are unintended and harmful eventsrelated to medication use or misuse. They account forone in nine emergency department visits, and remain aleading cause of unplanned admissions and deaths [1–5].Given the burden they place on patients, families, andthe health system, developing and evaluating effectivestrategies for prevention is an international research andhealthcare management priority [6].Studies report highly variable rates of preventable ad-verse drug events [4, 7–11]. Prospective studies indicatethat 28 to 80% of adverse drug events are consideredpreventable [12–19]. These differences may be due tovariations in study design, heath setting, and patientpopulations, and also to the lack of standardized uni-form preventability assessment methods [20–22]. Thislack of consistency in definition and ascertainment ofpreventability undermines our ability to reproduce andcompare results between studies, monitor trends overtime, and meta-analyze research results.The majority of adverse drug event preventability as-sessment methods described in the literature can be cate-gorized into one of three central themes: (1) grounded inthe concept of adherence to best medical practice [4, 22];(2) rooted in error avoidance and identification of modifi-able risk factors [23]; or, (3) application of an explicit algo-rithmic approach [24]. Our main objective was to assessthese three different approaches for determining adversedrug event preventability in patients presenting to theemergency department by measuring the inter-rateragreement of reviewers when applying each method. Sec-ondary objectives were to measure the inter-rater agree-ment between methods by comparing the consensusratings for each method, and to explore their strengthsand weaknesses using qualitative methods.MethodsStudy designThis was a mixed-methods sub-study of a multi-centreretrospective chart review investigating adverse drugevent preventability (Hohl CM, Woo SA, Cragg A, et al:Repeat Adverse Drug Events to Outpatient Medications,submitted).Setting and populationWe reviewed the medical and research records of all pa-tients diagnosed with a suspect adverse drug event inone of three previously conducted prospective studies[2, 3, 5, 25]. The first study enrolled 1591 patientspresenting to the emergency departments of two tertiarycare hospitals, Vancouver General (VGH) and St. Paul’sHospitals (SPH) in Vancouver, British Columbia, Canada,in 2008–2009, and derived tools to identify patients athigh-risk of having an adverse drug event [3]. The secondstudy enrolled 10,807 patients presenting to the emergencydepartments of VGH, Lions Gate Hospital (LGH), an urbancommunity hospital in North Vancouver, British Columbia,and Richmond General Hospital (RGH), an urban commu-nity hospital in Richmond, British Columbia, between 2011and 2013, and evaluated the impact of pharmacist-ledmedication review on health outcomes [5, 25]. The thirdstudy enrolled 1529 patients presenting to the emergencydepartments of VGH, LGH and the Ottawa Civic Hospital(OCH), an urban tertiary care hospital in Ottawa, Ontario,Canada, in 2014–2015, and validated the previously derivedclinical decision rules [2].Inclusion/exclusion criteriaWe have previously described the methodology of theoriginal studies [2, 3, 5, 25]. Briefly, research assistantsused a systematic selection algorithm to select and enrola representative sample of emergency department pa-tients. A clinical pharmacist and physician evaluated allenrolled patients for suspect adverse drug events at thepoint-of-care and documented the events in research andmedical records. All cases in which the clinical pharmacistand physician diagnoses were concordant were consideredfinal. An independent committee adjudicated all cases inwhich assessments were discordant or uncertain.We included all participants from the primary studieswho were diagnosed with a suspect adverse drug eventand excluded patients for whom an alternative diagnosiswas identified during the chart review process. We alsoexcluded patients whose records could not be linked orretrieved from the provincial medication database, andthose with illegible records.Chart review data collection methodsA clinical pharmacist (SW) and a physician (CH, DV orFS) conducted a structured explicit medical and researchrecord review of all patients with suspect adverse drugevents using standardized and piloted data collectionforms (Additional file 1: Appendix A and Additional file 2:Appendix B). All four were blinded to study outcomes.They independently assessed the preventability of eachadverse drug event using two different preventability defi-nitions and one algorithm and categorized events as defin-itely preventable, probably preventable, or not preventableusing each method [4, 22–24]. If preventability ratingsbetween raters were discordant using the same method,reviewers discussed the case until reaching consensus. Incases of remaining uncertainty, a third reviewer adjudi-cated the case.Qualitative data collection methodsResearchers trained in qualitative methods (DP or EB)independently observed reviewers conducting preventabilityassessments and discussing discordant cases to gatherTable 1 Modified Preventability Algorithm [24]Definitely Preventable ADE1. Was there a history of allergy or previousreactions to the drug or drug class?Yes/No/Uncertain-If yes, was the re exposure inappropriate?a* Yes/No/Uncertain2. Was any drug involved inappropriatefor the patient’s clinical condition?aYes/No/Uncertain3. Was the dose, route or frequency ofadministration inappropriate for thepatient’s age, weight or disease state?aYes/No/Uncertain4.Was a toxic serum drug concentration(or laboratory monitoring test)documented?aYes/No/Uncertain5. Was there a known treatment for theADE? (eg. To prevent predictable drugside effects)aYes/No/UncertainProbably Preventable ADE6. Was required therapeutic drugmonitoring or other necessary testsnot performed?bYes/No/Uncertain7. Was a drug interaction involved inthe ADE?bYes/No/Uncertain8. Was poor compliance involved inthe ADE?bYes/No/Uncertain9. Were preventative measures notprescribed or administered to thepatient? (eg. Untreated indication?)Yes/No/Uncertain-If yes, were preventative measures notcontraindicated?b*Yes/No/UncertainAdditional Criteria for ADE Preventability10. Was there an error in ADEdiagnosis that contributed to theevent persisting/getting worse?a*Yes/No/Uncertain11. Was there a delay in ADEdiagnosis that contributed to theevent persisting/getting worse?b*Yes/No/Uncertain12. Was there a failure to act onthe result of monitoring or testing?a*Yes/No/Uncertain13. Were there errors in thetranscription of the culprit drug(s)order?a*Yes/No/Uncertain14. Were there any errors indispensing of the culprit drug(s)order?a*Yes/No/Uncertain15. Were there any errors in theadministration of the culprit drug(s)?a*Yes/No/Uncertain16. Was a superior alternativetreatment available (withoutcontraindication) that is less likelyto cause an ADE?b*Yes/No/Uncertain17. Was there any failure incommunication that contributedto the ADE?a*Yes/No/Uncertain18. Was there any equipmentfailure that contributed to theADE?a*Yes/No/UncertainAutomated preventabilityassessment based on algorithmDefinitely/Probably/NotWoo et al. BMC Medical Research Methodology (2018) 18:160 Page 3 of 8insight into the review process and any challenges thatreviewers experienced, and to identify common areas ofuncertainty or discordance. We scheduled observationsfor ten, two- to three-hour data collection shifts at differ-ent hospital sites, with different observers to account forvariation in patient populations, hospital sites, chartingpractices, and reviewers (e.g., reasoning, habits, assump-tions, inter-personal relationships). We captured reflec-tions about the reviewers’ assessments, and documentedissues that they encountered. We also reviewed all freetext comments made by reviewers in a field provided onour data collection form.DefinitionsAdverse drug eventWe defined an adverse drug event as “harm caused bythe use or inappropriate use of a drug” [26]. Adversedrug events included events categorized as adverse drugreactions (i.e., “noxious and/or unintended responses tomedication which occurred despite appropriate drugdosage for prophylaxis, diagnosis or therapy of the indi-cating medical condition” [27]), drug interactions, supra-and sub-therapeutic doses, and events that occurred dueto non-adherence, being on an ineffective drug, needingan additional drug (i.e., in cases in which there was cleardocumentation at enrolment about an indication for thedrug and lack of contraindication to it), cases due to er-rors, and drug withdrawals [28].Best practice-based preventability assessmentA best practice-based preventability definition was de-veloped by Hallas et al. [22], and subsequently modifiedfor use in prospective studies [3, 4, 8]. It relies on re-viewers’ clinical expertise, and defines adverse drugevents as preventable if events were “avoidable by adher-ing to best medical practice, including inappropriatedrug, dosage, route or frequency of administration of adrug for the patient’s clinical condition, age, weight orrenal function; administration of a drug despite a knownallergy, a previous adverse reaction to, or a drug inter-action; noncompliance; laboratory monitoring not or in-appropriately performed; prescribing or dispensingerrors, or errors in drug administration” [4].Error-based preventability assessmentWe adopted the error-based preventability assessmentdefinition from a definition provided by Health Canada,Canada’s drug regulatory organization. As such, prevent-able events include a medication error, as well as modifi-able risk factors that were not addressed [23].Algorithm-based preventability assessmentSchumock and Thornton developed an explicit algorithmto rate preventability based on prescribing and monitoring*Added to the modified Schumock and Thornton algorithmaIf yes, event is rated as “definitely preventable”bIf yes, event is rated as “probably preventable”appropriateness for a drug (Table 1) [24]. Other authorshave expanded and modified the algorithm to capture druginteractions, known treatments for adverse drug events,and preventative therapies [24, 29]. In this study, we ex-panded the algorithm during our pilot phase to capturemedication-related errors and inappropriate re-exposuresto culprit drugs, which were not captured in any previousiteration (Table 1) [4, 30–33].AnalysisQuantitativeWe used descriptive statistics to describe all enrolled pa-tients and adverse drug events. We reported proportionswith 95% confidence intervals (95%CIs) or means or me-dians with appropriate measures of variance depending onthe data distribution. We used consensus preventabilityratings to calculate the proportion of adverse drug eventsdeemed definitely, probably and not preventable for eachpreventability assessment instrument. We calculated theproportion of preventable events by grouping definitelyand probably preventable events and dividing this numberdefinitely preventable versus probably preventable events.We used the pharmacist’s rating to determine which cri-teria from the algorithm-based preventability assessmentmost frequently contributed to a definitely or probably pre-ventable rating, by calculating the proportion of criteriadeemed positive, over all events.QualitativeWe coded field notes from our observations and freetext comments from reviewers using qualitative dataanalysis software (NVivo Version 11, QSR International,Doncaster, Victoria, Australia). We generated thematicsummaries that were verified and refined with reviewers’perspectives. The purpose of this approach was to gener-ate detailed accounts of the preventability assessmentprocess that resonated with reviewers [34]. We con-cluded observation shifts when they no longer yieldednovel insights. We used qualitative findings to providecontext to our quantitative results.ResultsWoo et al. BMC Medical Research Methodology (2018) 18:160 Page 4 of 8by the number of all events for each instrument. To evalu-ate inter-rater reliability of each method between clinicians,we compared the original pharmacist and physician ratingsfor each adverse drug event, by instrument. To evaluatebetween-instrument reliability, we compared the consensusrating for each adverse drug event between instruments. Ineach case, we calculated kappa scores with 95% confidenceintervals for definitely and probably preventable adversedrug events versus non-preventable events and forFig. 1 Flow diagram of patients through the studyQuantitative resultsWe reviewed the charts of 3202 patients with suspectadverse drug events, of whom 1234 were diagnosed withat least one adverse drug event (Fig. 1). We identified1356 adverse drug events. Reviewers rated 64.1% (95% CI61.5–66.6) of adverse drug events as definitely or probablypreventable by the best practice-based assessment, 64.3%(95% CI 61.8–66.9) by the error-based assessment and68.6% (95% CI 66.1–71.1) by the algorithm-basedTable 2 Consensus preventability ratings of 1356 adverse drug events, by assessment methodPreventability Rating Best Practice-Based [4] Error-Based [23] Algorithm-Based [24]Definitely or probably preventable, n (%, 95 CI) 869 (64.1, 61.5–66.6) 930 (68.5, 66.1–71.1) 873 (64.3, 61.8–66.9)Definitely, n (%, 95 CI) 87 (6.4, 5.1–7.7) 93 (6.9, 5.5–8.2) 613 (45.2, 42.6–47.9)Probably, n (%, 95 CI) 782 (57.7, 55.0–60.3) 780 (57.5, 54.9–60.2) 317 (23.4, 21.1–25.6)Not preventable, n (%, 95 CI) 487 (35.9, 33.4–38.5) 483 (35.6, 33.1–38.2) 426 (31.4, 28.9–33.9)Woo et al. BMC Medical Research Methodology (2018) 18:160 Page 5 of 8assessment (Table 2). The proportion of events rated asdefinitely preventable was highest using the algorithm-based assessment (45.2, 95% CI 42.6–47.9).Agreement between reviewers and assessment methodThe inter-rater agreement of each instrument applied bydifferent clinicians ranged from 0.53 (95% CIs 0.48–0.59)to 0.55 (0.50–0.60) when determining definitely and prob-ably preventable events versus non-preventable events,with overlapping confidence intervals, indicating no sig-nificant differences (Table 3). The inter-rater agreementsbetween clinicians for discerning definitely and probablypreventable events were more modest (Table 4).The consistency in ratings between the three prevent-ability instruments was excellent, with kappa scores ran-ging from 0.88 (95% CI 0.85–0.91) to 0.99 (95% CI 0.98–1.00; Table 5). Inter-instrument agreement was highestwhen comparing the best-practice and error-baseddefinitions.Qualitative resultsReviewers observed that the algorithm-based assessmentwas often unable to account for the realities ofday-to-day medication use and clinical practice andnoted that the algorithm frequently overestimated thepreventability of an event. Reviewers commonly felt thatan assertion of “definitely” preventable was problematicwithout knowing other circumstances of the patient’scare, their interaction with their care providers and thesystem, the frequency of the patient’s monitoring, andprior providers’ decision-making. The algorithm-basedassessment also asked reviewers to determine the pre-ventability of multi-factorial events based on one singlecriterion. The questions most frequently producing “def-initely” preventable ratings discordant with the other as-sessments were re-exposures to the same culpritTable 3 The inter-rater agreement between reviewersdetermining definitely and probably preventable events versusnon-preventable events, by assessment methodPreventability Assessment Inter-rater agreementKappa (95% CIs)n = 1356Best Practice-Based [4] 0.53 (0.48–0.59)Error -Based [23] 0.55 (0.50–0.60)Algorithm-Based [24] 0.55 (0.49–0.60)medication (which may have been appropriate in someclinical scenarios), and the presence of a toxic drugserum concentration or abnormal lab results (which re-viewers felt did not equate to the events being alwaysdefinitely preventable; Table 6). For example, reviewersnoted that some patients on warfarin came to the emer-gency department because their international normalizedratio (used to monitor warfarin therapy) was outside ofits therapeutic range yielding a “definitely” preventableassessment using the algorithm-based assessment, eventhough reviewers may have felt that the abnormal resultsmay have been unpredictable (e.g., the result of an inter-mitted illness) and therefore not definitively preventable.The best practice-based preventability assessmentasked reviewers to rely on their clinical impression ofthe adverse drug event. The notion of what constituted‘best medical practice’ often invoked reviewer discus-sions when adjudicating according to the best-practicedefinition. Reviewers with differing perspectives often at-tributed responsibility for preventable events to differentlevels of or roles within the health system. For example,reviewers argued preventability may be located at thelevel of patients (e.g., patient-level medication adher-ence), providers (e.g., prescribing practices), or the sys-tem (e.g., informational discontinuity between healthcaresectors).Reviewers noted that best practice was not alwaysclear-cut, especially when only having access to the med-ical record at a distance from the event. Definitions ofbest practice varied depending on the professional train-ing and perspective of the reviewer, and reviewers re-ferred to the clinical reality, in which optimizingtreatment could be a matter of trial and error, wheremany competing factors required consideration, andwhere treatment guidelines may differ and even contra-dict themselves. Consequently, reviewers generallyTable 4 The inter-rater agreement between reviewers indetermining definitely versus probably preventable events, byassessment methodPreventability Definition Inter-rater AgreementKappa (95% CIs)Best Practice-Based [4] (n = 869) 0.33 (0.23–0.44)Error -Based [23] (n = 873) 0.30 (0.20–0.40)Algorithm-Based [24](n = 930) 0.40 (0.33–0.46)preferred to rate events as probably, rather thandefinitely preventable, which made the best practiceapproach more favorable than the algorithm-basedtory value was outside a normal range, and how often itThe shortcomings of the algorithm and error-based as-sessments lead reviewers to prefer the best practice ap-proach. The principal weakness of the algorithm-basedassessment was its reductionist approach in determiningpreventability. Observational studies assessing quality ofcare indicate other similar algorithm-based assessmentsfail to distinguish small lapses in specific measures fromthe overall context of adequate care [35–37]. These as-sessments have a limited capacity to capture the complex-ities of care [37]. The reductionist nature of thealgorithm-based assessment also highlights the effect ofhindsight bias [38], which may influence preventabilityratings in light of an adverse drug event. For example, atypically benign drug interaction is considered “probablypreventable” only in the occurrence of an adverse drugevent. Similarly, discrepancies between the best practiceTable 5 Agreement between assessment methods, comparingpreventable versus non-preventable adverse drug events, usingconsensus ratingsConsensus Ratings by Definition Instrumental AgreementKappa (95% CIs)n = 1356Best Practice-Based [4] vs.Algorithm-Based [24]0.88 (0.85–0.91)Algorithm-Based [24] vs.Error -Based [23]0.89 (0.86–0.91)Best Practice-Based [4] vs.Error -Based [23]0.99 (0.98–1.00)YESWoo et al. BMC Medical Research Methodology (2018) 18:160 Page 6 of 8had been monitored.DiscussionWe sought to compare three methods of assessing adversedrug event preventability and explore their strengths andweaknesses. All methods to assess preventability foundapproximately two-thirds of adverse drug events to bepreventable, with excellent overall agreement betweenassessments.Table 6 Questions in the algorithm-based approach for which awithout the option of modifying it to probably preventableapproach.Reviewers noted that they often had to make categori-zations based on the information in the patient’s chart,which could be incomplete, vague, or offered conflictinginformation. For example, if information on blood workmonitoring was unavailable in the medical record systembecause it had been performed in a different institution,reviewers were unable to judge where a relevant labora-Schmock and Thornton algorithm questions (response)Was there a history of allergy or previous reactions to the drug or drug classappropriate? (NO)Was a toxic serum drug concentration (or laboratory monitoring test) documWas any drug involved inappropriate for the patient’s clinical condition? (YESWas the dose, route or frequency of administration inappropriate for the patstate? (YES)Was there a known treatment for the ADE? (e.g., to prevent predictable drugknown treatment not prescribed or administered to the patient? (YES) Was thcontraindicated? (NO)Was there any failure in communication that contributed to the ADE? (YES)Was there a failure to act on the results of monitoring or testing? (YES)Were there any errors in the administration of the culprit drug(s)? (YES)Was there an error in ADE diagnosis that contributed to the event persistingand error-based assessments may be attributed to theerror-based approach being more critical of errors andrisk factors found outside of the realms of best practice, inpart due to hindsight bias [39, 40]. As such, reviewersfound the best practice approach the most appropriate be-cause it allowed reviewers to rate the event according tomultiple factors, and the context of the circumstancesleading up to the event.For rating the preventability of adverse drug events, amore suitable classification may be a binary rating, in whichpreventable events comprise of probably and definitely pre-ventable events versus unlikely or non-preventable events.We found higher agreement between reviewers ratingevents as preventable or not preventable, which suggeststhis binary classification to be more reliable than differenti-ating between definitely and probably preventable events. Abinary rating system would also negate some of the flaws ofthe assessments that we studied, including situations wherereviewers found the algorithm to overestimate the deviationanswer automatically resulted in a definitely preventable ratingFrequency, n (%)n = 508? (YES) Was the re-exposure 148 (29.1)ented? (YES) 144 (28.4)) 78 (15.4)ient’s age, weight or disease 66 (13.0)side effects?) (YES) Was ae known treatment38 (7.5)17 (3.4)11 (2.2)5 (1.0)/getting worse? (YES) 1 (0.2)Woo et al. BMC Medical Research Methodology (2018) 18:160 Page 7 of 8from adequate care, a common observation in other retro-spective studies that used other criteria-based approachesto assess quality of care [36, 37, 41, 42]. By merging defin-itely and probably preventable events, the magnitude ofoverestimation in preventability that occurs due to hind-sight bias may be mitigated.The inter-rater reliability of preventability between cli-nicians was good for all assessment methods when usedin a binary approach. Individual and professional biaseshave previously been implicated in poor inter-rater reli-ability when assessing quality of care, which is consistentwith our observation that individual and professionalbackgrounds oriented reviewers to how they perceivedan event resulting in different ratings [35, 43, 44]. Webelieve there is value in using a second reviewer to ratethe preventability of adverse drug events, and to ask re-viewers to discuss discordant ratings until reaching con-sensus to minimize these biases by reflecting theexperiences and perspectives of multiple clinicians.The primary limitation of our findings is thegeneralizability of their application. The best practice anderror-based assessments provide greater context in theoverall assessment of preventability. However, assessingthe global impression of care leading to an adverse drugevent requires a high level of clinical experience and ex-pertise [45]. In a setting where resources only allow forstudents or care providers with a limited clinical back-ground, the algorithm-based assessment may be appropri-ate if “definitely” and “probably” preventable categoriesare combined to a probably preventable category to createa more global view of preventability, and reflect theremaining uncertainty in any retrospective assessment.ConclusionA standardized assessment of adverse drug event pre-ventability coordinates our efforts to understand areas ofcare where interventions may have a greater likelihood ofreducing adverse drug event burden on patients and thehealth system. We found good agreement of preventabilitybetween different reviewers and assessments, and recom-mend use of a best-practice definition assessed by two ormore clinicians with differing perspectives to produce aconservative preventability estimate. Our findings also in-dicate that there is little merit in categorizing events asdefinitely versus probably preventable; we suggest thatthese categories be collapsed together in future assess-ments of adverse drug event preventability.Additional filesAdditional file 1: Appendix A. Data Collection Form for PreventabilityAssessments. (DOCX 18 kb)Additional file 2: Appendix B. Data Collection Form for PhysicianPreventability Assessments and Consensus Ratings. (DOCX 19 kb)AbbreviationADE: Adverse drug eventAcknowledgementsWe would like to acknowledge Christine Ackerley, Puneet Vashisht and KaneLarson for their contributions to our team.FundingThis study was supported by Health Canada & the Canadian Institutes ofHealth Research (CIHR) through the Drug Safety and Effectiveness Network(DSEN). This federal not-for-profit organization had no role in its design,collection, cleaning, analysis or interpretation of data.Availability of data and materialsThe datasets generated and/or analyzed during the current study are notpublicly available in accordance with our ethics approval but are availablefrom the corresponding author on reasonable request.Authors’ contributionsCH, SW, AC and MW designed data collection form. SW, CH, FS and DVcollected data. AC and MW cleaned and analyzed the quantitative data. DPand EB performed qualitative observations and analysis. All authors contributedto, read, and approved of the final manuscript and take on full responsibilityand accountability for this work.Ethics approval and consent to participateThe Clinical Research Ethics Board at the University of British Columbia andthe institutional review board of all participating hospitals approved thestudy protocol and waived the requirement for informed patient consent.The physicians observed in this study were part of the research team. Theywere aware of the nature of the observations and provided written or verbalconsent.Consent for publicationNot applicableCompeting interestsThe authors declare they have no competing interests.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Author details1Pharmaceutical Sciences, Vancouver General Hospital, 855 West 12thAvenue, Vancouver, BC V5Z 1M9, Canada. 2Department of EmergencyMedicine, University of British Columbia, 855 West 12th Avenue, Vancouver,BC V5Z 1M9, Canada. 3Centre for Clinical Epidemiology and Evaluation,Vancouver Coastal Research Institute, 828 West 10th Ave, Vancouver, BC V5Z1M9, Canada. 4School of Population and Public Health, University of BritishColumbia, 2206 East Mall, Vancouver, BC V6T 1Z9, Canada. 5School ofCommunication, Simon Fraser University, Burnaby, BC, Canada. 6Division ofGeriatrics, Department of Medicine, Vancouver General Hospital, 855 West12th Avenue, Vancouver, BC V5Z 1M9, Canada. 7Emergency Department,Vancouver General Hospital, 855 West 12th Avenue, Vancouver, BC V5Z 1M9,Canada.Received: 25 June 2018 Accepted: 14 November 2018References1. Budnitz DS, Lovegrove MC, Shehab N, Richards CL. Emergencyhospitalizations for adverse drug events in older Americans. N Engl J Med.2011;365:2002–12.2. Hohl CM, Badke K, Zhao A, et al. Prospective validation of clinical criteria toidentify emergency department patients at high risk for adverse drugevents. Acad Emerg Med. 2018;25(9):1015–1026.3. Hohl CM, Yu E, Hunte GS, et al. Clinical decision rules to improve thedetection of adverse drug events in emergency department patients. AcadEmerg Med. 2012;19:640–9.monitoring are needed. Drug Saf. 2010;33:535–8.34. Mays N, Pope C. Rigour and qualitative research. Bmj. 1995;311:109–12.35. Hayward RA, McMahon LF Jr, Bernard AM. Evaluating the care of generalmedicine inpatients: how good is implicit review? Ann Intern Med. 1993;118:550–6.36. Kerr EA, Smith DM, Hogan MM, et al. Building a better quality measure: aresome patients with 'poor quality' actually getting good care? Med Care.2003;41:1173–82.37. Rubenstein LV, Kahn KL, Reinisch EJ, et al. Changes in quality of care for fivediseases measured by implicit review, 1981 to 1986. Jama. 1990;264:1974–9.38. Werth L, Strack F, Förster J. Certainty and uncertainty: the two faces of thehindsight Bias. Organ Behav Hum Decis Process. 2002;87:323–41.39. Henriksen K, Kaplan H. Hindsight bias, outcome knowledge and adaptivelearning. Qual Saf Health Care. 2003;12:ii46–50.40. Reason J. The contribution of latent human failures to the breakdown ofcomplex systems. Philos Trans R Soc Lond Ser B Biol Sci. 1990;327:475–84.41. Owen RR, Thrush CR, Hudson TJ, et al. Using an explicit guideline-basedWoo et al. BMC Medical Research Methodology (2018) 18:160 Page 8 of 84. Zed PJ, Abu-Laban RB, Balen RM, et al. Incidence, severity and preventabilityof medication-related visits to the emergency department: a prospectivestudy. CMAJ. 2008;178:1563–9.5. Hohl CM, Partovi N, Ghement I, et al. Impact of early in-hospital medicationreview by clinical pharmacists on health services utilization. PLoS One. 2017;12.6. The WHO Research Priority Setting Working Group. Global Priorities forResearch in Patient Safety (first edition): World Health Organization; 2008.7. Hohl CM, Dankoff J, Colacone A, Afilalo M. Polypharmacy, adverse drug-related events, and potential adverse drug interactions in elderly patientspresenting to an emergency department. Ann Emerg Med. 2001;38:666–71.8. Hohl CM, Nosyk B, Kuramoto L, et al. Outcomes of emergency departmentpatients presenting with adverse drug events. Ann Emerg Med. 2011;58:270–9.9. Krahenbuhl-Melcher A, Schlienger R, Lampert M, Haschke M, Drewe J,Krahenbuhl S. Drug-related problems in hospitals: a review of the recentliterature. Drug Saf. 2007;30:379–407.10. Thomsen LA, Winterstein AG, Sondergaard B, Haugbolle LS, Melander A.Systematic review of the incidence and characteristics of preventableadverse drug events in ambulatory care. Ann Pharmacother. 2007;41:1411–26.11. WHO Research Priority Setting Working Group. Global priorities for researchin patient safety (first edition): World Health Organization; 2008.12. Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events andpotential adverse drug events. Implications for prevention. ADE preventionstudy group. Jama. 1995;274:29–34.13. Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drugevents in hospitalized adults. J Gen Intern Med. 1993;8:289–94.14. Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance ofadverse drug events in hospital patients. Jama. 1991;266:2847–51.15. Franceschi M, Scarcelli C, Niro V, et al. Prevalence, clinical features andavoidability of adverse drug reactions as cause of admission to a geriatricunit: a prospective study of 1756 patients. Drug Saf. 2008;31:545–56.16. Gholami K, Shalviri G. Factors associated with preventability, predictability,and severity of adverse drug reactions. Ann Pharmacother. 1999;33:236–40.17. Gurwitz JH, Field TS, Avorn J, et al. Incidence and preventability of adversedrug events in nursing homes. Am J Med. 2000;109:87–94.18. Lagnaoui R, Moore N, Fach J, Longy-Boursier M, Begaud B. Adverse drugreactions in a department of systemic diseases-oriented internal medicine:prevalence, incidence, direct costs and avoidability. Eur J Clin Pharmacol.2000;56:181–6.19. Tafreshi MJ, Melby MJ, Kaback KR, Nord TC. Medication-related visits to theemergency department: a prospective study. Ann Pharmacother. 1999;33:1252–7.20. Ferner RE, Aronson JK. Preventability of drug-related harms - part I: asystematic review. Drug Saf. 2010;33:985–94.21. Hakkarainen KM, Andersson Sundell K, Petzold M, Hagg S. Methods forassessing the preventability of adverse drug events: a systematic review.Drug Saf. 2012;35:105–26.22. Hallas J, Harvald B, Gram LF, et al. Drug related hospital admissions: the roleof definitions and intensity of data collection, and the possibility ofprevention. J Intern Med. 1990;228:83–90.23. Health Canada. Adverse reaction information. 2012.24. Schumock GT, Thornton JP. Focusing on the preventability of adverse drugreactions. Hosp Pharm. 1992;27:538.25. Hohl CM, McGrail K, Sobolev B. The effect of pharmacist-led medicationreview in high-risk patients in the emergency department: an evaluationprotocol. CMAJ Open. 2015;3:E103–10.26. Nebeker J, Barach P, Samore M. Clarifying adverse drug events: a clinician’sguide to terminology, documentation, and reporting. Ann Intern Med. 2004;140:795–801.27. World Health Organization. International drug monitoring: the role of thehospital, report of a WHO meeting. Geneva: World Health Organization;1969.28. Verelst S, Jacques J, Van den Heede K, et al. Validation of hospitaladministrative dataset for adverse event screening. Qual Saf Health Care.2010;19:e25.29. Raut LA, Patel P, Patel C, Pawar A. Preventability, predictability andseriousness of adverse drug reactions in among medicine inpatients in ateaching hospital: a prospective observational study. IJPCS. 2012;1:1293–9.criterion and implicit review to assess antipsychotic dosing performance forschizophrenia. Int J Qual Health Care. 2002;14:199–206.42. Wheeler A. Explicit versus implicit review to explore combinationantipsychotic prescribing. J Eval Clin Pract. 2009;15:685–91.43. Duckett S. What problem is being solved: 'Preventability' and the case ofpricing for safety and quality. Asia Pacific J Health Manag. 2016;11:18–21.44. Smith MA, Atherly AJ, Kane RL, Pacala JT. Peer review of the quality of care.Reliability and sources of variability for outcome and process assessments.Jama. 1997;278:1573–8.45. Weingart SN, Davis RB, Palmer RH, et al. Discrepancies between explicit andimplicit review: physician and nurse assessments of complications andquality. Health Serv Res. 2002;37:483–98.30. Ducharme MM, Boothby LA. Analysis of adverse drug reactions forpreventability. Int J Clin Pract. 2007;61:157–61.31. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events inhospitalized patients. Results of the Harvard medical practice study II. NEngl J Med. 1991;324:377–84.32. van der Linden CM, Jansen PA, Grouls RJ, et al. Systems that preventunwanted represcription of drugs withdrawn because of adverse drugevents: a systematic review. Ther Adv Drug Saf. 2013;4:73–90.33. van der Linden CM, Jansen PA, van Marum RJ, Grouls RJ, Korsten EH,Egberts AC. Recurrence of adverse drug reactions following inappropriatere-prescription: better documentation, availability of information and