UBC Faculty Research and Publications

The Ottawa SAH search algorithms : protocol for a multi- centre validation study of primary subarachnoid… English, S. W; McIntyre, L.; Saigle, V.; Chassé, M.; Fergusson, D. A; Turgeon, A. F; Lauzier, F.; Griesdale, D.; Garland, A.; Zarychanski, R.; Algird, A.; van Walraven, C. Sep 15, 2018

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-12874_2018_Article_553.pdf [ 740.51kB ]
JSON: 52383-1.0372078.json
JSON-LD: 52383-1.0372078-ld.json
RDF/XML (Pretty): 52383-1.0372078-rdf.xml
RDF/JSON: 52383-1.0372078-rdf.json
Turtle: 52383-1.0372078-turtle.txt
N-Triples: 52383-1.0372078-rdf-ntriples.txt
Original Record: 52383-1.0372078-source.json
Full Text

Full Text

STUDY PROTOCOL Open AccessThe Ottawa SAH search algorithms:protocol for a multi- centre validation studyof primary subarachnoid hemorrhageprediction models using healthadministrative data (the SAHepi predictionstudy protocol)S. W. English1,2* , L. McIntyre1,2, V. Saigle2, M. Chassé3, D. A. Fergusson2, A. F. Turgeon4,5, F. Lauzier4,5,6,D. Griesdale7, A. Garland8, R. Zarychanski9, A. Algird10 and C. van Walraven2,11AbstractBackground: Conducting prospective epidemiological studies of hospitalized patients with rare diseases likeprimary subarachnoid hemorrhage (pSAH) are difficult due to time and budgetary constraints. Routinely collectedadministrative data could remove these barriers. We derived and validated 3 algorithms to identify hospitalizedpatients with a high probability of pSAH using administrative data. We aim to externally validate their performancein four hospitals across Canada.Methods: Eligible patients include those ≥18 years of age admitted to these centres from January 1, 2012 toDecember 31, 2013. We will include patients whose discharge abstracts contain predictive variables identified in themodels (ICD-10-CA diagnostic codes I60** (subarachnoid hemorrhage), I61** (intracranial hemorrhage), 162**(other nontrauma intracranial hemorrhage), I67** (other cerebrovascular disease), S06** (intracranial injury), G97(other postprocedural nervous system disorder) and CCI procedural codes 1JW51 (occlusion of intracranial vessels),1JE51 (carotid artery inclusion), 3JW10 (intracranial vessel imaging), 3FY20 (CT scan (soft tissue of neck)), and 3OT20(CT scan (abdominal cavity)). The algorithms will be applied to each patient and the diagnosis confirmed via chartreview. We will assess each model’s sensitivity, specificity, negative and positive predictive value across the sites.Discussion: Validating the Ottawa SAH Prediction Algorithms will provide a way to accurately identify large SAHcohorts, thereby furthering research and altering care.Keywords: Administrative health data, Prediction rule, Diagnosis, Subarachnoid hemorrhage* Correspondence: senglish@ohri.ca1Department of Medicine (Critical Care), University of Ottawa, Ottawa, ONK1Y 4E9, Canada2Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa,ON, CanadaFull list of author information is available at the end of the article© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.English et al. BMC Medical Research Methodology  (2018) 18:94 https://doi.org/10.1186/s12874-018-0553-3BackgroundIn the presence of an incomplete disease cohort, the epi-demiologic study of any disease can produce biased re-sults. This is because large and complete cohorts ofpatients are needed to accurately understand populationcharacteristics, prognostic factors, long-term outcomesand disease burden. Unfortunately, prospective studiesof rare diseases are often not feasible due to the exorbi-tant cost and time required to identify and assemble suf-ficient cohorts. Consequently, researchers turn toretrospective study designs that lack completeness or ac-curacy and may lead to inaccurate estimates of diseaseburden, mortality, and healthcare resource utilizationwhich, in turn, could prompt the development of in-appropriate or potentially harmful strategies to care forpeople with these rare diseases.One such rare disease is primary subarachnoidhemorrhage (pSAH), a devastating illness that is pre-dominantly the result of a ruptured arterial aneurysm orarteriovenous malformation (AVM) [1–4]. Most affectedpatients are between the ages of 40 and 60 years [5, 6].The incidence of SAH has varied between studies. A2007 systematic review of prospective studies specificallyexamining SAH incidence demonstrated a range from4.2 (95% CI 3.1 to 5.7) to 22.7 (95% CI 21.9 to 23.5) per100,000 person-years [7]. Although this review includedfour North American studies, they all predated 1990.More recent American and Canadian studies (from the1990s) suggest an incidence of 8.0 to 11.2/100,000 pa-tient years [2, 8]. These studies were retrospective in de-sign and used diagnostic codes for case-ascertainment.Case-fatality rates vary by region and by case-definingmethods. For example, they range from 23 to 62% instudies where diagnostic codes were used for diseaseidentification [2, 9–12].A possible source of these widely variable results is themethods used to retrospectively create these cohorts.Few studies have conducted a detailed examination ofthe validity of using diagnostic codes to identify pSAH(see Table 1). Ellekjaer et al. concluded from their studyusing discharge data of stroke patients that diagnosticcodes should not be used to identify the subtypes ofstroke, including pSAH, because of incidence overesti-mation [13]. In a review by Williams et al., the positivepredictive value (PPV) of diagnostic codes for pSAH wasfound to vary from 64 to 100%, with the higher valuescoming from the smaller studies (all under 30 patients)[8]. Mayo et al. [14] summarized 4 studies [14–17] thathave examined the accuracy of diagnostic coding. Al-though the probability that a patient with the diagnosticcode for pSAH (ICD-9-CM 430) actually had the disease(based on clinical review) ranged from 33 to 100%, theprevalence of pSAH in the populations examined rangedanywhere from 12.5 to < 1%. Further, the potential formissed cases was not accounted for and, thus, neitherthe specificity nor sensitivity can be calculated and onlyreported in two studies [18, 19].There is great potential in using health administra-tive data to study pSAH if cases can be accuratelyidentified. Health administrative data are routinelycollected to create a description of most health careencounters, including hospital admissions, by sum-marizing diagnostic, procedural, demographic, and ad-ministrative information. These summaries, commonlyknown as discharge abstracts, are created for each pa-tient hospitalization. The interventions and diagnosesthe patient received during their course of stay arecaptured with a code (ICD or CCI, respectively).Using this health administrative data to identifycomplete pSAH disease cohorts would significantlyreduce the time and effort needed to complete thesesorts of epidemiological surveys.We have previously derived and validated a predictionmodel to retrospectively identify all hospitalized patientswith a high probability of having suffered a pSAH usingadministrative data at a single centre [20, 21]. We havesince derived two other prediction models (publicationpending). Our published model had a sensitivity of96.5% (95% CI 93.9–98.0), a specificity of 99.8% (95% CI:99.6–99.9%), a positive likelihood ratio (+LR) of 483(95% CI: 254–876), and a positive predictive value of96.8% (95% CI: 94.3–98.3%). Patients with a highlikelihood of pSAH are identified by estimating theprobability a pSAH occurred based on the presenceor absence of a number of variables, including theICD and CCI codes. Externally validating The OttawaSAH Search Algorithms will determine whether theycan be used to identify pSAH patients admitted toother hospitals, thereby permitting the proper studyof this important disease and a better understandingof those affected by it, their prognosis, and long-termoutcomes. Additionally, these algorithms could beused to investigate the type and amount of careadministered to patients with pSAH during theirhospital stay. Here, we propose the methods of exter-nally validating these prediction models to assess theirgeneralizability to justify their further use.MethodsAimWe aim to test the accuracy of The Ottawa SAH SearchAlgorithms in patients ≥18 years of age admitted to fourCanadian tertiary care centres between January 1, 2012and December 31, 2013 (Vancouver General Hospital,Winnipeg Health Sciences Centre, Hamilton HealthSciences Centre, and Hôpital de L’Enfant-Jésus du CHUde Québec-Université Laval).English et al. BMC Medical Research Methodology  (2018) 18:94 Page 2 of 9ObjectivesPrimaryTo describe the performance metrics (sensitivity, specifi-city, positive and negative prediction values, and likeli-hood ratios) of The Ottawa SAH Search Algorithmsusing routinely collected health administrative data from4 Canadian academic tertiary care health centres.Secondarya) To identify and describe differences in performancecharacteristics of the pSAH prediction modelsbetween institutions.b) To identify and describe differences in performancecharacteristics of the pSAH prediction models usingvarying predicted probability thresholds(e.g., thresholds of 50, 75, 90%).c) To create a complete 2-year multi-centre cohort ofhospitalized patients with pSAH.Study hypothesisWe aim to prove our hypothesis that The Ottawa SAHSearch Algorithms will identify all patients with pSAH at4 separate Canadian Academic Hospitals with at least95% sensitivity (within a 2.5% margin of error).Study populationThe population of interest will include patients ≥18 yearsof age who were admitted to one of the study hospitalsbetween January 1, 2012 to December 31, 2013 andwhose discharge abstract contain specific variable values(detailed below).The Ottawa SAH search algorithmsWe will test each of 3 algorithms:Recursive partitioning modelThis original algorithm has been previously publishedand described in detail elsewhere [21]. Essentially thisalgorithm was created using recursive partitioning toidentify the combination of diagnostic and proceduralcodes, as well as other health administrative data, thatmost accurately identified SAH. The model was vali-dated on a separate data set (Table 2 A).SAH prediction point systemFor this model, the entire dataset from A (both deriv-ation and validation data sets) were combined to gener-ate the SAH prediction point system. This was doneusing logistic regression. We initially included allcovariates with a univariable association with SAHp-value of < 0.2, and then removed those that did nothave a p-value < 0.05 during subsequent backward vari-able selection. The final logistic regression model wasconverted into a point system [22]. Each significant vari-able was assigned a point value ranging from − 3 to 6based on the likelihood that the presence of this variablein discharge abstracts was associated with pSAH. Posi-tive points indicate that patients with these variableswere more likely to have pSAH, whereas negative pointsindicate variables rarely seen in the discharge abstractsof patients with pSAH. The sum of these points is tiedto each patient’s overall predictive probability of havingpSAH (Table 2 B).Prevalence-adjusted SAH prediction point systemFor this model, we repeated the methods in B except weused bootstrap methods described by Austin et al. forvariable selection [23]. This method is more restrictiveand resulted in a more parsimonious model. Thesemethods usually help avoid spurious variables to be in-cluded in the model. The final model was converted intoTable 1 Summary of literature describing the accuracy of diagnostic codes for SAHStudy Total sample size Number with ICDcode(s) for SAHProportion of those withcode truly having SAH (PPV)Diagnostic code sensitivity/specificity, % (95% CI)Liu L et al. (1993) [15] 683 14 92.9% Not examinedPhillips SJ et al. (1993) [16] 301 3 33% Not examinedMayo N et al. (1993) [17] 96 1 100% Not examinedMayo N et al. (1993) [17] 3197 247 94.7% Not examinedLeibson CL et al. (1994) [27] 364 11 100% Not examinedBroderick J et al. (1998) [28] Not stated 14 64% Not examinedRosamond WD et el. (1999) [29] 1185 22 86% Not examinedRoumie CL et al. (1998) [30] 231 2 100% Not examinedTirschwell et al. (2002) [18] 206 58 86% Sens = 98 (90–100)Spec = 92 (84–96)Kokotailo RA et al. (2005) [31] 461 (ICD-9)/256 (ICD-10) 51/32 98% (90–99)/91% (77–98) Not examinedJones SA et al. (2014) [19] 4260 56 79% (66–88) Sens = 93 (92–99)CI confidence interval, NR not reported, PPV positive predictive value, Sens sensitivity, Spec SpecificityEnglish et al. BMC Medical Research Methodology  (2018) 18:94 Page 3 of 9Table2OttawaSAHpredictionalgorithmpathwaysandanticipatedpredictiveprobabilityofeachpathwaybasedonourexperienceatTheOttawaHospitalPredictionModelA–RecursivePartitioningModelPredictionModelPathwaysDiagnosticcodes(ICD10CA)Proceduralcodes(CCI)AdmissioncharacteristicsPredictedprobabilityofpSAH(%)SAH(ICD160)ICH(ICD161)OthernontraumaticICH(ICD162)OtherCVD(ICD167)IntracranialInjury(ICDS06)TherapeuticCarotidArteryOcclusion(CCI1JE51)TherapeuticOcclusionIntracranialVessels(CCI1JW51)ExtracranialVesselImaging(CCI3JW10)LOS(days)AdmissionType1+97.62++1003+–≥283.74+–<21005–+76.96–+–41.27–+–Urgent28.68–+–Nonurgent1009––+–10010––+––26.711––+––+10012––+–––15.413––+––28.614––––+–10.315––––+–Urgent2.916––––+–Nonurgent7517––––––0.1PredictionModelB–SAHpredictionpointsystemPredictionVariablesDiagnosticcodes(ICD10CA)Proceduralcodes(CCI)AdmissioncharacteristicsSAH(ICD160)ICH(ICD161)OtherCVD(ICD167)IntracranialInjury(ICDS06)Otherpostproceduralnervoussystemdisorder(ICDG97)TherapeuticOcclusionIntracranialVessels(CCI1JW51)ExtracranialVesselImaging(CCI3JW10)CTscan(softtissueofneck)(3FY20)CTscan(abdominalcavity)(3OT20)LOSAdmissionType(UrgentorEmergent)AdmittedtoICU≤ 2days≥ 3daysPoints6443431−3−20−121ScorePredictedprobabilityofpSAH(%)≤0010.120.1English et al. BMC Medical Research Methodology  (2018) 18:94 Page 4 of 9Table2OttawaSAHpredictionalgorithmpathwaysandanticipatedpredictiveprobabilityofeachpathwaybasedonourexperienceatTheOttawaHospital(Continued)31.442.4517.2633.3763.9894.6997.010100111001298.8≥13100PredictionModelC–Prevalence-adjustedSAHPredictionPointSystemPredictionVariablesDiagnosticcodes(ICD10CA)Proceduralcodes(CCI)SAH(ICD160)ICH(ICD161)TherapeuticOcclusionIntracranialVessels(CCI1JW51)Points211Score50%tilePredictedprobabilityofpSAH(%)(2.5–97.5%ile)00(0–0.01)12.5(0–5.6)220(5–35.3)3100(100–100)4100(100–100)Cellscontaining“+”indicatethattherelevantcodewaspresentandofinteresttothepathway,“-”indicatesthatthecodewasofinteresttothepathway,butwasabsent.Blankcellsindicatethatthecorrespondingcodewasnotconsideredforthatpathway.ICD10CAInternationalStatisticalClassificationofDiseasesandRelatedHealthProblems,10thRevision,Canada,ICDInternationalClassificationofDiseasesandRelatedHealthProblems,CCICanadianClassificationofHealthInterventions,SAHsubarachnoidhemorrhage,ICHintracranialhemorrhage,CVDcerebrovasculardisease,LOSlengthofstay,pSAHprimarysubarachnoidhemorrhageEnglish et al. BMC Medical Research Methodology  (2018) 18:94 Page 5 of 9a point system again [22]. To more accurately predictthe probability of SAS with each possible point score, weused a baseline actual SAH hospital prevalence of 16.7per 10,000 admissions [20]. To do this, we createdbootstrap samples stratified by SAH status using ratiosbetween cases and controls that resulted in SAH preva-lence within each bootstrap sample that ranged ran-domly around 16.7 per 10,000. Within each bootstrapsample (with each having an overall SAH prevalencesimilar to that which would be expected in teachinghospitals similar to ours) we measured the probability ofSAH in each SAH point score. The final expected SAHprobability for each point score was the median of the500 bootstrap samples (Table 2 C).We are testing the three algorithms to identify whichworks best across multiple jurisdictions and health sys-tems, as in the sites included in our sample.Study proceduresThe study objectives will be achieved by following fourmain steps (Fig. 1):1) Identifying the validation cohort2) Applying The Ottawa SAH Search Algorithms3) Establishing “true” disease (pSAH) via chart review4) Testing The Ottawa SAH Search AlgorithmsperformanceStep 1: Identifying validation cohortWe will identify all patients in the validation cohortusing administrative health data from each site. Fromevery hospitalization over the study period, we will cre-ate a sampling frame comprised of a “high probability”and “low probability” sample. We define pSAH as a sub-arachnoid hemorrhage (identified on CT head scan,lumbar puncture or autopsy) that was the result of aruptured aneurysm or AVM (identified on angiographyor autopsy) [20].High probability sample The high probability samplewill be composed of all patients whose discharge ab-stracts contain at least one of the diagnostic or proced-ural predictor variables contained in the predictionmodel and that correspond to the prediction modelpathways outlined in Table 2.Low probability sample This sample will consist of arandom sample of unique patient admissions (max. N =300 per site) who are anticipated to have a very low pre-dicted probability of having pSAH based on The OttawaSAH Search Algorithms. That is to say, patients who ei-ther do not have any of the diagnostic or proceduralpSAH predictor variables (see pathway 17 in Table 2) orthat have one or more of the variables but who do nototherwise meet one of the prediction model pathways.Excluded from this sample will be all admissions topsychiatry or obstetrics as these admissions are ex-tremely unlikely to be related to pSAH and and, thus,could erroneously augment the negative predictive valueof the algorithm. Across 4 centres, we will ensure aminimum of 1200 ‘predicted pSAH negative’ group toallow adequate power for the calculation of external spe-cificity and sensitivity of the model.For each entry in the validation group, we will abstractthe following: admission and discharge date; age at ad-mission; sex; type of admission (emergency, nonurgent,unknown); length of stay; whether they were admitted tothe ICU; and the presence or absence of ICD-10-CAdiagnostic codes I60** (subarachnoid hemorrhage), I61**(intracranial hemorrhage), 162** (other nontraumaintracranial hemorrhage), I67** (other cerebrovasculardisease), S06** (intracranial injury), G97 (other postpro-cedural nervous system disorder) and CCI proceduralcodes 1JW51 (occlusion of intracranial vessels), 1JE51(carotid artery inclusion), 3JW10 (intracranial vessel im-aging), 3FY20 (CT scan (soft tissue of neck)), and3OT20 (CT scan (abdominal cavity)). An anonymizedunique study identifier will be assigned to each entryand tracked in a master log kept at each respective studycentre.Step 2: Applying the Ottawa SAH search algorithmsAt each study centre, all patient identifiers will beremoved from the study dataset and replaced with ade-identified study identification. These datasets will beencrypted and sent to the host centre to be merged intoone large validation set, where it will be password-protected and stored on the hospital server per hospitalpolicy. We will apply The Ottawa SAH SearchAlgorithms to the entire cohort to assign a predictedprobability of pSAH to each patient.Step 3: Establishing “true” disease (pSAH) statusThe ‘true’ disease status will be determined in all screenpositive patients (i.e. those who have a high predictedprobability of having pSAH) and the low probability sam-ple (i.e. 1200 screen negative patients). ‘True’ disease sta-tus will be established by: a) linking to a gold standardreference; or b) primary diagnosis verification (see Fig. 1).Linking to a “gold standard” reference cohortEstablished pSAH cohorts exist at each of the studycentres during the time period under investigation.Three centres previously participated in a cohort studyof patients with primary SAH [24] (Vancouver, Winnipeg,Quebec City) and one has a pre-existing pSAH registry(Hamilton). All admissions in these datasets have alreadybeen confirmed to have pSAH, based on the pSAHEnglish et al. BMC Medical Research Methodology  (2018) 18:94 Page 6 of 9definition outlined in Step 1. We will link patients fromthe study sample dataset to the “Gold Standard” reference.Therefore, we will know the true disease status of all pa-tients for which a link is made. Any patient with truepSAH in the “Gold Standard” reference cohort over thestudy period for which there is no link in the validationcohort will be considered a patient missed by the algo-rithm and count as a false negative. Any entry in the valid-ation cohort that is not linked with an entry in the “GoldStandard” reference cohort will undergo primary chart re-view verification (see b in Fig. 1).Primary diagnosis verification (chart review)All patients for which no linkage occurred in (a) and/or forwhom the true pSAH status is unknown will undergo amedical record review to ascertain their true pSAH status.Step 4: Testing algorithm performance (analysis)Algorithm performance will be measured by generating2 × 2 tables to determine sensitivity, specificity, andpredictive values when testing the expected outcome(see Table 2) compared with the observed outcome.Three classifications for determining expected outcomeswill be tested: one in which the predicted probability ofhaving pSAH is ≥50%, ≥75% and, finally, a more specificcriteria of an observed event rate ≥ 90%. Model perform-ance (accuracy) will be measured by comparing expectedand observed number of patients with pSAH with 2 × 2tables to calculate sensitivity, specificity and predictivevalues with 95% confidence intervals. SAS 9.3 (NorthCarolina USA, OHRI license) will be used to implementthe algorithm and run all analyses. We will assess per-formance across all 4 centres collectively and individually.Sample sizeTo ensure our algorithms have a sensitivity of 95% witha margin of error ≤ 2.5%, a minimum sample of 292patients with pSAH is required. Based on the previouswork [24, 25], we expect that over 500 patients from the4 study centres will have had an admission for pSAHFig. 1 Proposed Study Methods implementing a TOH SAH search algorithmEnglish et al. BMC Medical Research Methodology  (2018) 18:94 Page 7 of 9during the study period. Assuming a Poisson distribu-tion, a minimum of 1000 screen negative cases will needto be reviewed to ensure that the true false negative pro-portion of our models have an upper 95% confidencelimit of ≤5%. The proposed study design accomplishes:1) the necessary sample size to meet our goals; 2) mul-tiple years to demonstrate consistency of the predictionmodel over time; and 3) multiple sites to demonstrategeneralizability across the vast Canadian geography anddifferent health care models between provinces.Ethics and consentApproval from each site’s local research ethics boardsand that of the sponsoring institution will be main-tained throughout the duration of the study. Since nodirect patient contact is required given the retrospect-ive nature of this study, a waived consent model willbe implemented.Proposed timelinesWe estimate a study duration of 16 months from thetime of host centre REB approval (granted in December2016). Participating center identification is complete andfunding has been attained (Ontario Ministry of HealthInnovation Grant through The Ottawa Hospital Aca-demic Medical Organization). We intend to completeanalysis by October 2018.DiscussionFor rare diseases such as pSAH, retrospective identifica-tion of patient cohorts using single diagnostic codes isproblematic when their accuracy is based predominantlyon their positive predictive values. Since accuracy ofpositive predictive value is affected by disease prevalence[26] the reliability of the case ascertainment may belimited. Given the limited available accuracy measures[14–18, 27–29], it is highly plausible that any admittedpatient with an ICD code for SAH has only a 1 in 100chance of actually having the diagnosis [20, 21]. More ac-curate means to reliably identify pSAH retrospectively arenecessary. We intend to test if one or all of the OttawaSAH Prediction Algorithms could serve this purpose.Project influence on the healthcare system and patient careTo accurately study a disease and understand its prog-nosis and outcomes, complete and accurate groups ofpatients with that disease must be identified. Given thesubstantial challenges of accurately and effectively doingthis with rare diseases like pSAH, researchers haveturned to questionably reliable methods of creating suchgroups for study. Externally validating The Ottawa SAHSearch Algorithms, as proposed in this study, will pro-vide researchers (and, in turn, knowledge users) with areliable, easy and cost-effective means of accuratelyidentifying groups of hospitalized patients with pSAH tofurther our understanding of this disease process, itsprognosis, and its outcomes. Such a study will not onlyinfluence patient care and the healthcare systemapproach to this important population, but will also in-fluence our choice of intervention and its measure of ef-fect. This study has the potential to not only directlyimpact the researcher and his/her plight to understandand intervene on this important disease, but also on thefrontline healthcare provider whose understanding,prognostic decisions and interventions are predicated onthe fundamental understanding of pSAH.AbbreviationsAVM: Arteriovenous malformation; CCI: Canadian Classification of HealthInterventions; ICD10CA: International Statistical Classification of Diseases andRelated Health Problems, 10th Revision, Canada; pSAH: primary subarachnoidhemorrhage; SAH: Subarachnoid hemorrhageFundingThis project was supported by the Innovation Fund of the AlternativeFunding Plan for the Academic Health Sciences Centres of Ontario from theOntario Ministry of Health Innovation Fund Grant, administered through TheOttawa Hospital Academic Medical Organization (TOHAMO). The funders hadno role in the study design, access to the data, or writing of this report.Authors’ contributionsSE and CVW conceived the project design and derived the three predictionalgorithms. CVW provided expertise on statistical modeling. The multi-centrevalidation protocol was originally drafted by SE, LM and CVW and revised byMC, DF and the remaining authors. SE and VS wrote the first draft of themanuscript and all other co-authors edited, reviewed and have approved thisfinal version. SE and VS were the team at the coordinating centre whoensured data integrity, communicated with external sites, and analyzed thedata. AT, FL, DG, RZ, and AA, the lead investigators at their respective sites,ensured adherence to the study protocol and provided input on feasibilityon external centres prior to study finalization. There were no predefinedendpoints or data monitoring team. All authors read and approved the finalmanuscript.Dissemination policyOnly study members at the coordinating site, SE, VS, and CVW, will haveaccess to the full dataset once study data has been collected. Results will bedisseminated via publications in peer-reviewed journals. Per the data sharingagreement signed with each of the external sites, this anonymized datasetmay be transferred to one of the lead investigators at each site for futureresearch purposes if the study team agrees to a research proposal submittedby the lead investigator and all relevant ethics approvals have beenobtained. There are no plans to grant the public access to the dataset orstatistical code.Ethics approval and consent to participateWe originally obtained approval for this study from the Ottawa HealthScience Network Research Ethics Board (OHSN) on December 23, 2016. Aftertwo amendments, we obtained approval for this version of the protocol,version 2.5, We obtained approval for this version of the study protocol fromOHSN on May 11, 2017. Ethics approval for this version and all previousversions was also obtained from each of the four external centres (HamiltonIntegrated Research Ethics Board, le Comité de la recherche du CHU deQuébec-Université Laval, University of British Columbia Clinical ResearchEthics Board, and the University of Manitoba Health Research Board). Detailsabout the amendments were communicated to each site via email. We areusing a waived consent model.Consent for publicationNot applicable.English et al. BMC Medical Research Methodology  (2018) 18:94 Page 8 of 9Competing interestsSE has received peer-reviewed funding from Canadian Blood Services andthe Canadian Institutes of Health Research for unrelated work in subarachnoidhemorrhage. RZ receives salary support and operating grants from theCanadian Institutes of Health Research.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Author details1Department of Medicine (Critical Care), University of Ottawa, Ottawa, ONK1Y 4E9, Canada. 2Clinical Epidemiology Program, Ottawa Hospital ResearchInstitute, Ottawa, ON, Canada. 3Department of Medicine, Division of CriticalCare, Centre Hospitalier de l’Université de Montréal, Montreal, QC, Canada.4Centre de recherche du CHU de Québec, Population Health and OptimalHealth Practices Research Unit (Trauma – Emergency – Critical CareMedicine), Université Laval, Québec City, QC, Canada. 5Department ofAnesthesiology and Critical Care Medicine, Division of Critical Care Medicine,Université Laval, Québec City, QC, Canada. 6Centre de recherche du CentreHospitalier de l’Université de Québec, Université Laval, Québec City, QC,Canada. 7Deparment of Anesthesiology, Pharmacology & Therapeutics,University of British Columbia, Vancouver, Canada. 8Department of InternalMedicine, Sections of Critical Care and Respirology, University of Manitoba,Winnipeg, MB, Canada. 9Department of Internal Medicine, Sections of CriticalCare and Hematology/Medical Oncology, University of Manitoba, Winnipeg,MB, Canada. 10Department of Neurosurgy, McMaster University, HamiltonHealth Sciences, Hamilton, ON, Canada. 11Department of Medicine, Universityof Ottawa, Ottawa, Canada.Received: 11 March 2018 Accepted: 31 August 2018References1. Bonita R, Beaglehole R, North JD. Subarachnoid hemorrhage in NewZealand: an epidemiological study. Stroke A J Cereb Circ. 1983;14:342–7.2. Ostbye T, Levy AR, Mayo NE. Hospitalization and case-fatality rates forsubarachnoid hemorrhage in Canada from 1982 through 1991. TheCanadian collaborative study Group of Stroke Hospitalizations. Stroke. 1997;28:793–8.3. Knekt P, Reunanen A, Aho K, Heliövaara M, Rissanen A, Aromaa A, et al. Riskfactors for subarachnoid hemorrhage in a longitudinal population study.J Clin Epidemiol. 1991;44:933–9.4. Van GJ, GJE R. Subarachnoid haemorrhage: diagnosis , causes andmanagement. Brain. 2001;124:249–78.5. Bederson JB, Connolly ES, Batjer HH, Dacey RG, Dion JE, Diringer MN, et al.Guidelines for the management of aneurysmal subarachnoid hemorrhage: astatement for healthcare professionals from a special writing group of thestroke council, American Heart Association. Stroke. Acm. 2009;40:994–1025.6. Connolly ES, Rabinstein AA, Carhuapoma JR, Derdeyn CP, Dion J, HigashidaRT, et al. Guidelines for the management of aneurysmal subarachnoidhemorrhage: a guideline for healthcare professionals from the AmericanHeart Association/American Stroke Association. Stroke. 2012;43:1711–37.7. de Rooij NK, Linn FHH, van der Plas JA, Algra A, Rinkel GJE. Incidence ofsubarachnoid haemorrhage: a systematic review with emphasis on region,age, gender and time trends. J Neurol Neurosurg Psychiatry BMJ Group.2007;78:1365–72.8. Williams GR, Jiang JG, Matchar DB, Samsa GP. Incidence and Occurrence ofTotal (first-ever and recurrent) stroke. Stroke. 1999;30:2523–8.9. Ingall T, Asplund K, Mähönen M, Bonita R. A multinational comparison ofsubarachnoid hemorrhage epidemiology in the WHO MONICA stroke study.Stroke A J Cereb Circ Am Heart Assoc. 2000;31:1054–61.10. Pobereskin LH. Incidence and outcome of subarachnoid haemorrhage: aretrospective population based study. J. Neurol. Neurosurg. Psychiatry.BMJ Group. 2001;70:340–3.11. Sandvei MS, Mathiesen EB, Vatten LJ, Müller TB, Lindekleiv H, Ingebrigtsen T,et al. Incidence and mortality of aneurysmal subarachnoid hemorrhage intwo Norwegian cohorts, 1984-2007. Neurology. 2011;77:1833–9.12. Zimmerman JE, Kramer AA, Knaus WA. Changes in hospital mortality forUnited States intensive care unit admissions from 1988 to 2012. Crit Care.2013;17:R81.13. Ellekjaer H, Holmen J, Krüger O, Terent A. Identification of incident stroke inNorway: hospital discharge data compared with a population-based strokeregister. Stroke. 1999;30:56–60.14. Mayo NE, Chockalingam A, Reeder BA, Phillips S. Surveillance for stroke inCanada. Heal. Reports/stat. Canada, can. Cent. Heal. Inf. - Rapp. Sur la santé/stat. Canada, cent. Can. d’information Sur la santé. 1994;6:62–72.15. Liu L, Reeder B, Shuaib A, Mazagri R. Validity of stroke diagnosis on hospitaldischarge records in Saskatchewan, Canada: implications for strokesurveillance. Cerebrovasc Dis. 1993;9:224–30.16. Phillips S, Cameron K, Chung C. Stroke surveillance revisited. Can. J Cardiol.1993;9(Suppl D):124–5.17. Mayo N, Danys I, Carleton J. Validity of hospital discharge diagnosis ofstroke. Can. J Cardiol. 1993;9(Suppl D):121–4.18. Tirschwell DL. Validating administrative data in stroke research. Stroke.2002;33:2465–70.19. Jones SA, Gottesman RF, Shahar E, Wruck L, Rosamond WD. Validity ofhospital discharge diagnosis codes for stroke: the atherosclerosis risk incommunities study. Stroke. 2014;45:3219–25.20. English SW, McIntyre L, Fergusson D, Turgeon A, Sun C, Dos Santos MP, etal. Enriched administrative data can be used to retrospectively identify allknown cases of primary subarachnoid hemorrhage. J Clin Epidemiol.2016;70:146–54.21. English SW, McIntyre L, Fergusson D, Turgeon A, Dos Santos MP, Lum C, etal. Subarachnoid hemorrhage admissions retrospectively identified using aprediction model. Neurology. 2016;87:1557–64.22. Sullivan LM, Massaro JM, D’Agostino RB. Presentation of multivariate datafor clinical use: The Framingham Study risk score functions. Stat Med. 2004;23:1631–1660.23. Austin PC, Steyerberg EW. Events per variable (EPV) and the relativeperformance of different strategies for estimating the out-of-sample validityof logistic regression models. Stat Methods Med Res. SAGE Publications.2017;26:796–808.24. English SW, Chasse M, Turgeon A, Lauzier F, Griesdale D, Garland A, et al.Aneurysmal subarachnoid hemorrhage and anemia: a Canadian multi-Centre retrospective cohort study. Crit. Care. 2016;20:P337.25. English SW, Chasse M, Turgeon A, Lauzier F, Griesdale D, Garland A, et al.Red blood cell transfusion in aneurysmal subarachnoid hemorrhage – theSahara cohort study. Crit. Care. 2016;20:P336.26. Bewick V, Cheek L, Ball J. Statistics review 13: receiver operatingcharacteristic curves. Crit Care. 2004;8:508–12.27. Leibson CL, Naessens JM, Brown RD, Whisnant JP. Accuracy of hospitaldischarge abstracts for identifying stroke. Stroke. 1994;25:2348–55.28. Broderick J, Brott T, Kothari R, Miller R, Khoury J, Pancioli A, et al. The greaterCincinnati/northern Kentucky stroke study: preliminary first-ever and totalincidence rates of stroke among blacks. Stroke. 1998;29:415–21.29. Rosamond WD, Folsom AR, Chambless LE, Wang CH, McGovern PG, HowardG, et al. Stroke incidence and survival among middle-aged adults: 9-yearfollow-up of the atherosclerosis risk in communities (ARIC) cohort. Stroke.1999;30:736–43.30. Roumie CL, Mitchel E, Gideon PS, Varas-Lorenzo C, Castellsague J, Griffin MR.Validation of ICD-9 codes with a high positive predictive value for incidentstrokes resulting in hospitalization using Medicaid health data.Pharmacoepidemiol Drug Saf. 2008;17(1):20–6.31. Kokotailo R a, Hill MD. Coding of stroke and stroke risk factors usinginternational classification of diseases, revisions 9 and 10. Stroke. 2005;36(8):1776–81.English et al. BMC Medical Research Methodology  (2018) 18:94 Page 9 of 9


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items