UBC Faculty Research and Publications

Head to head comparison of the propensity score and the high-dimensional propensity score matching methods Guertin, Jason R; Rahme, Elham; Dormuth, Colin R; LeLorier, Jacques Feb 19, 2016

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-12874_2016_Article_119.pdf [ 796.98kB ]
JSON: 52383-1.0307509.json
JSON-LD: 52383-1.0307509-ld.json
RDF/XML (Pretty): 52383-1.0307509-rdf.xml
RDF/JSON: 52383-1.0307509-rdf.json
Turtle: 52383-1.0307509-turtle.txt
N-Triples: 52383-1.0307509-rdf-ntriples.txt
Original Record: 52383-1.0307509-source.json
Full Text

Full Text

RESEARCH ARTICLE Open AccessHead to head comparison of thepropensity score and the high-dimensionalpropensity score matching methodsJason R. Guertin1,2, Elham Rahme3,4, Colin R. Dormuth5 and Jacques LeLorier6*AbstractBackground: Comparative performance of the traditional propensity score (PS) and high-dimensional propensityscore (hdPS) methods in the adjustment for confounding by indication remains unclear. We aimed to identifywhich method provided the best adjustment for confounding by indication within the context of the risk ofdiabetes among patients exposed to moderate versus high potency statins.Method: A cohort of diabetes-free incident statins users was identified from the Quebec’s publicly funded medico-administrative database (Full Cohort). We created two matched sub-cohorts by matching one patient initiated on alower potency to one patient initiated on a high potency either on patients’ PS or hdPS. Both methods’ performancewere compared by means of the absolute standardized differences (ASDD) regarding relevant characteristics and bymeans of the obtained measures of association.Results: Eight out of the 18 examined characteristics were shown to be unbalanced within the Full Cohort. Althoughmatching on either method achieved balance within all examined characteristic, matching on patients’ hdPS createdthe most balanced sub-cohort. Measures of associations and confidence intervals obtained within the two matchedsub-cohorts overlapped.Conclusion: Although ASDD suggest better matching with hdPS than with PS, measures of association were almostidentical when adjusted for either method. Use of the hdPS method in adjusting for confounding by indication withinfuture studies should be recommended due to its ability to identify confounding variables which may be unknown tothe investigators.Keywords: Confounding by indication, Propensity scores, High-dimensional propensity scoresBackgroundObservational studies provide real world information ondrug use and their potential effect on health outcomesbut are prone to confounding by indication [1–4]. Thetraditional propensity score (PS) method is often used tocontrol for confounding by indication. It represents “theconditional probability of assignment to a particular treat-ment given a vector of observed covariates” and is gener-ally obtained thanks to a logistic regression model [5].The high-dimensional propensity score (hdPS) methodhas recently been proposed and has been rapidly andwidely adopted to address confounding by indication[6, 7]. Unlike the PS method which is limited toinvestigator-specified covariates; the hdPS method alsouses a computerized algorithm to select a large numberof potential confounders contained within the exam-ined database [5, 7].It is of interest to compare the performance of thesetwo methods in controlling for confounding by indica-tion to inform the design of future observational studies.Performance of both methods may be compared usingtwo distinct approaches, 1) by examining the balanceachieved on key potential confounders between sub-cohorts matched on these two scores [4, 8–11], and 2)by comparing the measures of associations obtained* Correspondence: jacques.le.lorier@sympatico.ca6Pharmacoeconomic and Pharmacoepidemiology unit, Research Center ofthe Centre hospitalier de l’Université de Montréal, Pavillon S, 850 St-Denis, 3eétage, Montreal, QC, CanadaFull list of author information is available at the end of the article© 2016 Guertin et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.Guertin et al. BMC Medical Research Methodology  (2016) 16:22 DOI 10.1186/s12874-016-0119-1from the matched sub-cohorts to a gold standard com-parator [7, 12–14].Recently, results of a meta-analysis of randomizedcontrolled trials (RCT) have found that exposure tohigher statin doses might be associated with higher risksof diabetes (Odds ratio [OR] =1.12 [95 % confidence in-tervals (CI) 1.04–1.22]) [15]. Although results obtainedfrom observational studies have been conflicting [16],four out of five published studies found a small in-creased but statistically significant dose-dependent rela-tionship [17–20]. However, it is possible that in thoseobservational studies, patients at higher risk of diabeteswere more likely to be initiated on higher statins doses:a classic example of confounding by indication offeringan excellent opportunity to compare the relative per-formance of these two scores. In this study, we aim tocompare the performance of the PS and hdPS methodsin adjusting for confounding by indication using the twoapproaches defined above.MethodsData sourcesThis study was performed using medico-administrativedatabases from the province of Quebec, Canada. Quebecis the second most populated province in Canada, withmore than 8 million inhabitants [21]. A unique identifi-cation number is assigned to every individual, and alldiagnoses and all health services provided are systemat-ically recorded within the Régie de l’assurance maladiedu Québec (RAMQ) databases. Pharmaceutical claimsare also recorded but only for residents covered by theRAMQ public drug insurance plan. Information was ob-tained from the Quebec physician’s service and claimsdatabases (i.e. RAMQ databases) and the Quebec hospi-talisation databases (i.e., Maintenance et Exploitationdes Données pour l’Étude de la Clientèle Hospitalière[MED-ECHO] databases), which have previously beenvalidated [22–25]. For this study we used three RAMQdatabases (i.e., the Demographic, Medical Services andClaims and Pharmaceutical databases) and three MED-ECHO databases (i.e. the Hospitalisation Descriptions,Diagnoses and Intervention databases). Patient recordswere linked across all databases by use of the uniqueidentification number. The identification numbers wereencrypted to protect patient confidentiality. Access to datawas granted by the Commission d’accès à l’informationand the protocol was approved by the Centre hospitalierde l’Université de Montréal ethics’ committee.Full CohortRAMQ provided us with a cohort of 800,551 incidentstatin users; the date of the first statin dispensation wasdefined as the cohort entry date. Patients were consid-ered to be incident statin users if they did not have aclaim for a statin dispensation in the year prior to thecohort entry date. Eligible patients had: 1) to have beennewly initiated on either simvastatin, lovastatin, prava-statin, fluvastatin, atorvastatin or rosuvastatin betweenJanuary 1st 1998 and December 31st 2010, 2) to be cov-ered by the RAMQ public drug insurance plan for atleast a year prior to the cohort entry date and 3) to be atleast 40 years of age at the cohort entry date. We ex-cluded every patient who, in the year prior or on thecohort entry date: 1) received any other cholesterol low-ering drug dispensation (including niacin, cerivastatin ora combination statin drug); 2) received a dispensationfor drugs used in the treatment of diabetes (WHO ATCA10) [26]; 3) received a diagnosis of diabetes (ICD-9code: 250.x; ICD-10 codes: E10.x – E14.x); 4) were ad-mitted in a long-term care facility or 5) received >1 sta-tin dispensation on the cohort entry date. Patients whomet both inclusion and exclusion criteria were enteredwithin the Full Cohort.Exposure statusPatients were categorized into two categories based onthe dose and relative potency of the first statin dispensa-tion they received [18, 27]. Patients whose first statindispensation was for a daily dose of ≥10 mg of rosuvas-tatin, ≥20 mg of atorvastatin or ≥40 mg of simvastatinformed the high potency group and remaining patientsformed the lower potency group.Outcome statusEvery patient who received either a dispensation of adrug used in the treatment of diabetes (WHO ATC A10)or a diagnosis of diabetes (ICD-9 code: 250.x; ICD-10codes: E10.x – E14.x) within the 2 years following thecohort entry date was defined as a case, all other pa-tients were considered to be diabetes-free.Propensity score methodWe used the same list of variables that were used byDormuth and colleagues to create the PS model [18].This list included the following covariates: patients’ sex,age and poverty level status (yes versus no) at the cohortentry date, year of entry within the cohort (as a categor-ical variable), medical resource utilization variables (≥1hospitalisation, ≥5 outpatient visits, ≥5 distinct drugsdispensed to the patient, all within the year prior to thecohort entry date), drug dispensation variables (dispen-sation of loop diuretics, dispensation of acetaminophen,dispensation of calcium blockers, dispensation of beta-blockers, dispensation of angiotensin receptor blockersand dispensation of angiotensin converting enzyme in-hibitors, all in the year prior to the cohort entry date)and comorbidity variables (hypertension, hypercholester-olemia, myocardial infarction (MI), stroke, peripheralGuertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 2 of 10vascular disease (PVD), congestive heart failure, coron-ary artery bypass graft, and percutaneous coronaryintervention (PCI), all in the year prior to the cohortentry date).Following the selection of the PS model, patients’ PSwere assessed for all patients included within the FullCohort. Trimming was performed and patients locatedwithin non-overlapping regions of the PS distributionwere excluded from the analysis, all other patients wereeligible for inclusion within the Matched PS Sub-Cohort[28]. Lower potency controls were found for patients inthe high potency group using a greedy, nearest neighbor1:1 matching algorithm. Matching occurred if the differ-ence in the logit of PS between nearest neighbors waswithin a caliper width equal to 0.2 times the standarddeviation (SD) of the logit of the PS [29]. Patients se-lected by the matching algorithm were included withinthe Matched PS Sub-Cohort.High-dimensional propensity score methodhdPS were estimated for all patients included in the FullCohort [7]. Detailed description of the hdPS method canbe found elsewhere [7]. Briefly, the construction of thehdPS model involves two processes, 1) investigators se-lect covariates to be forced within the hdPS model (simi-lar to what is done within an investigator-specified PSmodel) and 2) the hdPS algorithm selects an additionallist of covariates measured within the selected data di-mensions based on their multiplicative bias assessmentwhich is then also included within the final hdPS model.Within this study, estimation of patients’ hdPS wereconducted using the default setting of the SAS hdPSmacro v.1 (i.e., top 200 most prevalent variables per datadimensions, top 500 binary empirical covariates basedon multiplicative bias assessment).We structured the data collected from the year priorto the cohort entry date from the following 6 data di-mensions: 1) drugs dispensed in an outpatient setting,2) physician claims codes for inpatient and outpatientprocedures, 3) physician claims for inpatient and out-patient diagnostic codes, 4) specialty of the physicianproviding care, 5) hospitalisation discharge data forinpatient procedure codes and 6) hospitalisation dis-charge data for inpatient diagnostic code.In addition to the 500 variables selected by the defaultoption of the hdPS algorithm [7], we forced the follow-ing covariates within the hdPS model: patients’ sex, ageand poverty level status (yes versus no) at the cohortentry date, year of entry within the cohort (as a categor-ical variable), medical resource utilization variables (≥1hospitalisation, ≥5 outpatient visits, ≥5 distinct drugsdispensed to the patient, all within the year prior to thecohort entry date). These variables were forced in themodel since they could not be selected by the SAS hdPSalgorithm v.1 we were using. Trimming was performedand patients located within non-overlapping regions ofthe hdPS distribution were excluded from the analysis[28], all other patients were eligible for inclusion withinthe Matched hdPS Sub-Cohort. Lower potency controlswere found for patients in the high potency group usinga greedy, nearest neighbor 1:1 matching algorithm.Matching occurred if the difference in the logit of hdPSbetween nearest neighbors was within a caliper widthequal to 0.2 times the SD of the logit of the hdPS [29].Patients selected by the matching algorithm were in-cluded within the Matched hdPS Sub-Cohort.Statistical analysesAbsolute standardized differences (ASDD), defined asthe absolute between group difference over the pooledSD of the two groups, were used to compare patientcharacteristics between patients exposed to a highpotency versus lower potency statin within the FullCohort and both sub-cohorts [4, 8–11]. ASDD < 0.1are generally assumed to indicate good balance be-tween groups [2, 10]. Discrete data are presented inabsolute and relative values (n [%]) and continuousdata are presented as mean (SD). OR (95%CI) of dia-betes occurrence in the high over lower potency sta-tin groups were estimated within the Full Cohort andwithin both matched sub-cohorts; no adjustment be-yond matching was performed.All statistical analyses were conducted with SASversion 9.3 (SAS Institute, Cary, North Carolina).ResultsCharacteristics of the patients included within the FullCohortFigure 1 shows the flow chart of patients included withinthe Full Cohort, the Matched PS Sub-Cohort and theMatched hdPS Sub-Cohort.Baseline characteristics of the Full Cohort are shownin Table 1. Among the 404,129 patients included withinthe Full Cohort, 264,947 patients (65.6 %) were dis-pensed a lower potency statin and 139,182 patients(34.4 %) were dispensed a high potency statin on the co-hort entry date. About half of patients (192,964 [47.8 %])were males and the average age was 65.2 years (SD11.0). Among the 18 examined patient characteristics,eight (44.4 %) were shown to have an ASDD > 0.1 indi-cating the presence of unbalance. History of a PCI(ASDD = 0.30) and history of a MI (ASDD = 0.27), bothin the year prior to the cohort entry date, showed thegreatest degree of imbalance (Table 1). In addition, onsetof diabetes within 2-years follow-up was identified in12,978 patients (3.2 %) of the 404,129 patients includedwithin the Full Cohort.Guertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 3 of 10Characteristics of patients included within the MatchedPS Sub-CohortPS were calculated for all 404,129 patients includedwithin the Full Cohort (kernel density PS curves for allpatients included within the Full Cohort are provided inAdditional file 1). Fifty-five (0.0 %) patients, 33 (0.0 %)lower potency and 22 (0.0 %) high potency, had PS lo-cated within non-overlapping regions and were excludedfrom the analysis. Among the remaining 404,074 pa-tients, we matched 119,857 patients (29.7 %) initiated ona high potency statin to 119,857 patients (29.7 %) initi-ated on a lower potency statin based on their individualPS; selected patients formed the Matched PS Sub-Cohort(Fig. 1). This sub-cohort was comprised of 119,931 malepatients (50.0 %) and the average age was 64.7 years (SD11.2) (Table 2). Balance was obtained for all 18 examinedpatient characteristics (ASDD ranged from 0.002 to0.031 with an average of 0.015).Characteristics of patients included within the MatchedhdPS Sub-CohortThree hundred and one (0.1 %) patients, 54 (0.0 %)lower potency and 247 (0.1 %) high potency, had hdPSlocated within non-overlapping regions and wereexcluded from the analysis (kernel density hdPS curvesfor all patients included within the Full Cohort areprovided in Additional file 2). Among the remaining403,828 patients, we matched 116,014 patients (28.7 %)initiated on a high potency statin to 116,014 patients(28.7 %) initiated on a lower potency statin based ontheir individual hdPS; selected patients formed theMatched hdPS Sub-Cohort (Fig. 1).Patients included within the Matched hdPS Sub-Cohort were on average 64.6 years old (SD 11.2) and116,688 of them were males (50.3 %) (Table 3). Balancewas obtained in all 18 examined patient characteristics,whether or not they were forced within the hdPS model(ASDD ranged from 0.001 to 0.023 with an average of0.008).Performance of the PS and hdPS in adjusting forconfounding by indicationAs mentioned previously, performance of both methodsin adjusting for confounding by indication was tested bytwo distinct approaches, 1) by comparing the ASDD ob-tained within both sub-cohorts and 2) by comparing theadjusted OR of diabetes occurrence in the high overlower potency statin groups estimated by the logisticFig. 1 Patient flow-chart within the study. hdPS, High-dimensional propensity score; PS, Propensity scoreGuertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 4 of 10regression model used within the Full Cohort and bothmatched sub-cohorts. Figure 2 shows the direct compari-son of the ASDD for the examined patient characteristicswithin the Full Cohort, the Matched PS Sub-Cohort andthe Matched hdPS Sub-Cohort. Results indicate that bothmatched sub-cohorts were more balanced than the un-matched Full Cohort. Although the Matched PS Sub-Cohort provided greater balance on three of the 18 exam-ined patient characteristics (MI, hypercholesterolemia,and PVD), overall, the Matched hdPS Sub-Cohortachieved the most balanced sub-cohort (average ASDD=0.008 and average ASDD = 0.015 in the Matched hdPSSub-Cohort and Matched PS Sub-Cohort, respectively).Measures of associations obtained within the Full Cohortand the two matched sub-cohorts indicated that patientsin the high potency group had higher odds of developingdiabetes within 2-years follow-up. Results obtained withinboth sub-cohorts overlapped (OR = 1.10 [95 % CI 1.05–1.15] within the Matched PS Sub-Cohort and OR = 1.13[95 % CI 1.08–1.19] within the Matched hdPS Sub-Cohort)but both were lower than those obtained within the FullCohort (OR = 1.22 [95%CI 1.18–1.27]).DiscussionAs expected, overall patient profiles within the Full Cohortshowed imbalance on many key baseline characteristicssuggesting the presence of confounding by indication.Such results would tend to indicate the presence of biaswithin measures of associations estimated within theFull Cohort if appropriate adjustment were not used inthe analyses.In their original paper, Schneeweiss et al. [7] assessedthe performance of the hdPS method by comparingmeasures of associations adjusted for patients’ hdPS tothe results of a RCT. By showing that the adjusted mea-sures of association were closer to the results of theRCT than the crude measure of association, they showedthat hdPS method had improved the adjustment for con-founding by indication within their study. Performanceof the hdPS method has been assessed by others usingthe same approach and their results also supported itsuse [12–14]. Measures of association obtained withinboth matched sub-cohorts were closer to the null (OR =1.10 [95%CI 1.05–1.15] within the Matched PS Sub-Cohort and OR = 1.13 [95%CI 1.08–1.19] within theMatched hdPS Sub-Cohort) than within the Full Cohort(OR = 1.22 [95%CI 1.18–1.27]). However, since the CIsobtained from both methods overlap with each otherand are as precise, their performance cannot be differen-tiated based solely on this criterion.However, performance based on the level of balanceachieved within matched sub-cohorts does not requireTable 1 Demographic characteristics and comorbidity status of the Full Cohort at baselineLow potency n (%) High potency n (%) Absolute standardized differences264,947 (100.0) 139,182 (100.0)Age, mean (SD)a 65.6 (10.9) 64.5 (11.3) 0.098Male sex 118,262 (44.6) 74,702 (53.7) 0.181At least 5 medical outpatient visits 170,234 (64.3) 77,032 (55.4) 0.182At least 1 hospitalisation 59,591 (22.5) 45,777 (32.9) 0.234Myocardial infarction 15,056 (5.7) 18,899 (13.6) 0.270Stroke 7150 (2.7) 5480 (3.9) 0.069Hypertension 110,508 (41.7) 59,705 (42.9) 0.024Hypercholesterolemia 88,458 (33.4) 47,005 (33.8) 0.008Peripheral vascular disease 5446 (2.1) 3338 (2.4) 0.023Congestive heart failure 11,337 (4.3) 8830 (6.3) 0.092Coronary artery bypass graft 3589 (1.4) 3189 (2.3) 0.070Percutaneous coronary intervention 7742 (2.9) 14,089 (10.1) 0.295Dispensation of loop diuretics 16,612 (6.3) 10,188 (7.3) 0.042Dispensation of calcium blockers 64,569 (24.4) 32,192 (23.1) 0.029Dispensation of beta-blockers 77,669 (29.3) 49,147 (35.3) 0.128Dispensation of angiotensin receptor blockers 35,741 (13.5) 25,325 (18.2) 0.129Dispensation of angiotensin converting enzyme inhibitors 52,563 (19.8) 36,030 (25.9) 0.144At least 5 different drugs dispensed 151,395 (57.1) 84,503 (60.7) 0.073Comorbidity status, drug dispensations and medical utilization rates were assessed in the year prior to the cohort entry date. Absolute standardized differencesare defined as the between group difference over the pooled standard deviation of the two groupsaAt the cohort entry dateGuertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 5 of 10an additional comparator. Based on this second perform-ance criterion, we showed that using both methods cre-ated balanced matched sub-cohorts (i.e. ASDD were < 0.1for all patient characteristics in both matched sub-cohort).When directly comparing both sub-cohorts, use of thehdPS method was favored since 14 out of the 18 examinedpatient characteristics were more balanced within theMatched hdPS Sub-Cohort than within the Matched PSSub-Cohort which should tend to lead to less biased mea-sures of association within the Matched hdPS Sub-Cohort.Seeing as the hdPS model adjusted for more variables thanthe PS model, such a result was to be expected but itneeded to be verified in a situation where we have priorknowledge on which confounders to adjust for. The re-sults we show in this study support the idea that thehdPS method may be used to adjust for confounding byindication, but the possibility that residual confoundingremaining after this adjustment cannot be ruled out.Our study has several strengths. First, we comparedthe PS and hdPS method in a large cohort of incidentstatin users showing substantial imbalance suggestingthe potential for confounding by indication. As such,this provided an excellent situation in which to comparethe performance of both methods.Second, our conclusions favored the hdPS methodwhen our study design should have favored the PSmethod. Although all of the examined covariates wereforced within the PS model, only five investigator-selected covariates were forced within the hdPS model(only demographic, socio-economic and medical re-source utilization variables were forced within the hdPSmodel, all remaining covariates were selected by thehdPS algorithm [n = 500]) [7]. Therefore, the hdPSmethod performance was mainly achieved through theuse of the automated hdPS algorithm and not by investi-gator choice.Our study has also several limitations. First, we com-pared patients on a relatively small number of baselinepatient characteristics. It is possible that the perform-ance observed within the 18 prespecified patient char-acteristics may not be representative of the overallperformance regarding all potential patient characteris-tics. However, these variables were selected because webelieved, like others have [18], that they could lead toconfounding by indication and our results show thatthe hdPS method achieved substantial balance withinall of these even though most were not forced withinthe hdPS model.Second, we defined unbalance as ASDD > 0.1. Al-though this cut-off is frequently used [2, 10, 16], othervalues could have been used as well. Regardless of thecut-off value chosen, our results indicate that the hdPSTable 2 Demographic characteristics and comorbidity status of the Matched PS Sub-Cohort at baselineLow potency n (%) High potency n (%) Absolute standardized differences119,857 (100) 119,857 (100)Age, mean (SD)a 64.6 (11.2) 64.8 (11.2) 0.021Male sex 59,690 (49.8) 60,241 (50.3) 0.009At least 5 medical outpatient visits 68,696 (57.3) 69,017 (57.6) 0.005At least 1 hospitalisation 29,527 (24.6) 31,129 (26.0) 0.031Myocardial infarction 8457 (7.1) 8527 (7.1) 0.002Stroke 3824 (3.2) 4219 (3.5) 0.018Hypertension 49,335 (41.2) 50,719 (42.3) 0.023Hypercholesterolemia 38,760 (32.3) 38,887 (32.4) 0.002Peripheral vascular disease 2374 (2.0) 2691 (2.3) 0.018Congestive heart failure 5412 (4.5) 5852 (4.9) 0.017Coronary artery bypass graft 1756 (1.5) 1988 (1.7) 0.016Percutaneous coronary intervention 5255 (4.4) 4805 (4.0) 0.019Dispensation of loop diuretics 7202 (6.0) 7775 (6.5) 0.020Dispensation of calcium blockers 26,878 (22.4) 27,928 (23.3) 0.021Dispensation of beta-blockers 35,805 (29.9) 36,741 (30.7) 0.017Dispensation of angiotensin receptor blockers 21,228 (17.7) 21,776 (18.2) 0.012Dispensation of angiotensin converting enzyme inhibitors 25,537 (21.3) 26,484 (22.1) 0.019At least 5 different drugs dispensed 69,608 (58.1) 70,087 (58.5) 0.008Comorbidity status, drug dispensations and medical utilization rates were assessed in the year prior to the cohort entry date. Absolute standardized differencesare defined as the between group difference over the pooled standard deviation of the two groupsaAt the cohort entry dateGuertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 6 of 10method outperformed the PS method in achieving themost balanced sub-cohort. Although this added level ofbalance may not eliminate all biases within the observa-tional study (i.e. information bias, unmeasured con-founders, time-varying confounding), it will at least tendto reduce the level of bias caused by these baselinecharacteristics.Third, no mechanism of action by which statins couldcause diabetes has been identified. Although we com-pared both methods using a frequently used exposuredefinition, we cannot claim that this exposure definitionreflects the true mechanism of action by which statinscould cause diabetes. It is possible that the results ob-tained, had we used an exposure definition reflecting thetrue mechanism of action, could have differed fromthose obtained within this study. This also reflects thefact that we do not know what the true measure of asso-ciation between the exposure to statins and diabetes is.As mentioned, traditionally the hdPS method has beenvalidated by comparing the crude and hdPS-adjustedmeasures of association to a gold standard measure butin this case, such a true gold standard is not available.We recognize that this would not have been the casehad we conducted this comparison within an ordinarysimulation study in which the truth would be defined bythe investigators. However, as others have highlighted,[7, 30] the hdPS method cannot be evaluated throughthe use of ordinary simulation studies since its perform-ance depends on the complexity and quantity of dataavailable to the hdPS algorithm, levels of which cannotbe reproduced within a fully artificial setting. In order tocircumvent this issue, Franklin et al. [30] recently pro-posed that performance of the hdPS method comparedto the performance of the PS method be compared usingplasmode simulation studies. Using this framework,Franklin et al. showed that an investigator-independenthdPS method performed nearly as well as a fully speci-fied PS model further supporting the use of the hdPSmethod in situations where little prior knowledge re-garding potential confounding variables is available [30].Such an approach may be validated in future workaimed at further evaluating the performance of the hdPSmethod.Fourth, our results show that hdPS trimming removedslightly more patients than PS trimming (301 vs 55). Al-though this could impact our conclusion regarding thevalue of both methods, its impact should be marginalsince the total number of patients that were trimmed inboth settings remains trivial in comparison to the totalsample size of the Full Cohort.Finally, we only examined the relative performance ofthe PS and hdPS methods within a single context; theTable 3 Demographic characteristics and comorbidity status of the Matched hdPS Sub-Cohort at baselineLow potency n (%) High potency n (%) Absolute standardized differences116,014 (100.0) 116,014 (100.0)Age, mean (SD)a 64.6 (11.2) 64.6 (11.2) 0.002Male sex 58,194 (50.2) 58,494 (50.4) 0.005At least 5 medical outpatient visits 66,453 (57.3) 66,390 (57.2) 0.001At least 1 hospitalisation 28,265 (24.4) 28,604 (24.7) 0.007Myocardial infarction 7558 (6.5) 7995 (6.9) 0.015Stroke 3620 (3.1) 3897 (3.4) 0.013Hypertension 48,268 (41.6) 48,474 (41.8) 0.004Hypercholesterolemia 37,486 (32.3) 37,841 (32.6) 0.007Peripheral vascular disease 2293 (2.0) 2671 (2.3) 0.023Congestive heart failure 5198 (4.5) 5479 (4.7) 0.012Coronary artery bypass graft 1670 (1.4) 1661 (1.4) 0.001Percutaneous coronary intervention 4590 (4.0) 4846 (4.2) 0.011Dispensation of loop diuretics 7139 (6.2) 7256 (6.3) 0.004Dispensation of calcium blockers 26,510 (22.9) 26,716 (23.0) 0.004Dispensation of beta-blockers 33,901 (29.2) 34,389 (29.6) 0.009Dispensation of angiotensin receptor blockers 20,345 (17.5) 20,876 (18.0) 0.012Dispensation of angiotensin converting enzyme inhibitors 24,472 (21.1) 25,289 (21.8) 0.017At least 5 different drugs dispensed 66,600 (57.4) 66,820 (57.6) 0.004Comorbidity status, drug dispensations and medical utilization rates were assessed in the year prior to the cohort entry date. Absolute standardized differencesare defined as the between group difference over the pooled standard deviation of the two groupsaAt the cohort entry dateGuertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 7 of 10results obtained within this study may not be generalizableto other studies focusing on other exposure-disease asso-ciations. Furthermore, we only compared the traditionalPS method estimated using a logistic regression model tohdPS method, while other methods are also available(e.g., classification and regression trees and boostingmethods) [31, 32]. Future work will be needed to com-pare the relative performance of all these differentmethods.ConclusionsIn conclusion, we recommend comparing the PS andhdPS methods by means of their relative ability to selectbalanced sub-cohorts over their adjustment potentialwithin ethiological studies. Although both methods ad-equately adjusted for confounding by indication, wecannot rule out the possibility that the hdPS methodwill be dominant in other contexts since it has thepotential to identify confounders which are unknownto the investigators.Ethics approval and consent to participateAccess to data was granted by the Commission d’accès àl’information and the protocol was approved by theCentre hospitalier de l’Université de Montréal ethics’committee.Consent for publicationNot applicableAdditional filesAdditional file 1: Kernel density curves of the PS distribution withinthe Full Cohort. (TIF 261 kb)Additional file 2: Kernel density curves of the hdPS distributionwithin the Full Cohort. (TIF 460 kb)Fig. 2 Comparison of the level of balance achieved using the absolute standardized differences obtained within the Full Cohort, the Matched PSSub-Cohort and the Matched hdPS Sub-Cohort the examined patient characteristics. ACEI, Angiotensin converting enzyme inhibitors; ARB,Angiotensin receptor blockers; BB, Beta-blockers; CABG, Coronary artery bypass graft; Calc blockers, Calcium blockers; CHF, Congestive heart failure;hdPS, High-dimensional propensity score; PCI, Percutaneous coronary intervention; PS, Propensity score; PVD, Peripheral vascular diseaseGuertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 8 of 10AbbreviationsASDD: absolute standardized differences; CI: confidence interval; hdPS: high-dimensional propensity score; MED-ECHO: Maintenance et Exploitation desDonnées pour l’Étude de la Clientèle Hospitalière; MI: myocardial infarction;OR: odds ratio; PCI: percutaneous coronary intervention; PS: propensity score;PVD: peripheral vascular disease; RAMQ: Régie de l’assurance maladie duQuébec; RCT: randomized controlled trials; SD: standard deviation.Competing interestsJRG has received a CIHR Frederick Banting and Charles Best Doctoral Awardin work related to this paper. JRG has also received a Pfizer Canada Inc. Post-Doctoral Mentoree Award and the 2015-2016 Bernie O’Brien Post-DoctoralFellowship Award for work unrelated to this paper. ER has received fundsand consultancy fees from Janssen Inc. and from Pfizer Canada Inc. not relatedto this paper. JL has received funds and consultancy fees from Astra-Zeneca,Glaxo-Smith-Kline, Janssen Inc., Merck Canada, Novartis, Pfizer Canada Inc., andfrom Sanofi-Avantis not related to this paper. CRD had no conflicts of interestto disclose.Authors’ contributionsJRG contributed to the conception, design, analysis and interpretation of thedata, drafted the first draft of the manuscript and agreed to be accountable forthe all aspects of the work presented. ER contributed to the conception, design,analysis and interpretation of the data, critically revised the manuscript forimportant intellectual content and agreed to be accountable for all aspects ofthe work presented. CD contributed to the design, analysis and interpretationof the data, critically revised the manuscript for important intellectual contentand agreed to be accountable for all aspects of the work presented. JLLcontributed to the conception, design, acquisition, analysis and interpretationof the data, critically revised the manuscript for important intellectual contentand agreed to be accountable for all aspects of the work presented. All authorsread and approved the final manuscript.FundingThis study was financially supported in part by the Canadian Network forObservational Drug Effect Studies (CNODES), a collaborating centre of theDrug Safety and Effectiveness Network (DSEN) that is funded by the CanadianInstitutes of Health Research (Grant Number DSE-111845). This study was madepossible through data sharing agreements between CNODES and the provincialgovernment of Quebec. The opinions, results, and conclusions reported in thispaper are those of the authors. No endorsement by the province is intended orshould be inferred. We would like to also thank the CNODES investigators andcollaborators for their contribution in developing the study protocol evaluatedin this paper.Author details1Department of Clinical Epidemiology and Biostatistics, McMaster University,Hamilton, ON, Canada. 2Programs for Assessment of Technology in Health,St. Joseph’s Healthcare Hamilton, Hamilton, QC, Canada. 3Research Instituteof the McGill University Health Centre, Montreal, QC, Canada. 4Department ofMedicine, McGill University, Montreal, QC, Canada. 5Department ofAnesthesiology, Pharmacology & Therapeutics, University of British Columbia,Vancouver, BC, Canada. 6Pharmacoeconomic and Pharmacoepidemiologyunit, Research Center of the Centre hospitalier de l’Université de Montréal,Pavillon S, 850 St-Denis, 3e étage, Montreal, QC, Canada.Received: 20 November 2015 Accepted: 2 February 2016References1. Groenwold RH, Hak E, Hoes AW. Quantitative assessment of unobservedconfounding is mandatory in nonrandomized intervention studies. J ClinEpidemiol. 2009;62(1):22–8.2. Austin PC. An introduction to propensity score methods for reducing theeffects of confounding in observational studies. Multivar Behav Res. 2011;46(3):399–424.3. Shrank WH, Patrick AR, Brookhart MA. Healthy user and related biases inobservational studies of preventive interventions: a primer for physicians. JGen Intern Med. 2011;26(5):546–50.4. Austin PC. The relative ability of different propensity score methods tobalance measured covariates between treated and untreated subjects inobservational studies. Med Decis Making. 2009;29(6):661–77.5. Rosenbaum PR, Rubin DB. The central role of the propensity score inobservational studies for causal effects. Biometrika. 1983;70(1):41–55.6. Black CM, Tadrous M, Cadarette SM. Diffusion of methodological innovation inpharmacoepidemiology: high-dimensional propensity score co-authorshipnetwork analysis. CAPT; Toronto: J Popul Ther Clin Pharmacol. 2013;21(1):e138.7. Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional propensity score adjustment in studies of treatment effectsusing health care claims data. Epidemiology. 2009;20(4):512–22.8. Belitser SV, Martens EP, Pestman WR, Groenwold RHH, de Boer A, KlungelOH. Measuring balance and model selection in propensity score methods.Pharmacoepidemiol Drug Saf. 2011;20(11):1115–29.9. Austin PC. Balance diagnostics for comparing the distribution of baselinecovariates between treatment groups in propensity-score matched samples.Stat Med. 2009;28(25):3083–107.10. Mamdani M, Sykora K, Li P, Normand SL, Streiner DL, Austin PC, et al.Reader’s guide to critical appraisal of cohort studies: 2. Assessing potentialfor confounding. BMJ. 2005;330(7497):960–2.11. Ali MS, Groenwold RHH, Pestman WR, Belitser SV, Roes KCB, Hoes AW, et al.Propensity score balance measures in pharmacoepidemiology: a simulationstudy. Pharmacoepidemiol Drug Saf. 2014;23(8):802–11.12. Garbe E, Kloss S, Suling M, Pigeot I, Schneeweiss S. High-dimensional versusconventional propensity scores in a comparative effectiveness study ofcoxibs and reduced upper gastrointestinal complications. Eur J ClinPharmacol. 2012;69:549–57.13. Polinski JM, Schneeweiss S, Glynn RJ, Lii J, Rassen JA. Confronting“confounding by health system use” in Medicare Part D: comparativeeffectiveness of propensity score approaches to confounding adjustment.Pharmacoepidemiol Drug Saf. 2012;21:90–8.14. Rassen JA, Glynn RJ, Brookhart MA, Schneeweiss S. Covariate selection inhigh-dimensional propensity score analyses of treatment effects in smallsamples. Am J Epidemiol. 2011;173(12):1404–13.15. Preiss D, Seshasai SR, Welsh P, Murphy SA, Ho JE, Waters DD, et al. Risk ofincident diabetes with intensive-dose compared with moderate-dose statintherapy: a meta-analysis. JAMA. 2011;305(24):2556–64.16. Ko DT, Wijeysundera HC, Jackevicius CA, Yousef A, Wang J, Tu JV. Diabetesand cardiovascular events in older myocardial infarction patients prescribedintensive-dose and moderate-dose statins. Circ Cardiovasc Qual Outcomes.2013;6:315–22.17. Carter AA, Gomes T, Camacho X, Juurlink DN, Shah BR, Mamdani MM. Riskof incident diabetes among patients treated with statins: population basedstudy. BMJ. 2013;346:f2610.18. Dormuth CR, Filion KB, Paterson JM, James MT, Teare GF, Raymond CB, et al.Higher potency statins and the risk of new diabetes: multicentre,observational study of administrative databases. BMJ. 2014;348:g3244.19. Wang K-L, Liu C-J, Chao T-F, Huang C-M, Wu C-H, Chen S-J, et al. Statins, riskof diabetes, and implications on outcomes in the general population. J AmColl Cardiol. 2012;60(14):1231–8.20. Zaharan NL, Williams D, Bennett K. Statins and risk of treated incident diabetesin a primary care population. Br J Clin Pharmacol. 2013;75(4):1118–24.21. Soucy A. Québec Handy Numbers, 2015 Edition. Québec: Institut de lastatistique du Québec; 2015.22. Blais C, Lambert L, Hamel D, Brown K, Rinfret S, Cartier R, et al. Évaluationdes soins et surveillance des maladies cardiovasculaires: Pouvons-nous faireconfiance aux données médico-administratives hospitalières ? Montreal:Institut national d’excellence en santé et en services sociaux (INESSS); 2012.23. Lambert L, Blais C, Hamel D, Brown K, Rinfret S, Cartier R, et al. Evaluation ofcare and surveillance of cardiovascular disease: can we trust medico-administrative hospital data? Can J Cardiol. 2012;28(2):162–8.24. Tamblyn R, Lavoie G, Petrella L, Monette J. The use of prescription claimsdatabases in pharmacoepidemiological research: the accuracy andcomprehensiveness of the prescription claims database in Quebec. J ClinEpidemiol. 1995;48(8):999–1009.25. Tamblyn R, Reid T, Mayo N, McLeod P, Churchill-Smith M. Using medicalservices claims to assess injuries in the elderly: sensitivity of diagnostic andprocedure codes for injury ascertainment. J Clin Epidemiol. 2000;53(2):183–94.26. World Health Organisation Collaborating Centre for Drug StatisticsMethodology. ATC/DDD Index [July 23rd 2014]. Available from:http://www.whocc.no/atc_ddd_index/.Guertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 9 of 1027. Law MR, Wald NJ, Rudnicka AR. Quantifying effect of statins on low densitylipoprotein cholesterol, ischaemic heart disease, and stroke: systematicreview and meta-analysis. BMJ. 2003;326(7404):1423.28. Sturmer T, Rothman KJ, Avorn J, Glynn RJ. Treatment effects in the presenceof unmeasured confounding: dealing with observations in the tails of thepropensity score distribution–a simulation study. Am J Epidemiol. 2010;172(7):843–54.29. Austin PC. Optimal caliper widths for propensity-score matching whenestimating differences in means and differences in proportions inobservational studies. Pharm Stat. 2011;10(2):150–61.30. Franklin JM, Schneeweiss S, Polinski JM, Rassen JA. Plasmode simulation forthe evaluation of pharmacoepidemiologic methods in complex healthcaredatabases. Comput Stat Data Anal. 2014;72:219–26.31. Westreich D, Lessler J, Funk MJ. Propensity score estimation: neural networks,support vector machines, decision trees (CART), and meta-classifiers asalternatives to logistic regression. J Clin Epidemiol. 2010;63(8):826–33.32. Lee BK, Lessler J, Stuart EA. Improving propensity score weighting usingmachine learning. Stat Med. 2010;29(3):337–46.•  We accept pre-submission inquiries •  Our selector tool helps you to find the most relevant journal•  We provide round the clock customer support •  Convenient online submission•  Thorough peer review•  Inclusion in PubMed and all major indexing services •  Maximum visibility for your researchSubmit your manuscript atwww.biomedcentral.com/submitSubmit your next manuscript to BioMed Central and we will help you at every step:Guertin et al. BMC Medical Research Methodology  (2016) 16:22 Page 10 of 10


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items