Medicine, Faculty of
Science, Faculty of
Population and Public Health (SPPH), School of
Statistics, Department of
DSpace
BMC Medical Research Methodology. 2022 Feb 16;22(1):46
The Author(s)
Hossain, Md. Belal
Mosquera, Lucy
Karim, Mohammad Ehsanul
Background Two-stage least square [2SLS] and two-stage residual inclusion [2SRI] are popularly used instrumental variable (IV) methods to address medication nonadherence in pragmatic trials with point treatment settings. These methods require assumptions, e.g., exclusion restriction, although they are known to handle unmeasured confounding. The newer IV-method, nonparametric causal bound [NPCB], showed promise in reducing uncertainty compared to usual IV-methods. The inverse probability-weighted per-protocol [IP-weighted PP] method is useful in the same setting but requires different assumptions, e.g., no unmeasured confounding. Although all of these methods are aimed to address the same nonadherence problem, comprehensive simulations to compare performances of them are absent in the literature. Methods We performed extensive simulations to compare the performances of the above methods in addressing nonadherence when: (1) exclusion restriction satisfied and no unmeasured confounding, (2) exclusion restriction is met but unmeasured confounding present, and (3) exclusion restriction is violated. Our simulations varied parameters such as, levels of adherence rates, unmeasured confounding, and exclusion restriction violations. Risk differences were estimated, and we compared performances in terms of bias, standard error (SE), mean squared error (MSE), and 95% confidence interval coverage probability. Results For setting (1), 2SLS and 2SRI have small bias and nominal coverage. IP-weighted PP outperforms these IV-methods in terms of smaller MSE but produces high MSE when nonadherence is very high. For setting (2), IP-weighted-PP generally performs poorly compared to 2SLS and 2SRI in term of bias, and both-stages adjusted IV-methods improve precision than naive IV-methods. For setting (3), IV-methods perform worst in all scenarios, and IP-weighted-PP produces unbiased estimates and small MSE when confounders are adjusted. NPCB produces larger uncertainty bound width in almost all scenarios. We also analyze a two-arm trial to estimate vitamin-A supplementation effect on childhood mortality after addressing nonadherence. Conclusions Understanding finite sample characteristics of these methods will guide future researchers in determining suitable analysis strategies. Since assumptions are different and often untestable for IP-weighted PP and IV methods, we suggest analyzing data using both IP-weighted PP and IV approaches in search of a robust conclusion.
Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 https://doi.org/10.1186/s12874-022-01518-8RESEARCH Open AccessAnalysis approaches to addresstreatment nonadherence in pragmatic trialswith point-treatment settings: a simulationstudyMd. Belal Hossain1, Lucy Mosquera2 and Mohammad Ehsanul Karim1,3*AbstractBackground: Two-stage least square [2SLS] and two-stage residual inclusion [2SRI] are popularly used instrumentalvariable (IV) methods to address medication nonadherence in pragmatic trials with point treatment settings. Thesemethods require assumptions, e.g., exclusion restriction, although they are known to handle unmeasuredconfounding. The newer IV-method, nonparametric causal bound [NPCB], showed promise in reducing uncertaintycompared to usual IV-methods. The inverse probability-weighted per-protocol [IP-weighted PP] method is useful inthe same setting but requires different assumptions, e.g., no unmeasured confounding. Although all of thesemethods are aimed to address the same nonadherence problem, comprehensive simulations to compareperformances of them are absent in the literature.Methods: We performed extensive simulations to compare the performances of the above methods in addressingnonadherence when: (1) exclusion restriction satisfied and no unmeasured confounding, (2) exclusion restriction ismet but unmeasured confounding present, and (3) exclusion restriction is violated. Our simulations varied parameterssuch as, levels of adherence rates, unmeasured confounding, and exclusion restriction violations. Risk differences wereestimated, and we compared performances in terms of bias, standard error (SE), mean squared error (MSE), and 95%confidence interval coverage probability.Results: For setting (1), 2SLS and 2SRI have small bias and nominal coverage. IP-weighted PP outperforms theseIV-methods in terms of smaller MSE but produces high MSE when nonadherence is very high. For setting (2),IP-weighted-PP generally performs poorly compared to 2SLS and 2SRI in term of bias, and both-stages adjustedIV-methods improve precision than naive IV-methods. For setting (3), IV-methods perform worst in all scenarios, and(Continued on next page)*Correspondence: ehsan.karim@ubc.ca1School of Population and Public Health, University of British Columbia,Vancouver, BC, Canada3Centre for Health Evaluation and Outcome Sciences, St. Paul’s Hospital,Vancouver, BC, CanadaFull list of author information is available at the end of the article© The Author(s). 2022 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriatecredit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes weremade. The images or other third party material in this article are included in the article’s Creative Commons licence, unlessindicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and yourintended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directlyfrom the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The CreativeCommons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data madeavailable in this article, unless otherwise stated in a credit line to the data.Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 2 of 46(Continued from previous page) IP-weighted-PP produces unbiased estimates and small MSE when confounders areadjusted. NPCB produces larger uncertainty bound width in almost all scenarios. We also analyze a two-arm trial toestimate vitamin-A supplementation effect on childhood mortality after addressing nonadherence.Conclusions: Understanding finite sample characteristics of these methods will guide future researchers indetermining suitable analysis strategies. Since assumptions are different and often untestable for IP-weighted PP andIV methods, we suggest analyzing data using both IP-weighted PP and IV approaches in search of a robust conclusion.Keywords: Pragmatic trials, Nonadherence, Simulation, Unmeasured confoundingBackgroundRandomization is the core principle of clinical trials,that protects against noncomparability between treatmentgroups at baseline. Pragmatic trials are getting popular forexploring the effectiveness of treatments in settings thatmimic real-world clinical practice [1]. In this manuscript,we focus on nonadherence in a point-treatment set-ting, where treatment is assigned at baseline shortly afterrandomization. Although the treatments are randomlyassigned at baseline for pragmatic trials, it is possiblethat some subjects may deviate from the protocol becauseof switching to other treatments, loss to follow-up dueto side-effects, etc. In the presence of treatment nonad-herence, a treatment effect estimate that is agnostic tothe adherence pattern, is less useful for the patients andcaregivers to make a decision about the treatment. More-over, pragmatic trials are often unblinded [2], which oftenintroduces selection bias [3]. Typically, those adherent andnonadherent subjects are different in terms of prognos-tic factors [4], and it is necessary to take into accountthose factors while estimating the treatment effect. Anadditional challenge arises when adherence depends onsubject’s charmeasured during baseline. Inability to adjustfor the unmeasured confounding could bias the treatmenteffect, depending on the method of estimation [4, 5].An intention-to-treat (ITT) analysis is treated as adefault analytic technique to address nonadherence inrandomized and pragmatic trial settings [6]. The baselinerandomization preserved by this analysis, and hence base-line confounding is not a concern [7, 8]. The per-protocol(PP) and as-treated (AT) are two other common meth-ods that are popularly used to address nonadherence [9,10]. In practice, all these naive methods usually producebiased estimates if the nonadherence occurs in a nonran-dom fashion [4]. Typically that is the case, if some of thecharacteristics of the patient act as a confounder, e.g., arepredictive of the nonadherence pattern as well as the out-come of interest. Particularly for PP approaches, whensubjects who deviate from protocol are removed fromthe analysis, comparability of the subjects in both armsensured by the process of randomization is violated. Inthat case, baseline confounder adjusted-PP methods areutilized to address the nonadherence in pragmatic trials, ifthe baseline measurements of the necessary confoundersare available [7, 11–13]. As an alternative, inverse proba-bility (IP)-weighted PP is also used to produce marginalestimates, and that can adjust for the measured con-founders. However, if some of the necessary confoundersare not measured, the PP based methods usually pro-vide biased estimates [14]. In that case, the instrumentalvariable (IV)-based methods can still be used to get theunbiased estimate of the treatment effect [15], which isa known strength of these IV methods. Previous stud-ies used various versions of IV-based methods to addressthe nonadherence in the pragmatic trial settings [10, 16,17]. Two-Stage least squares (2SLS), two-stage residualinclusion (2SRI) are well-known IV-based methods, witha known limitation that they are usually inefficient. NewerIV-based method, nonparametric causal bound (NPCB),is proposed in the literature which uses a partial identifica-tion approach, but provides only bounds rather than pointestimates [18]. This method was touted as a promisingmethod in terms of reducing the levels of uncertainty, butthe original method can not adjust for any confounders.Two previous studies analyzed data from a two-arm ran-domized control trial in northern Sumatra using thisNPCB method [19, 20]. They reported very wide boundsfrom NPCB method. As a motivating example, we usedthe same dataset to explore the performance of two otherIV methods in the same scenario, and from the simula-tions under various parameter spaces, we investigated thepossible reasons of why the NPCB may have producedwide bounds in this study.Both the adjusted PP methods (baseline adjusted andIP-weighted) and IV-based methods (2SLS, 2SRI, andNPCB) aim to deal with adherence adjustment butrequire different assumptions. For example, adjusted PPapproaches assume there is no unmeasured confounding,while the IV-based methods assume the exclusion restric-tion (the IV is associated with the outcome only throughthe treatment) [21–23]. These two assumptions cannotbe empirically verified given the observed data, but theviolation of the assumptions can lead to biased estimates[14, 24]. Let we are interested in estimating the effectof a heart transplant on one-year mortality. No unmea-sured confounding implies that all necessary confounders(e.g., variables causally associated with both heart trans-plant and mortality) are measured. If randomization isHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 3 of 46the instrument, the exclusion restriction assumption sug-gests that randomization should not directly influenceany variable other than whether patients did the hearttransplant versus standard care. Despite the different sta-tistical assumptions required by these two classes of meth-ods dealing with the same nonadherence problem, thecomprehensive comparison of these adherence-adjustedmethods remains largely absent in the literature. More-over, there are two definitions of PP methods, such as (a)censoring those patients if and when deviating from theprotocol and (b) excluding those patients entirely from theanalysis [25]. There exist some recent simulations in theliterature for the first definition [26, 27] but we could notfind much explorations for the second definition. Besides,NPCB being a relatively newly proposed method, finitesample characteristics of its estimates under the abovesettings should be explored.In the present study, we aimed to compare two adjustedPP approaches and three versions of the IV-based meth-ods in the presence of nonadherence. To explore the bene-fits and limitations of the applications of these approaches(along with the naive approaches), and to evaluate howrobust these methods are if the respective assumptionsare violated, we propose comprehensive simulation stud-ies to compare these methods under different settings,such as when (1) exclusion restriction satisfied and nounmeasured confounding, (2) exclusion restriction satis-fied but unmeasured confounding present, and (3) exclu-sion restriction violated. Under these scenarios, we try toidentify which methods are more appropriate to use.MethodsEstimation methodsMethodsWe compared the estimates of the following methods fora binary outcome of interest: naive methods (ITT, naivePP, and naive AT), two PP methods (baseline adjusted PPand IP-weighted PP), and three classes of IV-based meth-ods (2SLS, 2SRI, and NPCB). Some earlier studies [28, 29]used the 2SLS method where confounders are adjustedonly in the first stage of the model. Wang et al. [29] usedthe 2SRI method with confounders adjusted only in thefirst stage of the model. To compare whether there are anyeffects of considering different versions of these IV meth-ods, we have added some variations of all these methods,such as the naive, first-stage adjusted, and both-stagesadjusted of 2SLS and 2SRI approaches. A brief descrip-tion of these models is provided in Table 1, and the fulldescription can be found in Appendix A.AssumptionsThe key assumption for the PP methods and IV-basedmethods is the no unmeasured confounding and exclusionrestriction, respectively [21–23]. The other assumptionsof all the methods are provided in Table 2 and are summa-rized in Appendix A.Confounders, confounding, and exclusion restrictionare the three important concepts to understand the sim-ulation settings we describe below. The traditional defini-tion of a “confounder" is that it meets all three conditions:the variable is (i) causally associated with the outcome,(ii) non-causally or causally associated with the treatment,and (iii) not in the causal pathway between the treatmentand outcome [32]. Consider the causal diagram is shownin Fig. 1(A), where Z is the randomization, A is the treat-ment, and Y is the outcome. Consider two variables in thefigure: L (measured) and U (unmeasured). Both L and Uare (i) causally associated with Y, (ii) causally associatedwith A, and (iii) not in the causal pathway between A andY (i.e., not mediators). L and U are also not colliders (i.e.,not common effect of A and Y ) [33], but confounders inthis example. On the other hand, “confounding" is a fun-damental concept in epidemiological studies to estimatecausal effects. No unmeasured confounding implies thatthe distribution of Y would be the same for the treated anduntreated subjects if both received the same treatment.That means, the treated and untreated subjects are com-parable in terms of any measured and unmeasured factorsif both had been treated [33]. To address confounding,in the causal diagram terminology, we must close allopen “backdoor paths" between A and Y (e.g., adjust forthose variables in the regression model that block theopen backdoor paths) [33, 34]. There could be scenar-ios where more than one variable lies along a backdoorpath. In such a scenario, adjusting for a single confounderon the path can be sufficient to block the backdoor path[33, 34]. For example, in Fig. 1(B), although we have two“confounders" (L and U) by the conventional confounderdefinition (A ← L → Y , and A ← L ← U → Y ), adjust-ing only for L is sufficient to close all open backdoor pathsbetween A and Y. That means, L constitutes the minimalsufficient adjustment set [35]. In such a case, adjustingonly for L can address “confounding" (even though thereremained U unadjusted, which is a confounder by con-ventional definition), and it is certainly possible to get anunbiased effect of A on Y [33], irrespective of whether wecan measure or adjust U. But in Fig. 1(C), adjusting onlyfor L is not sufficient to close all open backdoor pathsbetweenA and Y (since a path is open throughU), indicat-ing the presence of confounding in the A-Y relationship.For more discussion on confounders and confounding,please see [32, 33].The exclusion restriction assumption says the instru-ment is associated with the outcome only through thetreatment. For example, in Figs. 1(B)-(C), the instrumentZ is associated with Y only through its effect on A, mean-ing that the exclusion restriction is met [36]. As shown inFig. 1(D), if the instrument influences Y through a path-Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 4 of 46Table 1 Description of the estimation methods used for dealing with treatment nonadherence in pragmatic trials withpoint-treatment settingsName of the method DescriptionNaïve methodsITT It models the randomization variable (Z) on the outcome (Y) without adjustment for measured confounders L. Thismethod does not consider whether individuals adhered to the treatment [6].Naïve PP It models Z on Y among those subjects who receive the treatment according to the protocol but without adjustmentfor L. This method excludes those subjects who deviated from the protocol.Naïve AT It models the treatment actually received (A) on Y without adjustment for L. This method does not consider whetherindividuals randomized to the treatment groups.Adjusted methodsBaseline-adjusted ITT The same as ITT but it adjusts for L.Baseline-adjusted PP The same as naïve PP but it adjusts for L.IP-weighted PP This method creates inverse probability adherence weights to generate a pseudo population to estimate the treat-ment effect by removing the effect of nonadherence [25]. We used a logistic model that adjusts for L to estimatethe probabilities, and then used the marginal structural model to estimate the parameters of interest. The stabilizedweights were used to prevent from extreme weights [7, 30].IV-methodsNaïve 2SLS The instrument (Z) is modelled to the treatment (A) in the first stage, and then the predicted treatment is modelledto the outcome (Y) in the second stage [31]. There was no adjustment for L in either stage of the model.First-stage adjusted 2SLS The same as naive 2SLS except it adjusts for L in the first stage of the model [28, 29].Both-stages adjusted 2SLS The same as naive 2SLS except it adjusts for L in both stages of the model.Naïve 2SRI The instrument (Z) is modelled to the treatment variable (A) in the first stage, and then the residuals from the firststage and the treatment variable are modelled to the outcome (Y) in the second stage [22]. There was no adjustmentfor L in either stage of the model.First-stage adjusted 2SRI The same as naive 2SRI except it adjusts for L in the first stage of the model [29].Both-stages adjusted 2SRI The same as naive 2SRI except it adjusts for L in both stages of the model [22].NPCB This nonparametric method uses a constrained probability statement to provide bounds on the estimated treatmenteffect rather than a point estimate [18, 19].Note: The 2SLS, 2SRI, and NPCB are IV-based methods. Whether there is any adjustment for covariates, the 2SLS/2SRI are not termed as the naive, first-stage adjusted, orboth-stages adjusted 2SLS/2SRI in the literature. For comparison purposes, we termed these methods as the naive, first-stage adjusted, or both-stages adjusted 2SLS/2SRI;Abbreviations: ITT: intention-to-treat; PP: per-protocol; AT: as-treated; IP-weighted PP: inverse probability weighted per-protocol; 2SLS: two-stage least square; 2SRI: two-stageresidual inclusion model; NPCB: non-parametric causal bound.Table 2 Assumptions of the estimation methods described in Table 1 that are used for addressing treatment nonadherence inpragmatic trials with point-treatment settingsName of the method Key assumption Other assumptionsITT, baseline-adjusted ITT, naïve PP, naïve AT No unmeasured confounding Consistency, no interference, posi-tivity, nonadherence occurred com-pletely at randomBaseline-adjusted PP, IP-weighted PP No unmeasured confounding Consistency, no interference, posi-tivity, correct model specificationNaïve 2SLS, first-stage adjusted 2SLS, both-stages adjusted 2SLS Exclusion restriction Consistency, no interference, pos-itivity, correct model specification,relevance, monotonicityNaïve 2SRI, first-stage adjusted 2SRI, both-stages adjusted 2SRI Exclusion restriction Consistency, no interference, pos-itivity, correct model specification,relevance, monotonicity, linearity ofresidualsNPCB Exclusion restriction Consistency, no interference, posi-tivity, relevance, monotonicityAbbreviations: ITT: intention-to-treat; PP: per-protocol; AT: as-treated; IP-weighted PP: inverse probability weighted per-protocol; 2SLS: two-stage least square; 2SRI: two-stageresidual inclusion model; NPCB: non-parametric causal bound.Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 5 of 46Fig. 1 The causal diagrams representing the simulation mechanisms considered in this study. DAGs 1-3 are subsets of the main DAG and representthe simulation settings 1-3, respectively. Here, Z is the randomization variable, A is the treatment, L is a vector of measured confounders, U isunmeasured confounders, and Y is the outcome. Abbreviations: DAG: directed acyclic graphway other than A, the exclusion restriction assumption isviolated [37].Target parameterThe risk difference (RD) was calculated as the targetparameter of interest. The odds ratio is a widely used asso-ciation measure for a binary outcome. However, since theodds ratio is a non-collapsible measure, we chose not touse this measure [38]. Our effect estimates are marginalfor some methods and conditional over covariates forother methods. For example, the outcome model for thenaive methods, naive and first-stage adjusted 2SLS, andIP-weighted PP do not include any covariates, and theresulting effect estimates are marginal. On the other hand,the baseline-adjusted methods, both-stages adjusted 2SLSand 2SRI, include measured covariates in the outcomemodel, and the resulting effect estimates are conditionalon covariates. Therefore, we chose to use RD, which isa collapsible effect measure that gives equivalent andcomparable marginal or conditional estimates [39].To estimate RD, we can use the binomial regression withan identity link function as the outcome model. How-ever, the binomial model fitting frequently shows conver-gence issues with adjusting for covariates in the modeland requires guessing reasonable starting values which isalmost impossible in the simulation settings. Naimi andWhitcomb [40] suggested to use the generalized linearmodel (GLM)with a Gaussian (i.e., normal) distribution insuch scenarios as an alternative to the binomial regressioneven if the outcome variable is binary. The authors sug-gested using the robust sandwich standard error for validstandard errors in this setting. In our study, we followedthe GLM method described by Naimi and Whitcomb asthe outcome model for all the estimators other than theIP-weighted PP [40]. For the IP-weighted PP estimator,since we have to incorporate IP weights in the outcomemodel, we used the weighted GLM for the Gaussian familywith an identity link function and robust standard error.Simulation setupWe followed two established simulation structuresdescribed by Young et al. [26], and by Palmer et al. [19]with modifications. Under both structures, we considerpoint-treatment settings (treatment is administered onlyonce) and each individual is assigned to the treatment(Z = 1) or to standard care (Z = 0). We considered threedata generating mechanisms, where A is the treatment, Yis the binary outcome of interest, L = (L1, L2) is a vectorof measured baseline confounders, and U is unmeasuredconfounders.We considered one set of variables in the fol-lowing causal order for all three settings: Z,U , L1, L2,A,Y .One such example of a real-world scenario could beHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 6 of 46estimating the effect of the seasonal influenza vaccine onflu in 6 months among adults. The three data-generatingmechanisms considered in this study are different thaneach other in terms of the underlying assumption of mea-sured and unmeasured confounding and exclusion restric-tion. Figure 1(A) shows the overall directed acyclic graph(DAG) of the data generating process. For simulation set-tings 1-3 described below, we simulated data from thefollowing algorithms (Eqn. (1)):Z ∼ Bernoulli(0.5)U ∼ fUL1 ∼ Normal(λ0 + λ1U , σ)L2 ∼ Bernoulli(pL2)A ∼ Bernoulli(pA)Y ∼ Bernoulli(pY ).(1)The description of each three generating processes withparameterization is described below in detail.Setting 1: exclusion restriction satisfied and no unmeasuredconfoundingYoung et al. [26] described the simulation structure forlongitudinal setting, whereas we used a simplified ver-sion of this simulation by considering the point treatmentsetting and having only baseline covariate measurements.Data for setting 1 are generated from our example of theinfluenza vaccine effect on flu. Z and A are the random-ization and influenza vaccine status, receptively. In thisexample, physical activity (L1) and illness (L2) are twomeasured confounders, and smoking is an unmeasuredconfounder (U). Less physical activity can cause chronicillness, and thus, an arrow from L1 to L2 in our examplecould be justified [41]. Besides, smoking typically reducesthe level of physical activity (U → L1) [42], increases ill-ness (U → L2) [43], and increases the risk of flu (U → Y )[44]. In addition, people with less physical activity andhaving chronic illness could have influenza vaccine hes-itancy and are less likely to take the vaccine (L → A)[45]. DAG 1 in Fig. 1(B) is a subset of Fig. 1(A), thatshows the data generating process for setting 1. The exclu-sion restriction assumption is satisfied for this simulationframework. To get an unbiased effect of the influenzavaccine on flu, we must block all open backdoor pathsbetween the vaccine and flu. As shown in Fig. 1(B), eventhough we have an unmeasured confounder (e.g., smok-ing), adjusting for measured confounders can be sufficientto block the backdoor path between vaccine status and flustatus. Whether or not we measure or adjust for smokingdoes not impact our ability to obtain unbiased treatmenteffect estimates of the influenza vaccine on flu. In otherwords, adjusting only for measured confounders usingan appropriate method (e.g., IP-weighted PP, 2SLS, 2SRI)should give us an unbiased effect estimate of the vaccineon flu. Hence, this Setting 1 is a fair scenario for both PPand IV-based methods where PP methods that adjust forL and the IV-based methods should all be unbiased.Using the Eqn. (1), the following are considered to sim-ulate the data for settings 1: fU = Uniform(0,1), λ0 =0, λ1 = 6, σ = 2, pL2 = logit(−5 + 3U + 1.25L1), pA =logit(α0+α1Z+α2L1+α3L2+α4U), and pY = logit(θ0+θ1A + θ2L1 + θ3L2 + θ4U + θ5Z). Here, α0 is associatedwith the nonadherence rate; α1 = 0.6 is the coefficientassociated with randomization; α2 = 0.4 and α3 = 0.35are coefficients associated with the measured covariates;α4 = 0. Also, θ0 is associated with the event rate; θ1 = 0 isthe null treatment effect (and thus no arrow from A to Yin DAG 1); θ2 = 0; θ3 = 0; θ4 determines the magnitudesof unmeasured confounders, and θ5 = 0 indicates theexclusion restriction is satisfied in this setting. Under dif-ferent choices of α0, we considered six levels of deviationsfrom adherence in each arm: 10, 20, 40, 60, 80, and 90%.The parameter choices are summarized in Table 3(A). Foreach of the nonadherence scenarios, we considered twosets of θ4 values, making a total of 12 scenarios. We set θ4= 8 and 0.5 for strong and weak unmeasured confounders,respectively. The full list of parameters considered for all12 scenarios for setting 1 is shown in Appendix D Table 5.Setting 2: exclusion restriction satisfied, unmeasuredconfounding presentPalmer et al. [19] described the simulation structure forpoint-treatment settings where the exclusion restrictionassumption of the IV assumption is violated. We slightlymodified the structure and considered two versions ofthis simulation: the exclusion restriction of IV assumptionis not violated but there exists unmeasured confounding(Fig. 1(C); DAG 2), and the exclusion restriction assump-tion is violated (Setting 3; described later). Recall ourexample on estimating the influenza vaccine effect on fluwhere information on smoking is unmeasured. As shownin DAG 2, smoking status is a barrier to receive theinfluenza vaccine (U → A) [45], but it increases the riskof flu (U → Y ) [44]. Given that the study is a randomizedtrial, the exclusion restriction could be satisfied in this set-ting, but there could be unmeasured confounding. Unlikesetting 1, adjusting only for measured confounders (e.g.,physical activity and illness) may not be sufficient to getan unbiased effect estimate of the vaccine on flu using thePP methods [14].Using the Eqn. (1), the following are considered to sim-ulate the data for settings 2: fU = Bernoulli(0.5), λ0 =3, λ1 = 0, σ = 0.5, pL2 = logit(−3.5 + 0.6L1), pA =α0+α1Z+α2L1+α3L2+α4U , and pY = θ0+θ1A+θ2L1+θ3L2 + θ4U + θ5Z. We considered α1 = 0.25, α2 = 0.02,α3 = 0.04, and α4 = 0.05. Under different choices of α0,we considered six levels of nonadherence: 10, 20, 40, 60,80, and 90%. The parameter choices are summarized inHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 7 of 46Table 3 The nonadherence rates per arm for different choices of α0 for simulation settings 1-3 with (α1,α2,α3,α4) = (0.6, 0.4, 0.35, 0)for setting 1, (0.25, 0.02, 0.04, 0.05) for setting 2, and (0, 0.01, 0.04, 0) for setting 3Scenario ArmA. Setting 1 B. Setting 2 C. Setting 3α0 Nonadherence α0 Nonadherence α0 Nonadherence1Z = 1 0.72 10 0.55 11 0.86 10Z = 0 -4.06 10 0.02 11 0.06 102Z = 1 -0.23 20 0.46 20 0.76 20Z = 0 -3.14 20 0.12 21 0.16 203Z = 1 -1.47 40 0.25 41 0.56 41Z = 0 -1.92 40 0.32 41 0.36 404Z = 1 -2.52 60 0.05 61 0.36 60Z = 0 -0.85 60 0.52 61 0.57 605Z = 1 -3.76 80 -0.15 81 0.16 80Z = 0 0.39 80 0.70 80 0.77 806Z = 1 -4.72 90 -0.25 91 0.06 90Z = 0 1.35 90 0.80 89 0.86 90Table 3(B). For each of the six nonadherence scenarios, weconsidered five scenarios of the treatment effect of interest(θ1) and two versions of confounding (θ4), making a totalof 60 scenarios. We set the treatment effect of interest asθ1 = {−0.2,−0.05, 0, 0.05, 0.2} and confounding as θ4 =0.05 and 0.4 respectively for weak and strong confound-ing. In this setting, the exclusion restriction is satisfied sothat θ5 = 0. The complete list of parameters consideredfor all 60 scenarios for setting 2 is shown in Appendix DTable 6.Setting 3: exclusion restriction violatedIf the exclusion restriction is met, Z affects the outcomeY only through the treatment A [36]. For example, ifZ represents randomization in a double-blind random-ized controlled trial, Z → A → Y is expected and theexclusion restriction assumption usually met. However,as discussed by Brookhart et al. [37], if the instrument Zrepresents physician’s prescribing preference, there couldbe a direct effect of Z on Y because physicians tend toprescribe selective drugs based on their experience aboutsafety and efficacy of drugs. Recall our motivating exam-ple of exploring the effect of the seasonal influenza vaccineon flu. Let Z be the physician’s prescribing preferenceon whether an individual is recommended the seasonalinfluenza vaccine. Based on the experience, the physicianscould prescribe the vaccine only to subjects with a higherrisk of the flu [46]. Therefore, in addition to the influenceof Z on Y through A, Z could directly influence Y. Thedata generating mechanism for this setting is shown inFig. 1(D; DAG 3). Since the exclusion restriction assump-tion is violated in this setting, the IV methods (e.g., 2SLSand 2SRI) are expected to produce biased estimates nomatter whether we adjust for measured confounders (e.g.,physical activity and illness) [24].Using the Eqn. (1), the following are considered to sim-ulate the data for settings 3: fU = Bernoulli(0.5), λ0 =3, λ1 = 0.05, σ = 0.5, pL2 = logit(−3.5 + 0.6L1 +0.1U), pA = α0 + α1Z + α2L1 + α3L1 + α4U , and pY =θ0 + θ1A + θ2L1 + θ3L2 + θ4U + θ5Z. We consideredα1 = 0, α2 = 0.01, α3 = 0.04, and α4 = 0. Thesame as before, there are six levels of nonadherence fordifferent choices of α0: 10, 20, 40, 60, 80, and 90%. Theparameter choices are summarized in Table 3(C). For eachnonadherence scenario, we set two versions of θ1 and twoversions of θ5, making a total of 24 scenarios. We set θ1= 0 for the null treatment effect and 0.2 for the non-nulleffect. θ5 determines the severity of the exclusion restric-tion assumption violation. A small θ5 value indicates aminor violation of the exclusion restriction, and a largevalue indicates a severe violation. We set these θ5 val-ues as 0.05 and 0.2 in this study. The complete list ofparameters for these 24 scenarios is shown in Appendix DTable 7.SimulationSample size and iterationsTo assess the performance of the estimation methods, wegenerated 2,000 samples (approximately 1,000 per arm)with 1000 iterations for each scenario based on the aboveDAGs. R version 4.1.0 was used to perform the analysis.PerformancemetricsWe assessed the performance of the models through sev-eral measures, such as bias, standard error (SE), meansquared error (MSE), and 95% confidence interval (CI)coverage probability. This allows us to compare the per-formance of different methods in terms of accuracy, pre-cision, and coverage. The following definitions are used todefine these measures [47]:Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 8 of 46Bias = 1nsimnsim∑i=1βˆi − β = ˆ¯β − βSE =√√√√ 1nsim − 1nsim∑i=1(βˆi − ˆ¯β)2MSE = 1nsimnsim∑i=1(βˆi − β)2Coverage = 1nsimnsim∑i=11(βˆlower,i ≤ β ≤ βˆupper,i),where β (i.e., RD) is the true treatment effect of interest,nsim is number of iterations (1000 in our case), βˆi is theestimated RD in the ith iteration, and βˆlower,i and βˆupper,iare the lower and upper 95% bound of the RD in the ithiteration, respectively.ResultsWe presented the results for the baseline-adjusted PP, IP-weighted PP, naive 2SLS, both-stages adjusted 2SLS, naive2SRI, and both-stages adjusted 2SRI methods in the maintext, as they are the main focus of the study. The resultsfor all 14 methods described in Table 1 can be found inAppendix C.Setting 1: exclusion restriction satisfied and nounmeasured confoundingBiasFigure 2 shows the bias versus an incremental rate ofnonadherence using DAG 1 (null treatment effect) forbaseline-adjusted PP, IP-weighted PP, naive and both-stages adjusted 2SLS and 2SRI methods. Under the sce-nario of weak unmeasured confounders, all methods pro-duce small bias. In the presence of strong unmeasuredconfounders, the 2SLS and 2SRI estimates have nominalbiases (range 0 to 0.02). The baseline-adjusted PP andIP-weighted PP estimates are approximately unbiased,but these estimates are biased (the bias is approximately0.025) beyond 80% of nonadherence. The bias versus non-adherence comparison for all methods considered in thisstudy is shown in Appendix C Fig. 6.SE, MSE, coverage, and bounds of NPCBThe 2SLS and 2SRI methods produce higher SE than PPmethods (Appendix C Fig. 7). The baseline-adjusted PPand IP-weighted PP also produce high SE beyond 40%nonadherence. Also, the IP-weighted PP performs theworst beyond 60% nonadherence, producing very high SE(e.g., approximately twice than both-stages adjusted 2SLSand 2SRI methods). The pattern and amount of SE arethe same for weak and strong unmeasured confoundersscenarios.Under the null scenario with weak unmeasured con-founders, all methods produce small MSE and some shareapproximately the same line (Appendix C Fig. 8). Thenaive and both-stages adjusted 2SLS and 2SRI meth-ods have slightly higher MSE compared to the baseline-adjusted PP and IP-weighted PP when the nonadherenceFig. 2 Bias versus the nonadherence rate for simulation setting 1. The 2SLS and 2SRI share the same line, and both-stages adjusted 2SLS and 2SRIshare the same line as they produce the same amount of bias. Abbreviations: PP: per-protocol, IP-weighted PP: inverse probability-weightedper-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 9 of 46rate is ≤ 60%. For example, MSEs are close to zero for thebaseline-adjusted PP and IP-weighted PP methods, butabout 0.015 for naive and both-stages adjusted 2SLS and2SRI methods. On the other hand, the baseline-adjustedPP and IP-weighted PP have more than twice MSE thannaive and both-stages adjusted 2SLS and 2SRI methodsbeyond 60% nonadherence. Also, the both-stages adjusted2SLS and 2SRI produce comparatively smaller MSE thanthe naive 2SLS and 2SRI methods.The IP-weighted PP produces nominal 95% coveragefor less than 80% nonadherence, but this method pro-duces low coverage for≥ 80% nonadherence (Appendix CFig. 9). On the other hand, the naive and both-stagesadjusted 2SLS and 2SRI produce nominal 95% coverageregardless of weak or strong unmeasured confounders.The bounds of the NPCB method using DAG 1 areshown in Appendix C Fig. 10. The NPCB method pro-duces very wide bounds regardless of weak or strongunmeasured confounders. In contrast, the width ofbounds is small for 10% and 90% nonadherence, and highfor 20% to 80% nonadherence.Setting 2: exclusion restriction satisfied, unmeasuredconfounding presentVarying the effect of nonadherenceWepresented bias, SE,MSE, and 95% coverage probabilityfor different nonadherence rates.We presented the resultsfor a null (RD = 0) and non-null (RD = 0.2) treatmenteffect scenarios.Bias Under the scenario of weak unmeasured confound-ing, all methods produce small bias (Fig. 3). In the pres-ence of strong unmeasured confounding, the baseline-adjusted PP and IP-weighted PP produce biased estimates.The amount of bias (range from 0.02 to 0.10) is approx-imately the same for the null and non-null treatmenteffect scenario. In contrast, the naive 2SLS and 2SRI, andboth-stage adjusted 2SLS and 2SRI methods consistentlyproduce unbiased estimates under the null or non-nulland weak or strong confounding scenarios. The bias ver-sus nonadherence rate for all methods using DAG 2 isshown in Appendix C Fig. 11.SE, MSE, coverage, and bounds of NPCB Appendix CFigs. 12 and 13 show the SE and MSE versus nonad-herence using DAG 2, respectively. The baseline-adjustedPP and IP-weighted PP produce low coverage for strongconfounding scenarios, and these two methods per-form poorly beyond 60% nonadherence (Appendix CFig. 14). On the other hand, the naive and both-stagesadjusted 2SLS and 2SRI consistently have nominal cover-age under both null or non-null effects and weak or strongunmeasured confounding scenarios. The NPCB methodFig. 3 Bias versus the nonadherence rate for the null and non-null effect for simulation setting 2. The naïve and both-stages adjusted 2SLS and 2SRIshare the same line as they produce the same amount of bias. The baseline-adjusted PP and IP-weighted PP also share the same line. Abbreviations:RD: risk difference, PP: per-protocol, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stageresidual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 10 of 46produces a wide bound for both the null and the non-null effect, regardless of the weak or strong confounding(Appendix C Fig. 15). The width of bounds is very wide for20% to 80% nonadherence.Varying the effect of the treatmentWepresented bias, SE,MSE, and 95% coverage probabilityfor different treatment effects. We showed the results for10% and 40% nonadherence rates.Bias Figure 4 shows the bias versus treatment effect for10% and 40% nonadherence using DAG 2. All methodsproduce approximately an unbiased estimate under theweak confounding scenario with 10% nonadherence. Thebias is increased to about 0.015 for baseline-adjusted PPand IP-weighted PP methods under the strong confound-ing scenario. The bias is more pronounced (about 0.02) for40% nonadherence. However, the naive and both-stagesadjusted 2SLS and 2SRI methods produce approximatelyunbiased estimates for any treatment effects under weakor strong confounding. The bias versus treatment effectfor all methods using DAG 2 is shown in Appendix CFig. 16.SE, MSE, and coverage SE and MSE versus treatmenteffect for 10% and 40% nonadherence using DAG 2 arepresented in Appendix C Figs. 17 and 18. The baseline-adjusted PP and IP-weighted PP methods produce smallcoverage under the strong unmeasured confounding sce-nario (Fig. 19). On the other hand, the naive and both-stages adjusted 2SLS and 2SRI consistently have nominalcoverage regardless of the treatment effect and weak orstrong unmeasured confounding.Setting 3: exclusion restriction assumption violatedBiasFigure 5 shows the bias versus nonadherence rate usingDAG 3.We observed that the 2SLS and 2SRImethods pro-duce high biases when the exclusion restriction assump-tion is violated. The biases are between -0.3 and 0.3 fora minor violation of the exclusion restriction but beyondthe range of [-0.3,0.3] when the violation is severe. Thebaseline-adjusted PP and IP-weighted PP estimates arealso biased for minor or severe violations of the exclusionrestriction. However, the bias is slightly smaller (range0.05 to 0.25) compared to 2SLS and 2SRI methods. Thebias versus nonadherence rate for all methods using DAG3 is shown in Appendix C Fig. 20.SE, MSE, coverage, and bounds of NPCBThe SE and MSE versus nonadherence can be found inAppendix C Figs. 21 and 22. In terms of 95% coverageprobability, both PP and IV methods produce low cover-age for either minor or severe violations of the exclusionrestriction assumption (Appendix C Fig. 23). The NPCBFig. 4 Bias versus treatment effect in risk difference for 10 and 40% nonadherence for simulation setting 2. The naive and both-stages adjusted 2SLSand 2SRI share the same line as they produce the same amount of bias. The baseline-adjusted PP and IP-weighted PP also share the same line.Abbreviations: PP: per-protocol, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residualinclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 11 of 46Fig. 5 Bias versus the nonadherence rate for the null and non-null effect for simulation setting 3. The 2SLS and 2SRI methods share the same line,and the baseline-adjusted PP and IP-weighted PP share the same line. The bias is out of the bound [-0.6,0.6] for the 2SLS and 2SRI methods forsevere violation of the exclusion restriction. Abbreviations: PP: per-protocol, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS:two-stage least square, 2SRI: two-stage residual inclusionproduces wide bounds for all scenarios, regardless ofthe minor or severe violation of the exclusion restrictionassumption (Appendix C Fig. 24).Sensitivity analysisExclusion restriction satisfied, unmeasured confoundingpresentWe produced results for a simplified version of DAG 2 tomimic the possible set of variables as in our case study.The DAG is shown in Appendix C Fig. 25(A) and detailsare described in Appendix B. The results are shown inAppendix C Figs. 26–29. As expected, the ITT, PP, and ATmethods have biased estimates, while the 2SLS and 2SRImethods produce unbiased estimates.Exclusion restriction violatedWe produced results when the exclusion restrictionassumption is violated, where the setting mimics the pos-sible set of variables as in our case study. The DAG isshown in Appendix C Fig. 25(B) and details are describedin Appendix B. The results for bias versus nonadherenceare shown in Appendix C Fig. 30. We observed that the2SLS and 2SRI methods perform worse than even naivemethods, no matter whether there is a minor or severeviolation of the exclusion restriction assumption.Sensitivity analyses with a smaller sample sizeWe also simulated data for simulation settings 1-3 by con-sidering 500 samples (approximately 250 per arm). Theresults are shown in Appendix C Figs. 31–45. The patternsof bias remain the same for all scenarios. As expected,the SEs are higher for the scenarios with 500 samplesthan those with 2000 samples. Consequently, the MSEsare higher for the scenarios with 500 samples than thosewith 2000 samples. Also, the IP-weighted PP estimatessuffer from non-convergence issues when the nonadher-ence rate is very high (e.g., 90%). Overall, the conclusionsremain the same for the scenarios with a smaller sam-ple size (500) compared to those from a relatively largersample size (2000). We further attempted generating databy considering 100 samples (approximately 50 per arm),but faced non-convergence issues due to the low samplesize, impacting the stability of the estimates; and hence theresults are not shown here.We have added a flow chart showing the present study’srecommendations in Appendix C Fig. 46.Case study: vitamin A supplementation onchildhoodmortalityWe used a dataset from a two-arm randomized controltrial with 450 villages in northern Sumatra, exploringHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 12 of 46the effect of vitamin A supplementation on childhood(1-year) mortality [48, 49]. The villages were random-ized to either receive vitamin A supplementation or actas a control group for a year. A total of 23,682 chil-dren aged 12-71 months from these villages receiveda large oral dose of vitamin A supplementation (n =12,094) or no treatment (n = 11,588). Not every childtook the assigned vitamin A supplementation in the treat-ment group, and the nonadherence rate was 20% in thegroup receiving vitamin A supplements. Vitamin A sup-plements were not available to children in the controlgroup. This was a randomized trial, and extraneous fac-tors should not influence the instrument Z (the random-ization variable) [48]. Therefore, there should not be anydirect arrow from Z to Y , or Z to Y through anotherpathway other than A, meaning that the exclusion restric-tion assumption for this trial should be reasonably met.Previous studies reported very wide bounds from thesame dataset when analyzed by the NPCB method [19,20]. In the present study, we also assessed the perfor-mances of two other IV methods (2SLS and 2SRI) in thesame data. Since there was no measured confounder inthe dataset, we did not apply baseline covariate-adjustedmethods. Instead, we reported the results for the ITT,naive PP, naïve AT, naïve 2SLS, naïve 2SRI, and NPCBmethods.The event rate was 5.1 per 1,000. Table 4 shows therisk difference per 1,000 in the treatment group than thecontrol, standard error, and associated 95% CI. Accord-ing to the ITT estimate, 2.6 fewer deaths (95% CI: 0.8-4.4)were associated with vitamin A supplementation. How-ever, the ITT estimate do not take into account that alarge proportion of children failed to receive vitamin A asprescribed. The PP and AT analyses estimate how effica-cious was vitamin A supplementation among those chil-dren who actually took the treatment. According to thenaive PP, approximately 5.2 less deaths (95% CI: 3.5-6.8)were associated with vitamin A supplementation amongthose children who actually took the treatment, while itwas 6.5 fewer deaths (95% CI: 4.9-8.1) as per naive AT.The 2SLS and 2SRI produce identical results; approxi-mately 3.2 less deaths (95% CI: 1.0-5.5) among the com-pliers. In contrast, the SE was slightly higher for 2SLSand 2SRI compared to other methods. The bound forthe NPCB was reported to be very wide (-5.4 to 194.6)[19, 20].DiscussionSummary of the findingsWe used three data-generating mechanisms to study theperformance of various statistical methods to deal withnonadherence in pragmatic trial settings with a binarypoint treatment and a binary outcome. We consideredthree settings such as (1) exclusion restriction is satisfiedand there is no unmeasured confounding; (2) the exclu-sion restriction assumption is satisfied but there is unmea-sured confounding present; (3) the exclusion restrictionassumption is violated. No single method is the best inall situations. For setting 1, it is expected that baselineadjusted and IP-weighted PP, and 2SLS and 2SRI estimatesare unbiased. We observed that the naive and both-stagesadjusted 2SLS and 2SRI methods perform very well interms of bias and coverage for any nonadherence rate. Thebaseline-adjusted PP and IP-weighted PP methods out-perform the 2SLS and 2SRI methods for below a certainnonadherence rate but show very high bias beyond thatpoint. For our simulation settings, 80% was that cut-point.As expected, the baseline-adjusted PP and IP-weighted PPalso have high SE beyond that point. For setting 2, thebaseline-adjusted PP and IP-weighted PP perform well interms of bias, SE, and coverage compared to the 2SLS and2SRI methods when there is weak unmeasured confound-ing. However, these PP methods perform poorly underthe strong unmeasured confounding. Only naive 2SLS and2SRI methods and both-stages adjusted 2SLS and 2SRImethods produce approximately unbiased estimates irre-spective of weak or strong unmeasured confounding. Asexpected, these IV methods have high SEs in almost allTable 4 Estimated effect of vitamin A supplementation on childhood mortality in a two-arm randomized control trial with 450 villagesin northern SumatraMethod RD SE 95% CIITT -2.58 0.93 -4.40, -0.76Naïve PP -5.15 0.82 -6.76, -3.53Naïve AT -6.47 0.82 -8.08, -4.86Naïve 2SLS -3.23 1.16 -5.50, -0.95Naïve 2SRI -3.23 1.16 -5.50, -0.96NPCB 1 - - -5.39, 194.62Abbreviations: RD: risk difference per 1,000 in the treatment group than the control; SE: standard error, CI: confidence interval; ITT: intention-to-treat; PP: per-protocol; AT:as-treated; 2SLS: two-stage least square; 2SRI: two-stage residual inclusion; NPCB: nonparametric causal bound;1The interval estimate from the NPCB is the bound for the average causal estimate per 1,000 than a 95% CI.Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 13 of 46scenarios. For setting 3, all methods (PP and IV) per-form poorly in terms of bias, SE, MSE, and coverage. Asexpected, the 2SLS and 2SRI methods perform poorly dueto the exclusion restriction assumption violation. On theother hand, the baseline-adjusted PP and IP-weighted PPmethods also suffer in terms of bias and coverage, butfor a different reason: the instrument acts as an unad-justed confounder so that there exists an open backdoorpath between the treatment variable and the outcome.For all situations, the first stage adjusted 2SLS and 2SRIhave high biases and small coverage probabilities. TheNPCB method produces very wide bounds in almost allscenarios, including not capturing the true value in thepresence of a severe violation of the exclusion restrictionassumption.Context in the literatureIt is expected from the theory that the ITT, naive PP,and naive AT methods usually produce biased estimateswhen there is nonadherence [14, 50–52]. TheNICE guide-line also recommended avoiding these methods in thepresence of treatment nonadherence [53].The adjusted PP methods offer consistently excellentperformance under the scenarios without unmeasuredconfounding [14]. However, it is expected from the liter-ature that the baseline adjusted PP and IP-weighted PPmethods produce biased estimates when there is unmea-sured confounding [14]. Notably, the presence of unmea-sured confounding means the violation of the exchange-ability assumption [54]. Using the second definition of thePP methods (i.e., excluding those patients entirely fromthe analysis), we observed that the adjusted PP meth-ods, such as baseline adjusted PP and IP-weighted PP,have high biases in the presence of strong unmeasuredconfounding. Therefore, we should not use these meth-ods when it is believed that there is strong unmeasuredconfounding. On the other hand, the IV-based methodscan be used to get the unbiased treatment effect estimateeven if some of the necessary confounders are unmea-sured [15, 55]. However, the IVmethods typically producea higher SE, which is in-line with the theory [14]. Sincethe first-stage of the 2SLS and 2SRI is used to predictthe treatment, researchers may include covariates only inthe first-stage but not in the second stage [28, 29]. Ourstudy showed that adding the covariate only in the firststage of the model but not in the second stage leads tobias estimates. Also, covariates adjustment in IV methodscould reduce the SE, which is consistent with the theory[14]. Therefore, we recommend against using first-stageadjusted 2SLS and 2SRI methods, and prefer both-stagesadjusted 2SLS or 2SRI methods.The newer IV-based method, the NPCB, is used bymany researchers with the understanding that it can pro-duce a more precise causal bound [20]. The causal boundis a different metric compared to a 95% confidence inter-val, and hence coverage cannot be calculated directly. Oursimulations, however, showed that this method includedthe true parameter 100% of the time in almost all sce-narios, except for a severe violation of the exclusionrestriction assumption. Although high SE in 2SLS and2SRI methods motivate using the NPCB method, thismethod does not adjust for confounders directly and oftenproduces wide bounds [18–20].It is also expected from the theory that the IV-methodsperform poorly when the key assumption of an IV, suchas the exclusion restriction, is violated [24]. The exclu-sion restriction is untestable using the observed data.Usually, the randomization variable in a trial is an idealinstrument. But a weak instrument can lead to biasedand imprecise estimates [56, 57]. Therefore, the 2SLSand 2SRI methods should only be used when this IVassumption is plausible. Subject-matter knowledge shouldbe applied to rule out the possibility of violation of theIV assumption. Besides, the no unmeasured confound-ing assumption is also untestable from the observeddata [58]. As expected, the baseline-adjusted PP and IP-weighted PP perform poorly and producing high biaswhen there is unmeasured confounding [14]. However,these PP methods outperform the 2SLS and 2SRI meth-ods in terms of bias, MSE, and coverage if there is noor weak unmeasured confounding. Therefore, we suggestcollecting information on covariates, when possible, thatcould influence the treatment nonadherence to rule outthe possibility of having strong unmeasured confound-ing. Based on empirical observations, previous studiesproposed using the IV-methods in the presence of unmea-sured confounding, while IP-weighted PP can be used toaddress nonadherence with no unmeasured confounding[4, 59]. In the current work, we applied both of thesetypes of methods under a series of data generating mech-anisms and compared the results. We additionally suggestusing the both-stages adjusted 2SLS or 2SRI, but not thebaseline-adjusted PP or IP-weighted PP method whennonadherence is very high (e.g., ≥ 80% in our case). TheIV-based methods can also be used as complementaryanalysis because the assumptions of these IV methodsare different from the non-IV-based methods. If the pri-mary and the complementary analyses result in a similarconclusion, we can have more confidence in the overallconclusion.Case studyUsing the ITT analysis, 2.6 fewer deaths per 1,000 wereassociated with vitamin A supplementation was previ-ously reported in the literature [20]. The naive PP andnaive AT found 5.2 and 6.5 fewer deaths among thoseHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 14 of 46children who actually took the treatment than the con-trol group, respectively. Both naive 2SLS and 2SRI foundfewer deaths (RD 3.2) among the compliers. The high esti-mate by the naive AT is not surprising as this methodmostly gives a biased treatment effect when we have treat-ment nonadherence. The PP estimate is relatively differentfrom the results from 2SLS and 2SRI, might be indicativeof the presence of unmeasured confounding. Our sec-ond simulation setting shows that the PP methods have ahigh bias when there is strong unmeasured confounding,but the 2SLS and 2SRI methods still perform very well.Consistent with the literature [19, 20], NPCBmethod pro-duces a very wide bound, which might be due to the highnonadherence rate, as we observed in our simulation.StrengthsThe present study has several strengths. We consideredthree comprehensive simulation settings to explore theperformance of different methods to deal with treatmentnonadherence. We considered both the null and non-nulltreatment effect scenarios and low to high nonadher-ence (e.g., 10% to 90%), which enabled us to comparethe methods rigorously. For more than 90% nonadher-ence, however, most of the algorithms do not converge orproduce a very high effect estimate. Moreover, we consid-ered an approximately equal nonadherence rate per armso that the result has not influenced by the unequal non-adherence rate. We also considered a collapsible effectmeasure to estimate the treatment effect to ensure that themarginal and conditional treatment effects for any levelof the measured confounders are equal and comparable[39]. Utility of our simulation study is that this peda-gogic work confirmed theoretical results and helped makethe theoretical ideas more accessible to the practitionersand applied researchers. Additionally, we observed similarfindings from sensitivity analyses for unmeasured con-founding, violation of the exclusion restriction assump-tion, and small sample size, meaning that our study resultsare robust.Limitations and future directionDespite its strengths, the study has a few limitations. First,to clearly identify the effect of each factor, we startedfrom a simplistic scenario, but this is easily extendable.Future studies can consider more complex DAGs and addmore confounders with both linear and non-linear forms.Second, we did not consider any loss to follow-up, whilethe differential loss to follow-up can bias the results [5].The oversimplified point treatment settings and no loss tofollow-up also limit guiding about decisions about bias-variance trade-off. Third, our simulation settings haveequal nonadherence rates per arm versus one-sided differ-ential nonadherence in our case study. The pattern of bias,SE, MSE, and coverage of all methods under equal versusdifferential nonadherence are expected to be in the samedirection. Our team is exploring this issue in more detailto verify whether differential nonadherence could leadto different conclusions. Fourth, the results in this papermay not be generalizable to other types of outcomes.Some previous studies compared different methods toaddress treatment nonadherence when the outcome iscontinuous [60, 61] and time-to-event [26, 62–64]. How-ever, these studies mostly focused on comparing eitherIV or PP methods with the naive methods, or considereddifferent data structures or treatment strategies (e.g., sus-tained treatment strategies in longitudinal studies) [26,64]. Future studies should explore the performance ofboth IV and PPmethods in the point-treatment setting forcontinuous and time-to-event outcomes, multiple treat-ment arms, and when effect modifiers are present. Inaddition to the 2SLS, 2SRI, and NPCB methods, someother IV methods can be used in the same setting to dealwith treatment nonadherence. For example, the two-stagepredictor substitution (2SPS) can handle non-linearity inthe first stage of the 2SLS model and the three-stage leastsquares method (3SLS) methods can correct the correla-tion between error terms in the first and second stages ofthe 2SLS models [65, 66]. The IV-based structural meanmodel (SMM) is another method that can address treat-ment nonadherence [67]. This semi-parametric methoduses g-estimation for identification and estimation of thetreatment effect after addressing the nonadherence issue.Future studies can explore the performance of these IVmethods in contrast to the other IV methods and non-IVmethods when dealing with the treatment nonadherencein pragmatic trials.It is also worth mentioning that the 2SRI estimates canbe biased even without violating the exclusion restrictionassumption [65], and the level of bias may increase as theseverity of confounding increases. Cai et al. [65] reportedanalytic estimates of such bias in terms of log-odds ratiosfor a variety of situations. In our study, we observed min-imal or no increase in bias as the confounding severityincreases. This observation might be due to the fact thatwe were using RD as effect measure, and the impact wasless noticeable compared to that when effect measure wasodds ratio. Further studies could assess the bias in the2SRI method when the effect measure is collapsible.We require the positivity assumption, i.e., non-zeroprobability of being exposed or unexposed at every com-bination of the values of the observed confounders [68].For randomized trials (e.g., as our case study), that prob-ability of being exposed or unexposed is usually known,and so is belonging to one particular confounder cate-gory [68]. However, theoretical violations of positivitymayarise when patients with certain characteristics may beineligible to receive a treatment (e.g., absolute contraindi-cation for a given treatment) [69]. If this is the case, theHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 15 of 46weights from the IP-weighted PP methods can be infiniteor very large, and the estimates can be biased and unstable[69]. The high nonadherence rate or small sample sizemaylead to near positivity violations, which is also responsi-ble for large IP weights. Therefore, even when there is nounmeasured confounding but high nonadherence rates,the biased and large SE in our simulation results couldbe due to the near positivity violations. Methods such astruncation and overlap weights have been proposed in theliterature to deal with the large IP weight problem [70,71]. On the other hand, a modified causal estimator hasbeen proposed to deal with the positivity issue in the IVestimation [72, 73].When the treatment is sustained but time-varying, thesame methods that we considered in this study are notgenerally adequate, and methods have been extended toaddress those scenarios [26, 27]. Naïve methods suchas ITT, naïve PP, or naïve AT ignore time-varying con-founding and can produce biased estimates [9]. Thereexists some new literature exploring nonadherence ina sustained treatment strategy when longitudinal post-baseline prognostic variable measures are available, whichis beyond the scope of our point-treatment study.ConclusionBesides the research question, the choice of the meth-ods should come from researcher’s understanding of theunderlying data generating mechanism and available data(e.g., extent of unmeasured confounding and a potentialviolation of exclusion restriction). An additional consid-eration includes the nonadherence rate (e.g., high vs.low). Under nearly an ideal situation where there areunmeasured confounders but adjusting for measured con-founders can give an unbiased effect estimate, the naiveand both-stages adjusted but not first-stage adjusted 2SLSand 2SRI methods perform very well in terms of biasand coverage for any nonadherence rate. Although thenaive and both-stages adjusted 2SLS and 2SRI methodsgive almost identical bias, both-stages adjusted 2SLS and2SRI methods improve the precision and reduce overallMSE. In the same setting, the baseline-adjusted PP andIP-weighted PP outperform these 2SLS and 2SRI methodsin terms of bias, SE, and MSE for <80% nonadherence,but these PP methods show very high bias and MSE for≥ 80% nonadherence rate. If there is no unmeasured con-founding, the baseline-adjusted PP and IP-weighted PPconsistently outperform the 2SLS and 2SRI methods interms of bias, SE, MSE, and coverage. However, thesetwo PP methods perform poorly when some necessaryconfounders are unmeasured and measured confounderscannot sufficiently block the backdoor paths between thetreatment and the outcome. On the other hand, the 2SLSand 2SRI produce high biases and low coverage probabili-ties when there exists violation of the exclusion restrictionassumption. The baseline-adjusted PP and IP-weighted PPcan also have biased estimates when the exclusion restric-tion is violated. However, the baseline-adjusted PP andIP-weighted PP can produce unbiased estimates if all openbackdoor paths between the treatment variable and theoutcome can be blocked so that the association betweenthe instrument and outcome is nullified. Therefore, whenpossible, we recommend collecting information on nec-essary covariates that predict adherence and addressingthem appropriately in the analyses. Collecting informationon those covariates or augmentation of external datasources from electronic health records could reduce theimpact of having strong unmeasured confounding. Sinceassumptions of PP methods and IV methods are differentand often untestable, we suggest analyzing the data usingboth PP methods (baseline-adjusted PP or IP-weightedPP) and IV-methods (both-stages adjusted 2SLS or 2SRI)and reporting both results. Analysts are more likely tocome up with a robust conclusion of the real-world effectof a treatment if they have similar findings from differentanalyses requiring different assumptions.Appendix AThis section describes the estimation methods consid-ered in this study and the assumptions of these estimationmethods.Description of the estimation methodsLet Z is the two-arm randomization variable, A is thebinary treatment variable, L = (L1, L2) is a vector ofmeasured confounders, and Y is the binary outcome.The risk difference (RD) is our target parameter of inter-est. The binomial model with an identity link functioncan be used to estimate RD. Since the binomial modelfitting frequently shows convergence issues with adjust-ing for covariates in the model, the Poisson or Gaussianregression with an identity link function and robust sand-wich standard error can be used as an alternative [40]. Inthe present study, we considered intention-to-treat (ITT),naive per-protocol (PP), naive as-treated (AT), baselineadjusted ITT, baseline adjusted PP, IP-weighted PP, two-stage least square (2SLS), two-stage residual inclusion(2SRI), and non-parametric causal bound (NPCB) meth-ods to estimate RD and associated parameters for a two-arm pragmatic trial. The description of calculating RDusing these methods is given below.ITTThe ITT models Z on Y without adjustment for L. Themodel can be written asPr(Y = 1) = β0 + βITTZ.Then βˆITT is the maximum likelihood estimate (MLE) ofβITT and is the estimated RD.Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 16 of 46Naive PPThe naive PP models Z on Y among those subjects whoreceive the treatment according to the protocol (Z = A)but without adjustment for L. The model can be writtenas:Pr(Y = 1) = β0 + βNaive-PPZ for Z=A.Then βˆNaive-PP is the estimated RD.Naive ATThe naive AT models A on Y, but does not considerwhether individuals randomized to the treatment groups.The model cannot adjust for L and can be written as:Pr(Y = 1) = β0 + βNaive-ATA.Then βˆNaive-AT is the estimated RD.Baseline-adjusted ITTThe baseline-adjusted ITT is the same as ITT but itadjusts for L:Pr(Y = 1) = β0 + βB-ITTZ + β2L1 + β3L2.Then βˆB-ITT is the estimated RD.Baseline-adjusted PPThe baseline-adjusted PP is the same as naive PP but itadjusts for L:Pr(Y = 1) = β0 + βB-PPZ + β2L1 + β3L2 for Z=A.Then βˆB-PP is the estimated RD.IP-weighted PPThemethod creates inverse probability adherence weightsto generate a pseudo population to estimate the treatmenteffect by removing the effect of nonadherence [25]. Weused a logistic regression to estimate the adherence prob-abilities among those subjects who receive the treatmentaccording to the protocol (Z = A) as followslogit(Pr(A = 1)) = γ0 + γ1L1 + γ2L2.The predicted probability from the above model is theprobability of adherence PA. We calculated the stabilizedweights to prevent from extreme weights [7, 30]. The sta-bilized inverse probability of adherence weights can becalculated asWstabilized = Pr(A = 1) APA + Pr(A = 0)1 − A1 − PA .To estimate the RD, the weighted outcome model is used,which can be written asPr(Y = 1) = β0+βIPW-PPZ with weight = Wstabilized.Then βˆIPW-PP is the estimated RD.Naïve 2SLSThere are two stages in the 2SLS method. The instru-ment (Z) is modelled to A in the first stage, and then thepredicted treatment is modelled to Y in the second stage[31]. There is no adjustment for L in either stage of themodel. We used a logistic regression in the first stage ofthe model, which can be written aslogit(Pr(A = 1)) = γ0 + γ1Z.Then the second stage model can be written as followsPr(Y = 1) = β0 + βNaive-2SLSÂ,where Aˆ is the predicted treatment from the first stage,and then βˆNaive-2SLS is the estimated RD.First-stage adjusted 2SLSThe first-stage adjusted 2SLS is the same as naive 2SLSexcept it adjusts for L in the first stage of the model [28,29]. The same as naive 2SLS, we used a logistic regressionin the first stage of the modellogit(Pr(A = 1)) = γ0 + γ1Z + γ2L1 + γ3L2.Then the second stage model can be written as followsPr(Y = 1) = β0 + β1Stage-2SLSÂ,where Aˆ is the predicted probabilities for A from the firststage, and then βˆ1Stage-2SLS is the estimated RD.Both-stages adjusted 2SLSThe both-stages adjusted 2SLS is the same as naive 2SLSexcept it adjusts for L in both stages of the model. Thesame as naive and first-stage adjusted 2SLS, we used alogistic regression in the first stage of the modellogit(Pr(A = 1)) = γ0 + γ1Z + γ2L1 + γ3L2.Then the second stage model can be written as followsPr(Y = 1) = β0 + βBStage-2SLSÂ + β2L1 + β3L2,where Aˆ is the predicted probabilities for A from the firststage, and then βˆBStage-2SLS is the estimated RD.Naïve 2SRISimilar to the 2SLS, there are two stages in the 2SRImethod. But the instrument (Z) is modelled to A in thefirst stage, and then the residuals from the first stage andA are modelled to Y in the second stage [22]. There wasno adjustment for L in either stage of the model. We useda logistic regression in the first stage of the model, whichcan be written aslogit(Pr(A = 1)) = γ0 + γ1Z.The residuals from the first stage can be extracted asfollows:r = A − Â,Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 17 of 46where Aˆ is the predicted probability for A from the firststage. Then the second stage model can be written asfollowsPr(Y = 1) = β0 + βNaive-2SRIA + β2r,and then βˆNaive-2SRI is the estimated RD.First-stage adjusted 2SRIThe first-stage adjusted 2SRI is the same as naive 2SRIexcept it adjusts for L in the first stage of the model [29].The same as naive 2SRI, we used a logistic regression inthe first stage of the modellogit(Pr(A = 1)) = γ0 + γ1Z + γ2L1 + γ3L2.The same as before, the residuals from the first stage canbe extracted as follows:r = A − Â,where Aˆ is the predicted probability for A from the firststage. Then the second stage model can be written asfollowsPr(Y = 1) = β0 + β1Stage-2SRIA + β2r,and then βˆ1Stage-2SRI is the estimated RD.Both-stages adjusted 2SRIThe both-stages adjusted 2SRI is the same as naive 2SRIexcept it adjusts for L in both stages of the model [22].The same as naive and first-stage adjusted 2SRI, we useda logistic regression in the first stage of the modellogit(Pr(A = 1)) = γ0 + γ1Z + γ2L1 + γ3L2.Then the second stage model can be written as followsPr(Y = 1) = β0 + β1Stage-2SRIA + β2r + β2L1 + β3L2,where r = A − Â is the residuals from the first stage withAˆ is the predicted probability for A from the first stage.Then βˆ1Stage-2SRI is the estimated RD.NPCBThis nonparametric method estimates the bounds forthe effect of interest rather than a point estimate and isrestricted to a binary outcome (Y ) with a binary or tri-nary instrument [18, 19]. This method uses a constrainedprobability statement to provide bounds on the estimatedtreatment effect. But these bounds are the range of thetrue causal effect of interest, not the confidence interval[18, 19]. The bounds for the RD can be written asmax⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩p00.0 + p11.1 − 1p00.1 + p11.1 − 1p11.0 + p00.1 − 1p00.0 + p11.0 − 12p00.0 + p11.0 + p11.0 + p11.1 − 2p00.0 + 2p11.0 + p00.1 + p01.1 − 2p10.0 + p11.0 + 2p00.1 + p11.1 − 2p00.0 + p01.0 + p00.1 + 2p11.1 − 2⎫⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎬⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎭≤ RD≤ min⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩1 − p10.0 − p01.11 − p01.0 − p10.11 − p01.0 − p10.01 − p01.1 − p10.12 − 2p01.1 − p10.0 − p10.1 − p11.12 − p01.0 − 2p10.0 − p00.1 − p01.12 − p10.0 − p11.0 − 2p01.1 − p10.12 − p00.0 − p01.0 − p01.1 − 2p10.1⎫⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎬⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎭,where pya.z = Pr(Y = y, A = a|Z = z) with 0 ≤ pya.z ≤ 1and∑y,a pya.z = 1 [19].Assumptions of the estimation methodsTo estimate a causal effect, all models described in theMethods section assume consistency (i.e., the potentialoutcome under the observed treatment is the observedoutcome), no interference (i.e., potential outcome fora subject does not depend on the treatment status ofanother subject), exchangeability (i.e., no unmeasuredconfounding), positivity (i.e., probability of receivingeither treatment is greater than zero), and well-definedinterventions (e.g., taking 5mg aspirin) [74, 75]. All para-metric methods (all methods except the NPCB) furtherassume correct model specification [76]. The naive meth-ods assume a random pattern of nonadherence [4]. TheIV-based methods assume the IV is associated with thetreatment (relevance assumption), there are no defiers(monotonicity assumption), and the IV affects the out-come only through its effect on treatment [77]. The thirdassumption is also known as the exclusion restriction.Additionally, the 2SRI assumes the linearity of residu-als, i.e., residuals are linearly associated with the outcomeof interest [10, 77]. Furthermore, all methods assumemissing at random and no measurement error [78].Appendix BThis section is for the results of sensitivity analyses. TheDAGs are shown in Appendix C Fig. 25. Compared to themain text DAG, there is nomeasured confounder for theseappendix DAGs. We consider these simplified scenariosto replicate the scenario for our case study.Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 18 of 46Sensitivity analysis: exclusion restriction satisfied,unmeasured confounding presentSetupThis sensitivity analysis was done for a simplified versionof DAG 2 in the main text, where the exclusion restrictionis satisfied but unmeasured confounding is present. TheDAG is shown in Appendix C Fig. 25(A). For this scenario,we simulated data from the following algorithms, whereA is the treatment, Y is the outcome of interest, Z is therandomization, and U is unmeasured confounding.Z ∼ Bernoulli(0.5)U ∼ Bernoulli(0.5)A ∼ Bernoulli (pA) with pA = α0 + α1Z + α4UY ∼ Bernoulli (pY ) with pY = θ0 + θ1A + θ4U + θ5Z.(2)In Eqn. (2), α0 is associated with the nonadherence rate,α1 = 0.1 is the coefficient associated with Z, α4 = 0.1is the coefficient associated with U ; θ0 is associated withthe event rate, θ1 is the treatment effect of interest, andθ4 determines the strength of confounding, and θ5 deter-mines the strength of the direct effect of IV assumption ofexclusion restriction violation. Under different choices ofα0, we considered six levels of nonadherence: 10, 20, 40,60, 80, and 90%.We set θ0 = 0.2, and θ5 = 0 (i.e., no violation ofexclusion restriction assumption). For each of the sixnonadherence scenarios (α0), we considered five sce-narios of the treatment effect of interest (θ1) and twoversions of confounding (θ4), making a total of 60 sce-narios. We set the treatment effect of interest as θ1 ={−0.2,−0.05, 0, 0.05, 0.2} and confounding as θ4 = 0.1 and0.5 respectively for weak and strong confounding.ResultsSince there was no measured confounder (L) associatedwith our data generating mechanism here, there are nobaseline adjustedmethods as well as first-stage adjusted orboth-stages adjusted 2SLS and 2SRI methods. Instead, wereported the results for the ITT, naive PP, naive AT, naive2SLS, naive 2SRI, and NPCB methods.Under the null treatment effect scenario with weak con-founding, all methods produce small bias (Appendix CFig. 26). In the presence of strong confounding, the naivePP and AT produce slightly large bias. The amount of biasremains approximately the same for the non-null treat-ment effect. The ITT method produces a very small biasfor the null effect, while it performs the worst when thetreatment effect is non-null. In contrast, the 2SLS and2SRI produce unbiased estimates in all scenarios.Under the null treatment effect scenario, all methodsproduce small MSE in the presence of weak confounding,while naive PP shows higher MSE beyond 60% of nonad-herence (Appendix C Fig. 27). Under the non-null effect,we observed a similar pattern of MSE except for the ITT.The ITT method produces very high MSE regardless ofweak or strong confounding.Appendix C Fig. 28 shows the 95% coverage probabil-ity for different nonadherence rates using Appendix DAG1. As expected, the ITT method has the highest cover-age for the null treatment effect scenario, while it hasminimal coverage for the non-null effect. The naive PPand AT methods also produce noticeable small coveragewhen there is weak confounding, but these methods pro-duce very small coverage under the strong confoundingscenarios. On the other hand, naive 2SLS and 2SRI con-sistently have very high coverage under both null or non-null effects and weak or strong unmeasured confoundingscenarios.The NPCB method produces a wider bound for boththe null and the non-null effect, regardless of the weakor strong confounding (Appendix C Fig. 29). However,the width of bounds is only smaller for 10% and 90%nonadherence.Sensitivity analysis: violation of the exclusion restrictionSetupIn this example, the exclusion restriction is violated (Zdirectly affects Y ) and there exists an unmeasured con-founder U. The DAG is shown in Appendix C Fig. 25(B).We used Eqn. (2) to generate the data for Appendix DAG2. We set θ0 = 0.2, θ1 = 0.05, two versions of θ4,and two versions of θ5 with 10, 20, 40, 60, 80, and 90%nonadherence, making a total of 24 scenarios. We set θ5= 0.2 meaning that the exclusion restriction of the IVassumption violation is larger and θ5 = 0.05 indicating theviolation is smaller. The same as Appendix DAG 1, weset θ4 = 0.1 and 0.5 for weak and strong confounding,respectively.ResultsSince there was no measured confounder (L) associatedwith our data generating mechanism here, there are nobaseline adjustedmethods as well as first-stage adjusted orboth-stages adjusted 2SLS and 2SRI methods. Instead, wereported the results for the ITT, naive PP, naive AT, naive2SLS, naive 2SRI, and NPCB methods.Appendix C Fig. 30 shows the bias versus nonadher-ence rate using Appendix DAG 2. Appendix C Fig. 30(A)and (B) show the results when the violation of the exclu-sion restriction assumption is minor, and Appendix CFig. 30(C) and (D) show the results when violation issevere. We observed that the 2SLS and 2SRI methods pro-duce a greater bias when the exclusion restriction assump-tion is violated. The bias is more pronounced when theviolation of exclusion restriction assumption is severe.Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 19 of 46Appendix CThis section contains all the supplementary figures considered in this study.Supplementary figures for simulation setting 1Fig. 6 Bias versus the nonadherence rate using DAG 1. The naive 2SLS and 2SRI share the same line, the first stage adjusted 2SLS and 2SRI share thesame line, and both-stages adjusted 2SLS and 2SRI share the same line as they produce the same amount of bias. The first stage adjusted 2SLS and2SRI, and naive PP produce larger than 0.20 bias for strong unmeasured confounders so that the bias is out of the bound [0, 0.20]. Abbreviations: ITT:intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI:two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 20 of 46Fig. 7 Standard error (SE) versus the nonadherence rate using DAG 1. The naïve 2SLS and 2SRI share the same line, the first stage adjusted 2SLS and2SRI share the same line, and both-stages adjusted 2SLS and 2SRI share the same line as they produce approximately the same SE. Abbreviations:ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI:two-stage residual inclusionFig. 8Mean squared error (MSE) versus the nonadherence rate for the null effect using DAG 1. The naïve 2SLS and 2SRI share the same line, the firststage adjusted 2SLS and 2SRI share the same line, and both-stages adjusted 2SLS and 2SRI share the same line as they produce approximately thesame MSE. The first stage adjusted 2SLS and 2SRI, and naïve PP produce MSEs larger than 0.10 for strong unmeasured confounders so that the MSEis out of the bound [0, 0.1]. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weightedper-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 21 of 46Fig. 9 95% coverage probability versus the nonadherence rate using DAG 1. The naïve 2SLS and 2SRI share the same line, the first-stage adjusted2SLS and 2SRI share the same line, and both-stages adjusted 2SLS and 2SRI share the same line as they produce approximately the same coverageprobability. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS:two-stage least square, 2SRI: two-stage residual inclusionFig. 10Mean risk difference (RD) versus the nonadherence rate for the null effect using DAG 1. The grey lines are the lower and upper bound of theNPCB method. The naïve 2SLS and 2SRI share the same line, the first stage adjusted 2SLS and 2SRI share the same line, and both-stages adjusted2SLS and 2SRI share the same line as they produce the same RD estimate. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated,IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusion, NPCB: nonparametriccausal boundHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 22 of 46Supplementary figures for simulation setting 2Fig. 11 Bias versus the nonadherence rate for the null and non-null effect using DAG 2. The naïve 2SLS and 2SRI share the same line, the first stageadjusted 2SLS and 2SRI share the same line, and both-stages adjusted 2SLS and 2SRI share the same line as they produce the same amount of bias.The bias for the ITT and baseline-adjusted ITT are out of the bound [-0.1,0.1] for the non-null effect. Abbreviations: RD: risk difference, ITT:intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI:two-stage residual inclusionFig. 12 Standard error (SE) versus the nonadherence rate for null and non-null effect using DAG 2. All 2SLS and 2SRI methods share the same line asthey produce approximately the same SE. The PP methods also share approximately the same line. Abbreviations: RD: risk difference, ITT:intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI:two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 23 of 46Fig. 13Mean squared error (MSE) versus the nonadherence rate for null and non-null effect using DAG 2. The 2SLS and 2SRI share the same line asthey produce approximately the same amount of MSE. The ITT and baseline-adjusted ITT produce MSEs larger than 0.03 for the non-null effectscenario so that the MSE is out of the bound [0, 0.03]. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT: as-treated,IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionFig. 14 95 percentage coverage probability versus the nonadherence rate for the null and non-null effect using DAG 2. The naive and both-stagesadjusted 2SLS and 2SRI share the same line as they produce approximately the same coverage probability. Abbreviations: RD: risk difference, ITT:intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI:two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 24 of 46Fig. 15Mean risk difference (RD) versus the nonadherence rate for DAG 2. The grey lines are the lower and upper bound of the NPCB method. TheITT and baseline-adjusted ITT superimposed on each other. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT: as-treated,IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusion, NPCB: nonparametriccausal boundFig. 16 Bias versus treatment effect in risk difference for 10 percent and 40 percent nonadherence using DAG 2. The naive and both-stages adjusted2SLS and 2SRI share the same line as they produce the same amount of bias. The first-stage adjusted 2SLS and 2SRI methods also share the sameline. The ITT and baseline-adjusted ITT produce high bias for 40 percent nonadherence so that the bias is out of the bound [-0.05, 0.05].Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stageleast square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 25 of 46Fig. 17 Standard error (SE) versus treatment effect in risk difference for 10 percent and 40 percent nonadherence using DAG 2. All 2SLS and 2SRImethods share the same line as they produce approximately the same SE. The PP methods also superimposed on each other. Abbreviations: ITT:intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI:two-stage residual inclusionFig. 18Mean squared error (MSE) versus treatment effect in risk difference for 10 percent and 40 percent nonadherence using DAG 2.Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stageleast square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 26 of 46Fig. 19 95 percentage coverage probability versus treatment effect in risk difference for 10 percent and 40 percent nonadherence using DAG 2. Thecoverage probability for the ITT and baseline-adjusted ITT is less than 0.5 for some scenarios and out of the bound [0.5, 1]. Abbreviations: ITT:intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI:two-stage residual inclusionSupplementary figures for simulation setting 3Fig. 20 Bias versus the nonadherence rate for the null and non-null effect using DAG 3. Here, violation indicates the violation of the exclusionrestriction assumption. The naive and both-stages adjusted 2SLS and 2SRI, and the first-stage adjusted 2SLS and 2SRI methods share the same lineas they produce the same amount of bias. The ITT, 2SLS, and 2SRI methods have high bias in some scenarios and thus out of the bound [-0.6,0.6].Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stageleast square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 27 of 46Fig. 21 Standard error (SE) versus the nonadherence rate for the null and non-null effect using DAG 3. Here, violation indicates the violation of theexclusion restriction assumption. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weightedper-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionFig. 22Mean squared error (MSE) versus the nonadherence rate for the null and non-null effect using DAG 3. Here, violation indicates the violationof the exclusion restriction assumption. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverseprobability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 28 of 46Fig. 23 95 percentage coverage probability versus the nonadherence rate for the null and non-null effect using DAG 3. Here, violation indicates theviolation of the exclusion restriction assumption. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverseprobability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionFig. 24Mean risk difference (RD) versus the nonadherence rate for DAG 3. Here, violation indicates the violation of the exclusion restrictionassumption. The grey lines are the lower and upper bound of the NPCB method. The ITT and baseline-adjusted ITT; naive and both-stages adjusted2SLS and 2SRI superimposed on each other. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP:inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusion, NPCB: nonparametric causal boundHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 29 of 46Appendix DAGs for sensitivity analysisFig. 25 Two simplified versions of causal diagrams representing the simulation mechanisms considered in this study. Appendix DAG 1 is simplifiedversions of DAG 2 in the main text (unmeasured confounding), and Appendix DAG 2 is the simplified versions of DAG 3 in the main text (exclusionrestriction violated). Here, Z is the randomization variable, A is the treatment, U is unmeasured confounders, and Y is the outcomeResults for sensitivity analyses when exclusion restriction satisfied but unmeasured confounding present (Appendix DAG 1)Fig. 26 Bias versus the nonadherence rate for the null and non-null effect using Appendix DAG 1. The 2SLS and 2SRI share the same line as theyproduce the same amount of bias. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT: as-treated, 2SLS: two-stage leastsquare, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 30 of 46Fig. 27Mean squared error (MSE) versus the nonadherence rate for null and non-null effect using Appendix DAG 1. The 2SLS and 2SRI share thesame line as they produce approximately the same amount of MSE. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT:as-treated, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionFig. 28 95 percentage coverage probability versus the nonadherence rate for the null and non-null effect using Appendix DAG 1. The 2SLS and 2SRIshare the same line as they produce approximately the same coverage probability. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP:per-protocol, AT: as-treated, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 31 of 46Fig. 29Mean risk difference (RD) versus the nonadherence rate for Appendix DAG 1. The grey lines are the lower and upper bound of the NPCBmethod. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, 2SLS: two-stage least square, 2SRI: two-stage residual inclusion, NPCB:nonparametric causal boundResults for sensitivity analyses when exclusion restriction violated (Appendix DAG 2)Fig. 30 Bias versus the nonadherence rate using Appendix DAG 2. In both scenarios (small/large violation), the target treatment effect is assumed tobe 0.05. The 2SLS and 2SRI share the same line as they produce the same amount of bias. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT:as-treated, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 32 of 46Results for sensitivity analyses with 500 sample sizeFig. 31 Bias versus the nonadherence rate using DAG 1 for 500 samples. The naive 2SLS and 2SRI share the same line, the first stage adjusted 2SLSand 2SRI share the same line, and both-stages adjusted 2SLS and 2SRI share the same line as they produce the same amount of bias. The first stageadjusted 2SLS and 2SRI, and naive PP produce larger than 0.20 bias for strong unmeasured confounders so that the bias is out of the bound.Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stageleast square, 2SRI: two-stage residual inclusionFig. 32 Standard error (SE) versus the nonadherence rate using DAG 1 for 500 samples. The naïve 2SLS and 2SRI share the same line, the first stageadjusted 2SLS and 2SRI share the same line, and both-stages adjusted 2SLS and 2SRI share the same line as they produce approximately the sameSE. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stageleast square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 33 of 46Fig. 33Mean squared error (MSE) versus the nonadherence rate using DAG 1 for 500 samples. The naïve 2SLS and 2SRI share the same line, the firststage adjusted 2SLS and 2SRI share the same line, and both-stages adjusted 2SLS and 2SRI share the same line as they produce approximately thesame MSE. The first stage adjusted 2SLS and 2SRI, and naïve PP produce MSEs larger than 0.15 for strong unmeasured confounders so that the MSEis out of the bound [0, 0.15]. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weightedper-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionFig. 34 95 percentage coverage probability versus the nonadherence rate using DAG 1 for 500 samples. The naïve 2SLS and 2SRI share the sameline, the first-stage adjusted 2SLS and 2SRI share the same line, and both-stages adjusted 2SLS and 2SRI share the same line as they produceapproximately the same coverage probability. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverseprobability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 34 of 46Fig. 35Mean risk difference (RD) versus the nonadherence rate using DAG 1 for 500 samples. The grey lines are the lower and upper bound of theNPCB method. The naïve 2SLS and 2SRI share the same line, the first stage adjusted 2SLS and 2SRI share the same line, and both-stages adjusted2SLS and 2SRI share the same line as they produce the same RD estimate. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated,IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusion, NPCB: nonparametriccausal boundFig. 36 Bias versus the nonadherence rate using DAG 2 for 500 samples. The naïve and both-stages adjusted 2SLS and 2SRI share the same line, andthe first stage adjusted 2SLS and 2SRI share the same line as they produce the same amount of bias. Abbreviations: RD: risk difference, ITT:intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI:two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 35 of 46Fig. 37 Standard error (SE) versus the nonadherence rate using DAG 2 for 500 samples. The naïve and both-stages adjusted 2SLS and 2SRI share thesame line, and the first stage adjusted 2SLS and 2SRI share the same line as they produce approximately the same SE. The PP methods also shareapproximately the same line. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverseprobability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionFig. 38Mean squared error (MSE) versus the nonadherence rate using DAG 2 for 500 samples. The 2SLS and 2SRI share the same line as theyproduce approximately the same amount of MSE. The ITT and baseline-adjusted ITT produce MSEs larger than 0.10 for the non-null effect scenarioso that the MSE is out of the bound [0, 0.10]. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP:inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 36 of 46Fig. 39 95 percentage coverage probability versus the nonadherence rate for using DAG 2 for 500 samples. The naive and both-stages adjusted2SLS and 2SRI share the same line as they produce approximately the same coverage probability. The coverage probability is out of bound for theITT methods with the non-null effect. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP:inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionFig. 40Mean risk difference (RD) versus the nonadherence rate for using DAG 2 for 500 samples. The grey lines are the lower and upper bound ofthe NPCB method. The ITT and baseline-adjusted ITT superimposed on each other. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP:per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residualinclusion, NPCB: nonparametric causal boundHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 37 of 46Fig. 41 Bias versus the nonadherence rate using DAG 3 for 500 samples. Here, violation indicates the violation of the exclusion restrictionassumption. The both-stages adjusted 2SLS and 2SRI methods share approximately the same line. These methods have high biases when theexclusion restriction violation is severe and thus out of the bound. The biases for ITT methods are also out of the bound for the non-null effect.Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stageleast square, 2SRI: two-stage residual inclusionFig. 42 Standard error (SE) versus the nonadherence rate using DAG 3 for 500 samples. Here, violation indicates the violation of the exclusionrestriction assumption. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weightedper-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 38 of 46Fig. 43Mean squared error (MSE) versus the nonadherence rate using DAG 3 for 500 samples. Here, violation indicates the violation of the exclusionrestriction assumption. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weightedper-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionFig. 44 95 percentage coverage probability versus the nonadherence rate using DAG 3 for 500 samples. Here, violation indicates the violation of theexclusion restriction assumption. Abbreviations: ITT: intention-to-treat, PP: per-protocol, AT: as-treated, IP-weighted PP: inverse probability-weightedper-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 39 of 46Fig. 45Mean risk difference (RD) versus the nonadherence rate using DAG 3 for 500 samples. Here, violation indicates the violation of the exclusionrestriction assumption. The grey lines are the lower and upper bound of the NPCB method. The ITT and baseline-adjusted ITT; naive and both-stagesadjusted 2SLS and 2SRI superimposed on each other. Abbreviations: RD: risk difference, ITT: intention-to-treat, PP: per-protocol, AT: as-treated,IP-weighted PP: inverse probability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusion, NPCB: nonparametriccausal boundPresent study’s recommendationFig. 46 Flow chart showing the conclusion or recommendation from the present study. Abbreviations: PP: per-protocol, IP-weighted PP: inverseprobability-weighted per-protocol, 2SLS: two-stage least square, 2SRI: two-stage residual inclusionHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 40 of 46Appendix DThis section is for the parameterization of each three data-generating processes in this study. Appendix D Tables 5–7show the parameters for the simulation settings 1-3 outlined in DAGs 1-3, respectively. In each table, we have the sameset of parameters: α0 determines the nonadherence rate, α1 is the coefficient for Z-A association, α2 is the coefficientfor L1-A association, α3 is the coefficient for L2-A association, and α4 is the coefficient for U-A association. Also, θ0determines the event rate, θ1 is treatment effect of interest (i.e., the coefficient for A-Y association), θ2 is the coefficientfor L1-Y association, θ3 is the coefficient for L2-Y association, θ4 is the coefficient forU-Y association, and θ5 determinesthe Z-Y association.Table 5 Parameterization of simulation for data generating process for simulation setting 1 outlines in DAG 1 (exclusion restrictionsatisfied and no unmeasured confounding)# Arm α0 α1 α2 α3 α4 Nonadherence θ0 θ1 θ2 θ3 θ4 θ51Z = 1 0.72 0.6 0.4 0.35 0 10 -1 0 0 0 0.5 0Z = 0 -4.06 0.6 0.4 0.35 0 10 -1 0 0 0 0.5 02Z = 1 -0.23 0.6 0.4 0.35 0 20 -1 0 0 0 0.5 0Z = 0 -3.14 0.6 0.4 0.35 0 20 -1 0 0 0 0.5 03Z = 1 -1.47 0.6 0.4 0.35 0 40 -1 0 0 0 0.5 0Z = 0 -1.92 0.6 0.4 0.35 0 40 -1 0 0 0 0.5 04Z = 1 -2.52 0.6 0.4 0.35 0 60 -1 0 0 0 0.5 0Z = 0 -0.85 0.6 0.4 0.35 0 60 -1 0 0 0 0.5 05Z = 1 -3.76 0.6 0.4 0.35 0 80 -1 0 0 0 0.5 0Z = 0 0.39 0.6 0.4 0.35 0 80 -1 0 0 0 0.5 06Z = 1 -4.72 0.6 0.4 0.35 0 90 -1 0 0 0 0.5 0Z = 0 1.35 0.6 0.4 0.35 0 90 -1 0 0 0 0.5 07Z = 1 0.72 0.6 0.4 0.35 0 10 -5.5 0 0 0 8 0Z = 0 -4.06 0.6 0.4 0.35 0 10 -5.5 0 0 0 8 08Z = 1 -0.23 0.6 0.4 0.35 0 20 -5.5 0 0 0 8 0Z = 0 -3.14 0.6 0.4 0.35 0 20 -5.5 0 0 0 8 09Z = 1 -1.47 0.6 0.4 0.35 0 40 -5.5 0 0 0 8 0Z = 0 -1.92 0.6 0.4 0.35 0 40 -5.5 0 0 0 8 010Z = 1 -2.52 0.6 0.4 0.35 0 60 -5.5 0 0 0 8 0Z = 0 -0.85 0.6 0.4 0.35 0 60 -5.5 0 0 0 8 011Z = 1 -3.76 0.6 0.4 0.35 0 80 -5.5 0 0 0 8 0Z = 0 0.39 0.6 0.4 0.35 0 80 -5.5 0 0 0 8 012Z = 1 -4.72 0.6 0.4 0.35 0 90 -5.5 0 0 0 8 0Z = 0 1.35 0.6 0.4 0.35 0 90 -5.5 0 0 0 8 0Table 6 Parameterization of simulation for data generating process for simulation setting 2 outlines in DAG 2 (Exclusion restrictionsatisfied, unmeasured confounding present)# Arm α0 α1 α2 α3 α4 Nonadherence θ0 θ1 θ2 θ3 θ4 θ51Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.35 -0.2 0.02 0.05 0.05 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.35 -0.2 0.02 0.05 0.05 02Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.35 -0.2 0.02 0.05 0.05 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.35 -0.2 0.02 0.05 0.05 03Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.35 -0.2 0.02 0.05 0.05 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.35 -0.2 0.02 0.05 0.05 0Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 41 of 46Table 6 Parameterization of simulation for data generating process for simulation setting 2 outlines in DAG 2 (Exclusion restrictionsatisfied, unmeasured confounding present) (Continued)# Arm α0 α1 α2 α3 α4 Nonadherence θ0 θ1 θ2 θ3 θ4 θ54Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.35 -0.2 0.02 0.05 0.05 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.35 -0.2 0.02 0.05 0.05 05Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.35 -0.2 0.02 0.05 0.05 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.35 -0.2 0.02 0.05 0.05 06Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.35 -0.2 0.02 0.05 0.05 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.35 -0.2 0.02 0.05 0.05 07Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.2 -0.2 0.02 0.05 0.4 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.2 -0.2 0.02 0.05 0.4 08Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.2 -0.2 0.02 0.05 0.4 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.2 -0.2 0.02 0.05 0.4 09Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.2 -0.2 0.02 0.05 0.4 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.2 -0.2 0.02 0.05 0.4 010Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.2 -0.2 0.02 0.05 0.4 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.2 -0.2 0.02 0.05 0.4 011Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.2 -0.2 0.02 0.05 0.4 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.2 -0.2 0.02 0.05 0.4 012Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.2 -0.2 0.02 0.05 0.4 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.2 -0.2 0.02 0.05 0.4 013Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.3 -0.05 0.02 0.05 0.05 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.3 -0.05 0.02 0.05 0.05 014Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.3 -0.05 0.02 0.05 0.05 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.3 -0.05 0.02 0.05 0.05 015Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.3 -0.05 0.02 0.05 0.05 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.3 -0.05 0.02 0.05 0.05 016Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.3 -0.05 0.02 0.05 0.05 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.3 -0.05 0.02 0.05 0.05 017Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.3 -0.05 0.02 0.05 0.05 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.3 -0.05 0.02 0.05 0.05 018Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.3 -0.05 0.02 0.05 0.05 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.3 -0.05 0.02 0.05 0.05 019Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.15 -0.05 0.02 0.05 0.4 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.15 -0.05 0.02 0.05 0.4 020Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.15 -0.05 0.02 0.05 0.4 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.15 -0.05 0.02 0.05 0.4 021Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.15 -0.05 0.02 0.05 0.4 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.15 -0.05 0.02 0.05 0.4 022Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.15 -0.05 0.02 0.05 0.4 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.15 -0.05 0.02 0.05 0.4 023Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.15 -0.05 0.02 0.05 0.4 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.15 -0.05 0.02 0.05 0.4 024Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.15 -0.05 0.02 0.05 0.4 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.15 -0.05 0.02 0.05 0.4 0Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 42 of 46Table 6 Parameterization of simulation for data generating process for simulation setting 2 outlines in DAG 2 (Exclusion restrictionsatisfied, unmeasured confounding present) (Continued)# Arm α0 α1 α2 α3 α4 Nonadherence θ0 θ1 θ2 θ3 θ4 θ525Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.28 0 0.02 0.05 0.05 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.28 0 0.02 0.05 0.05 026Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.28 0 0.02 0.05 0.05 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.28 0 0.02 0.05 0.05 027Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.28 0 0.02 0.05 0.05 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.28 0 0.02 0.05 0.05 028Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.28 0 0.02 0.05 0.05 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.28 0 0.02 0.05 0.05 029Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.28 0 0.02 0.05 0.05 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.28 0 0.02 0.05 0.05 030Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.28 0 0.02 0.05 0.05 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.28 0 0.02 0.05 0.05 031Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.13 0 0.02 0.05 0.4 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.13 0 0.02 0.05 0.4 032Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.13 0 0.02 0.05 0.4 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.13 0 0.02 0.05 0.4 033Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.13 0 0.02 0.05 0.4 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.13 0 0.02 0.05 0.4 034Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.13 0 0.02 0.05 0.4 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.13 0 0.02 0.05 0.4 035Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.13 0 0.02 0.05 0.4 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.13 0 0.02 0.05 0.4 036Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.13 0 0.02 0.05 0.4 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.13 0 0.02 0.05 0.4 037Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.25 0.05 0.02 0.05 0.05 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.25 0.05 0.02 0.05 0.05 038Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.25 0.05 0.02 0.05 0.05 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.25 0.05 0.02 0.05 0.05 039Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.25 0.05 0.02 0.05 0.05 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.25 0.05 0.02 0.05 0.05 040Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.25 0.05 0.02 0.05 0.05 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.25 0.05 0.02 0.05 0.05 041Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.25 0.05 0.02 0.05 0.05 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.25 0.05 0.02 0.05 0.05 042Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.25 0.05 0.02 0.05 0.05 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.25 0.05 0.02 0.05 0.05 043Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.1 0.05 0.02 0.05 0.4 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.1 0.05 0.02 0.05 0.4 044Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.1 0.05 0.02 0.05 0.4 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.1 0.05 0.02 0.05 0.4 045Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.1 0.05 0.02 0.05 0.4 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.1 0.05 0.02 0.05 0.4 0Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 43 of 46Table 6 Parameterization of simulation for data generating process for simulation setting 2 outlines in DAG 2 (Exclusion restrictionsatisfied, unmeasured confounding present) (Continued)# Arm α0 α1 α2 α3 α4 Nonadherence θ0 θ1 θ2 θ3 θ4 θ546Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.1 0.05 0.02 0.05 0.4 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.1 0.05 0.02 0.05 0.4 047Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.1 0.05 0.02 0.05 0.4 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.1 0.05 0.02 0.05 0.4 048Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.1 0.05 0.02 0.05 0.4 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.1 0.05 0.02 0.05 0.4 049Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.2 0.2 0.02 0.05 0.05 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.2 0.2 0.02 0.05 0.05 050Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.2 0.2 0.02 0.05 0.05 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.2 0.2 0.02 0.05 0.05 051Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.2 0.2 0.02 0.05 0.05 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.2 0.2 0.02 0.05 0.05 052Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.2 0.2 0.02 0.05 0.05 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.2 0.2 0.02 0.05 0.05 053Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.2 0.2 0.02 0.05 0.05 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.2 0.2 0.02 0.05 0.05 054Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.2 0.2 0.02 0.05 0.05 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.2 0.2 0.02 0.05 0.05 055Z = 1 0.55 0.25 0.02 0.04 0.05 11 0.02 0.2 0.02 0.05 0.4 0Z = 0 0.02 0.25 0.02 0.04 0.05 11 0.02 0.2 0.02 0.05 0.4 056Z = 1 0.46 0.25 0.02 0.04 0.05 20 0.02 0.2 0.02 0.05 0.4 0Z = 0 0.12 0.25 0.02 0.04 0.05 21 0.02 0.2 0.02 0.05 0.4 057Z = 1 0.25 0.25 0.02 0.04 0.05 41 0.02 0.2 0.02 0.05 0.4 0Z = 0 0.32 0.25 0.02 0.04 0.05 41 0.02 0.2 0.02 0.05 0.4 058Z = 1 0.05 0.25 0.02 0.04 0.05 61 0.02 0.2 0.02 0.05 0.4 0Z = 0 0.52 0.25 0.02 0.04 0.05 61 0.02 0.2 0.02 0.05 0.4 059Z = 1 -0.15 0.25 0.02 0.04 0.05 81 0.02 0.2 0.02 0.05 0.4 0Z = 0 0.7 0.25 0.02 0.04 0.05 80 0.02 0.2 0.02 0.05 0.4 060Z = 1 -0.25 0.25 0.02 0.04 0.05 91 0.02 0.2 0.02 0.05 0.4 0Z = 0 0.8 0.25 0.02 0.04 0.05 89 0.02 0.2 0.02 0.05 0.4 0Table 7 Parameterization of simulation for data generating process for simulation setting 3 outlines in DAG 3 (Exclusion restrictionviolated)# Arm α0 α1 α2 α3 α4 Nonadherence θ0 θ1 θ2 θ3 θ4 θ51Z = 1 0.86 0 0.01 0.04 0 10 0.2 0 0.03 0.1 0.05 0.05Z = 0 0.06 0 0.01 0.04 0 10 0.2 0 0.03 0.1 0.05 0.052Z = 1 0.76 0 0.01 0.04 0 20 0.2 0 0.03 0.1 0.05 0.05Z = 0 0.16 0 0.01 0.04 0 20 0.2 0 0.03 0.1 0.05 0.053Z = 1 0.56 0 0.01 0.04 0 41 0.2 0 0.03 0.1 0.05 0.05Z = 0 0.36 0 0.01 0.04 0 40 0.2 0 0.03 0.1 0.05 0.054Z = 1 0.36 0 0.01 0.04 0 60 0.2 0 0.03 0.1 0.05 0.05Z = 0 0.57 0 0.01 0.04 0 60 0.2 0 0.03 0.1 0.05 0.05Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 44 of 46Table 7 Parameterization of simulation for data generating process for simulation setting 3 outlines in DAG 3 (Exclusion restrictionviolated) (Continued)# Arm α0 α1 α2 α3 α4 Nonadherence θ0 θ1 θ2 θ3 θ4 θ55Z = 1 0.16 0 0.01 0.04 0 80 0.2 0 0.03 0.1 0.05 0.05Z = 0 0.77 0 0.01 0.04 0 80 0.2 0 0.03 0.1 0.05 0.056Z = 1 0.06 0 0.01 0.04 0 90 0.2 0 0.03 0.1 0.05 0.05Z = 0 0.86 0 0.01 0.04 0 90 0.2 0 0.03 0.1 0.05 0.057Z = 1 0.86 0 0.01 0.04 0 10 0.1 0 0.03 0.1 0.05 0.2Z = 0 0.06 0 0.01 0.04 0 10 0.1 0 0.03 0.1 0.05 0.28Z = 1 0.76 0 0.01 0.04 0 20 0.1 0 0.03 0.1 0.05 0.2Z = 0 0.16 0 0.01 0.04 0 20 0.1 0 0.03 0.1 0.05 0.29Z = 1 0.56 0 0.01 0.04 0 41 0.1 0 0.03 0.1 0.05 0.2Z = 0 0.36 0 0.01 0.04 0 40 0.1 0 0.03 0.1 0.05 0.210Z = 1 0.36 0 0.01 0.04 0 60 0.1 0 0.03 0.1 0.05 0.2Z = 0 0.57 0 0.01 0.04 0 60 0.1 0 0.03 0.1 0.05 0.211Z = 1 0.16 0 0.01 0.04 0 80 0.1 0 0.03 0.1 0.05 0.2Z = 0 0.77 0 0.01 0.04 0 80 0.1 0 0.03 0.1 0.05 0.212Z = 1 0.06 0 0.01 0.04 0 90 0.1 0 0.03 0.1 0.05 0.2Z = 0 0.86 0 0.01 0.04 0 90 0.1 0 0.03 0.1 0.05 0.213Z = 1 0.86 0 0.01 0.04 0 10 0.2 0.2 0.03 0.1 0.05 0.05Z = 0 0.06 0 0.01 0.04 0 10 0.2 0.2 0.03 0.1 0.05 0.0514Z = 1 0.76 0 0.01 0.04 0 20 0.2 0.2 0.03 0.1 0.05 0.05Z = 0 0.16 0 0.01 0.04 0 20 0.2 0.2 0.03 0.1 0.05 0.0515Z = 1 0.56 0 0.01 0.04 0 41 0.2 0.2 0.03 0.1 0.05 0.05Z = 0 0.36 0 0.01 0.04 0 40 0.2 0.2 0.03 0.1 0.05 0.0516Z = 1 0.36 0 0.01 0.04 0 60 0.2 0.2 0.03 0.1 0.05 0.05Z = 0 0.57 0 0.01 0.04 0 60 0.2 0.2 0.03 0.1 0.05 0.0517Z = 1 0.16 0 0.01 0.04 0 80 0.2 0.2 0.03 0.1 0.05 0.05Z = 0 0.77 0 0.01 0.04 0 80 0.2 0.2 0.03 0.1 0.05 0.0518Z = 1 0.06 0 0.01 0.04 0 90 0.2 0.2 0.03 0.1 0.05 0.05Z = 0 0.86 0 0.01 0.04 0 90 0.2 0.2 0.03 0.1 0.05 0.0519Z = 1 0.86 0 0.01 0.04 0 10 0.1 0.2 0.03 0.1 0.05 0.2Z = 0 0.06 0 0.01 0.04 0 10 0.1 0.2 0.03 0.1 0.05 0.220Z = 1 0.76 0 0.01 0.04 0 20 0.1 0.2 0.03 0.1 0.05 0.2Z = 0 0.16 0 0.01 0.04 0 20 0.1 0.2 0.03 0.1 0.05 0.221Z = 1 0.56 0 0.01 0.04 0 41 0.1 0.2 0.03 0.1 0.05 0.2Z = 0 0.36 0 0.01 0.04 0 40 0.1 0.2 0.03 0.1 0.05 0.222Z = 1 0.36 0 0.01 0.04 0 60 0.1 0.2 0.03 0.1 0.05 0.2Z = 0 0.57 0 0.01 0.04 0 60 0.1 0.2 0.03 0.1 0.05 0.223Z = 1 0.16 0 0.01 0.04 0 80 0.1 0.2 0.03 0.1 0.05 0.2Z = 0 0.77 0 0.01 0.04 0 80 0.1 0.2 0.03 0.1 0.05 0.224Z = 1 0.06 0 0.01 0.04 0 90 0.1 0.2 0.03 0.1 0.05 0.2Z = 0 0.86 0 0.01 0.04 0 90 0.1 0.2 0.03 0.1 0.05 0.2Hossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 45 of 46Abbreviations2SLS: Two-stage least square; 2SRI: Two-stage residual inclusion; AT:As-treated; CI: Confidence interval; DAG: Directed acyclic graph; IP-weightedPP: Inverse probability-weighted per-protocol; ITT: Intention-to-treat; IV:Instrumental variable; MSE: Mean squared error; NPCB: Nonparametric causalbound; PP: Per-protocol; RD: Risk difference; SE: Standard errorAcknowledgementsAuthors thank the anonymous reviewers for their constructive comments,which helped improve the manuscript. Authors would like to acknowledge AliShajarizadeh for his valuable comments on an earlier version of the draftmanuscript. We also thank Sharon Roman, a patient partner of the BCSUPPORT Unit methods clusters, for her involvement with the project.Authors’ contributionsMBH performed the simulations and analysis, drafted the initial manuscript,and revised it based on received comments. LM initially assessed the feasibilityof the work via a different set of simulations that informed some directions ofthis work, critically reviewed the manuscript, and provided comments. MEKconceptualized the research question, secured funding, supervised theresearch, critically reviewed and revised the manuscript. All authors read andapproved the final manuscript.FundingThis work was supported by BC Support Unit’s Real-World Clinical TrialsMethods Cluster, Project #2, led by Dr. Karim (with research members PaulGustafson, Joan Hu, Hubert Wong and Derek Ouyang), and Dr. Karim’s NaturalSciences and Engineering Research Council of Canada (NSERC) fund andNSERC Discovery Accelerator Supplements (PG#: 20R01603 and 20R12709; PI:M.E.K.).Availability of data andmaterialsThe dataset used during the current study is available from the correspondingauthor upon request.DeclarationsEthics approval and consent to participateNot applicable.Consent for publicationNot applicable.Competing interestsMBH and LM declare no potential conflict of interests. MEK is supported by theMichael Smith Foundation for Health Research Scholar award (ID#: 17661).Over the past three years, MEK has received consulting fees from Biogen Inc.for consulting in projects unrelated to the current project.Author details1School of Population and Public Health, University of British Columbia,Vancouver, BC, Canada. 2Department of Statistics, University of BritishColumbia, Vancouver, BC, Canada. 3Centre for Health Evaluation and OutcomeSciences, St. Paul’s Hospital, Vancouver, BC, Canada.Received: 13 April 2021 Accepted: 14 January 2022References1. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes intherapeutical trials. J Chronic Dis. 1967;20(8):637–48. https://doi.org/10.1016/j.jclinepi.2009.01.012.2. Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375(5):454–63.https://doi.org/10.1056/NEJMra1510059.3. Chalmers TC, Celano P, Sacks HS, Smith Jr H. Bias in treatmentassignment in controlled clinical trials. N Engl J Med. 1983;309(22):1358–61. https://doi.org/10.1056/NEJM198312013092204.4. Hernan MA, Robins JM. Per-protocol analyses of pragmatic trials. N Engl JMed. 2017;377(14):1391–8.5. Ware JH, Hamel MB. Pragmatic trials-guides to better patient care. N EnglJ Med. 2011;364(18):1685–7. https://doi.org/10.1056/NEJMp1103502.6. Gupta SK. Intention-to-treat concept: a review. Perspect Clin Res.2011;2(3):109. https://doi.org/10.4103/2229-3485.83221.7. Toh S, Hernandez-Diaz S, Logan R, Robins JM, Hernan MA. Estimatingabsolute risks in the presence of nonadherence: an application to afollow-up study with baseline randomization. Epidemiology. 2010;21(4):528.8. Hernan MA, Scharfstein D. Cautions as regulators move to end exclusivereliance on intention to treat. Ann Intern Med. 2018. https://doi.org/10.7326/M17-3354.9. Hernan MA, Hernandez-Diaz S. Beyond the intention-to-treat incomparative effectiveness research. Clin Trials. 2012;9(1):48–55.10. Cook JA, MacLennan GS, Palmer T, Lois N, Emsley R. Instrumentalvariable methods for a binary outcome were used to informativelyaddress noncompliance in a randomized trial in surgery. J Clin Epidemiol.2018;96:126–32. https://doi.org/10.1016/j.jclinepi.2017.11.011.11. Chlebowski RT, Anderson GL, Gass M, Lane DS, Aragaki AK, Kuller LH,Manson JE, Stefanick ML, Ockene J, Sarto GE. Estrogen plus progestinand breast cancer incidence and mortality in postmenopausal women. JAm Med Assoc. 2010;304(15):1684–92. https://doi.org/10.1001/jama.2010.1500.12. Blewer AL, Putt ME, Becker LB, Riegel BJ, Li J, Leary M, Shea JA,Kirkpatrick JN, Berg RA, Nadkarni VM. Video-only cardiopulmonaryresuscitation education for high-risk families before hospital discharge: amulticenter pragmatic trial. Circ Cardiovasc Qual Outcomes. 2016;9(6):740–8. https://doi.org/10.1161/CIRCOUTCOMES.116.002493.13. Mash RJ, Rhode H, Zwarenstein M, Rollnick S, Lombard C, Steyn K,Levitt N. Effectiveness of a group diabetes education programme inunder-served communities in south africa: a pragmatic clusterrandomized controlled trial. Diabet Med. 2014;31(8):987–93. https://doi.org/10.1111/dme.12475.14. Little RJ, Long Q, Lin X. A comparison of methods for estimating thecausal effect of a treatment in randomized clinical trials subject tononcompliance. Biometrics. 2009;65(2):640–9.15. Baiocchi M, Cheng J, Small DS. Instrumental variable methods for causalinference. Stat Med. 2014;33(13):2297–340. https://doi.org/10.1002/sim.6128.16. Lobban F, Akers N, Appelbe D, Chapman L, Collinge L, Dodd S, FlowersS, Hollingsworth B, Johnson S, Jones SH. Clinical effectiveness of aweb-based peer-supported self-management intervention for relatives ofpeople with psychosis or bipolar (react): online, observer-blind,randomised controlled superiority trial. BMC Psychiatry. 2020;20:1–16.https://doi.org/10.1186/s12888-020-02545-9.17. Powell J, Williams V, Atherton H, Bennett K, Yang Y, Davoudianfar M,Hellsing A, Martin A, Mollison J, Shanyinde M. Effectiveness andcost-effectiveness of a self-guided internet intervention for social anxietysymptoms in a general population sample: Randomized controlled trial. JMed Internet Res. 2020;22(1):16804. https://doi.org/10.2196/16804.18. Manski CF. Nonparametric bounds on treatment effects. Am Econ Rev.1990;80(2):319–23.19. Palmer TM, Ramsahai RR, Didelez V, Sheehan NA. Nonparametricbounds for the causal effect in a binary instrumental-variable model. StataJ. 2011;11(3):345–67. https://doi.org/10.1177/1536867X1101100302.20. Balke A, Pearl J. Bounds on treatment effects from studies with imperfectcompliance. J Am Stat Assoc. 1997;92(439):1171–6.21. Murray EJ, Hernan MA. Improved adherence adjustment in the coronarydrug project. Trials. 2018;19(1):158. https://doi.org/10.1186/s13063-018-2519-5.22. Terza VJ. Two-stage residual inclusion estimation in health servicesresearch and health economics. Health Serv Res. 2018;53(3):1890–9.https://doi.org/10.1111/1475-6773.12714.23. Rassen JA, Schneeweiss S, Glynn RJ, Mittleman MA, Brookhart MA.Instrumental variable analysis for estimation of treatment effects withdichotomous outcomes. Am J Epidemiol. 2009;169(3):273–84. https://doi.org/10.1093/aje/kwn299.24. Angrist JD, Imbens GW, Rubin DB. Identification of causal effects usinginstrumental variables. J Am Stat Assoc. 1996;91(434):444–55.25. Danaei G, Rodríguez LAG, Cantero OF, Logan R, Hernan MA.Observational data for comparative effectiveness research: an emulationof randomised trials of statins and primary prevention of coronary heartdisease. Stat Methods Med Res. 2013;22(1):70–96.26. Young JG, Vatsa R, Murray EJ, Hernan MA. Interval-cohort designs andbias in the estimation of per-protocol effects: a simulation study. Trials.2019;20(1):552.27. Mosquera L. Exploring inverse probability weighted per-protocolestimates to adjust for non-adherence using post-randomizationHossain et al. BMCMedical ResearchMethodology (2022) 22:46 Page 46 of 46covariates: a simulation study MSc thesis, University of British Columbia.2020. http://hdl.handle.net/2429/75673.28. Garrido MM, Deb P, Burgess Jr JF, Penrod JD. Choosing models forhealth care cost analyses: issues of nonlinearity and endogeneity. HealthServ Res. 2012;47(6):2377–97.29. Wang F, Meyer NJ, Walley KR, Russell JA, Feng R. Causal geneticinference using haplotypes as instrumental variables. Genet Epidemiol.2016;40(1):35–44.30. Hernan MA, Robins JM. Estimating causal effects from epidemiologicaldata. J Epidemiol Community Health. 2006;60(7):578–86. https://doi.org/10.1136/jech.2004.029496.31. Dunn G, Maracy M, Tomenson B. Estimating treatment effects fromrandomized clinical trials with noncompliance and loss to follow-up: therole of instrumental variable methods. Stat Methods Med Res. 2005;14(4):369–95. https://doi.org/10.1191/0962280205sm403oa.32. McNamee R. Confounding and confounders. Occup Environ Med.2003;60(3):227–34.33. Lederer DJ, Bell SC, Branson RD, Chalmers JD, Marshall R, Maslove DM,Ost DE, Punjabi NM, Schatz M, Smyth AR, et al. Control of confoundingand reporting of results in causal inference studies. guidance for authorsfrom editors of respiratory, sleep, and critical care journals. Ann AmThorac Soc. 2019;16(1):22–8.34. VanderWeele TJ. Principles of confounder selection. Eur J Epidemiol.2019;34(3):211–9.35. Textor J, van der Zander B, Gilthorpe MS, Lis´kiewicz M, Ellison GT.Robust causal inference using directed acyclic graphs: the r package‘dagitty’. Int J Epidemiol. 2016;45(6):1887–94.36. Hernán MA, Robins JM. Instruments for causal inference: anepidemiologist’s dream?. Epidemiology. 2006;17(4):360–72.37. Brookhart MA, Wang P, Solomon DH, Schneeweiss S. Evaluatingshort-term drug effects using a physician-specific prescribing preferenceas an instrumental variable. Epidemiology. 2006;17(3):268.38. Greenland S, Pearl J, Robins JM. Confounding and collapsibility in causalinference. Stat Sci. 1999;14(1):29–46.39. Kaufman JS. Marginalia: comparing adjusted effect measures.Epidemiology. 2010;21(4):490–3.40. Naimi AI, Whitcomb BW. Estimating risk ratios and risk differences usingregression. Am J Epidemiol. 2020;189(6):508–10.41. Booth FW, Roberts CK, Laye MJ. Lack of exercise is a major cause ofchronic diseases. Compr Physiol. 2012;2(2):1143.42. Garcia-Aymerich J, Lange P, Benet M, Schnohr P, Antó JM. Regularphysical activity modifies smoking-related lung function decline andreduces risk of chronic obstructive pulmonary disease: a population-basedcohort study. Am J Respir Crit Care Med. 2007;175(5):458–63.43. Crothers K, Griffith TA, McGinnis KA, Rodriguez-Barradas MC, Leaf DA,Weissman S, Gibert CL, Butt AA, Justice AC. The impact of cigarettesmoking on mortality, quality of life, and comorbid illness amonghiv-positive veterans. J Gen Intern Med. 2005;20(12):1142–5.44. Cohen S, Tyrrell D, Russell MA, Jarvis MJ, Smith AP. Smoking, alcoholconsumption, and susceptibility to the common cold. Am J Public Health.1993;83(9):1277–83.45. Schmid P, Rauber D, Betsch C, Lidolt G, Denker M-L. Barriers of influenzavaccination intention and behavior–a systematic review of influenzavaccine hesitancy, 2005–2016. Plos ONE. 2017;12(1):0170550.46. Hirano K, Imbens GW, Rubin DB, Zhou X-H. Assessing the effect of aninfluenza vaccine in an encouragement design. Biostatistics. 2000;1(1):69–88.47. Morris TP, White IR, Crowther MJ. Using simulation studies to evaluatestatistical methods. Stat Med. 2019;38(11):2074–102. https://doi.org/10.1002/sim.8086.48. Sommer A, Djunaedi E, Loeden A, Tarwotjo I, West JR K, Tilden R, MeleL, Group AS, et al. Impact of vitamin A supplementation on childhoodmortality: a randomised controlled community trial. Lancet.1986;327(8491):1169–73.49. Sommer A, Zeger SL. On estimating efficacy from clinical trials. Stat Med.1991;10(1):45–52.50. McNamee R. Intention to treat, per protocol, as treated and instrumentalvariable estimators given non-compliance and effect heterogeneity. StatMed. 2009;28(21):2639–52.51. Bang H, Davis CE. On estimating treatment effects under non-compliancein randomized clinical trials: are intent-to-treat or instrumental variablesanalyses perfect solutions?. Stat Med. 2007;26(5):954–64.52. Nagelkerke N, Fidler V, Bernsen R, Borgdorff M. Estimating treatmenteffects in randomized clinical trials in the presence of non-compliance.Stat Med. 2000;19(14):1849–64.53. NICE. Guide to themethods of technology appraisal. Technical report. 2013.54. Hernán MA, Robins JM. Confounding. In: Causal Inference: What If. BocaRaton: Chapman and Hall/CRC; 2020.55. Terza VJ, Basu A, Rathouz PJ. Two-stage residual inclusion estimation:addressing endogeneity in health econometric modeling. J Health Econ.2008;27(3):531–43. https://doi.org/10.1016/j.jhealeco.2007.09.009.56. Bound J, Jaeger DA, Baker RM. Problems with instrumental variablesestimation when the correlation between the instruments and theendogenous explanatory variable is weak. J Am Stat Assoc. 1995;90(430):443–50. https://doi.org/10.2307/2291055.57. Small DS, Rosenbaum PR. War and wages: the strength of instrumentalvariables and their sensitivity to unobserved biases. J Am Stat Assoc.2008;103(483):924–33. https://doi.org/10.1198/016214507000001247.58. Cole SR, Hernan MA. Constructing inverse probability weights formarginal structural models. Am J Epidemiol. 2008;168(6):656–64. https://doi.org/10.1093/aje/kwn164.59. Hernán MA, Hernández-Díaz S, Robins JM. A structural approach toselection bias. Epidemiology. 2004;15(5):615–25.60. Moerbeek M, Schie S. v. What are the statistical implications of treatmentnon-compliance in cluster randomized trials: a simulation study. StatMed. 2019;38(26):5071–84.61. Ye C, Beyene J, Browne G, Thabane L. Estimating treatment effects inrandomised controlled trials with non-compliance: a simulation study.BMJ Open. 2014;4(6):005362.62. Gao X, Zheng M. Estimating the causal effects in randomized trials forsurvival data with a cure fraction and non compliance. Commun StatTheory Methods. 2017;46(8):4065–87.63. Nie H, Cheng J, Small DS. Inference for the effect of treatment on survivalprobability in randomized trials with noncompliance and administrativecensoring. Biometrics. 2011;67(4):1397–405.64. Latimer N, Abrams K, Siebert U. Two-stage estimation to adjust fortreatment switching in randomised trials: a simulation study investigatingthe use of inverse probability weighting instead of re-censoring. BMCMed Res Methodol. 2019;19(1):1–19.65. Cai B, Small DS, Have TRT. Two-stage instrumental variable methods forestimating the causal odds ratio: analysis of bias. Stat Med. 2011;30(15):1809–24.66. Uddin M, Groenwold R, de Boer Ton A., Belitser S, Roes K, Klungel O.Instrumental variable analysis in epidemiologic studies: an overview ofthe estimation methods. Pharma Analytica Acta. 2015;6(353):2.67. Robins J, Rotnitzky A. Estimation of treatment effects in randomised trialswith non-compliance and a dichotomous outcome using structuralmean models. Biometrika. 2004;91(4):763–83.68. Westreich D, Cole SR. Positivity in practice. Am J Epidemiol. 2010;171(6):674–7.69. Platt RW, Delaney JAC, Suissa S. The positivity assumption and marginalstructural models: the example of warfarin use and risk of bleeding. Eur JEpidemiol. 2012;27(2):77–83.70. Crump RK, Hotz VJ, Imbens GW, Mitnik OA. Dealing with limited overlapin estimation of average treatment effects. Biometrika. 2009;96(1):187–99.71. Li F, Morgan KL, Zaslavsky AM. Balancing covariates via propensity scoreweighting. J Am Stat Assoc. 2018;113(521):390–400.72. Lee Y, Kennedy EH, Mitra N. Doubly robust nonparametric instrumentalvariable estimators for survival outcomes. Biostatistics. 2021. Publishedonline: 17 Oct 2021. https://doi.org/10.1093/biostatistics/kxab036.73. Choi BY. Instrumental variable estimation of truncated local averagetreatment effects. Plos ONE. 2021;16(4):0249642.74. Cole SR, Frangakis CE. The consistency statement in causal inference: adefinition or an assumption?. Epidemiology. 2009;20(1):3–5. https://doi.org/10.1097/EDE.0b013e31818ef366.75. Hernan MA. Beyond exchangeability: the other conditions for causalinference in medical research. Stat Methods Med Res. 2012;21(1):3–5.https://doi.org/10.1177/0962280211398037.76. Vansteelandt S, Bekaert M, Claeskens G. On model selection and modelmisspecification in causal inference. Stat Methods Med Res. 2012;21(1):7–30. https://doi.org/10.1177/0962280210387717.77. Brookhart MA, Rassen JA, Schneeweiss S. Instrumental variable methodsin comparative safety and effectiveness research. PharmacoepidemiolDrug Saf. 2010;19(6):537–54. https://doi.org/10.1002/pds.1908.78. Hernan MA, Robins JM. Selection bias. In: Causal Inference: What If. BocaRaton: Chapman and Hall/CRC; 2020.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.
Article
eng
Reviewed
Vancouver : University of British Columbia Library
BioMed Central
Attribution 4.0 International (CC BY 4.0)
Faculty
Researcher
Pragmatic trials
Nonadherence
Simulation
Unmeasured confounding
Analysis approaches to address treatment nonadherence in pragmatic trials with point-treatment settings: a simulation study
Text