Medicine, Faculty of
Population and Public Health (SPPH), School of
DSpace
UBCV
Zhan, Denghuang
Doctor of Philosophy - PhD
University of British Columbia
A learning health system (LHS) is a framework designed to generate and apply evidence for improving healthcare decisions and outcomes. In practice, bridging the gap from data generation to evidence is challenging, often constrained by the lack of statistical methodologies to utilize available LHS data optimally. This dissertation contributes to methodologies for embedding clinical trials into learning health systems, addressing challenges in adaptive trial designs, real-world data integration, and personalized medicine. First, we developed an R package, BayesAET, to evaluate Bayesian adaptive enrichment designs via simulation, allowing for early termination, flexible randomization, and incorporation of external information. This tool supports efficient trial designs tailored to subpopulations, facilitating more personalized treatment recommendations. Second, we investigated the properties of Bayesian group-sequential noninferiority trials, highlighting trade-offs in power, sample size, and interoperability compared to Frequentist designs. Using plasmode simulations based on a real-world dataset, the study demonstrated Bayesian groupsequential designs reduce expected sample sizes significantly but careful consideration is required to plan the number of interim analyses to optimize the specific design. We found that in our test scenarios, response adaptive randomization (RAR) resulted in more participants being randomized to the inferior treatment in group-sequential non-inferiority trials, emphasizing the need for careful evaluation of RAR’s appropriateness in such settings. Third, we introduced a novel method for efficiently constructing a prior distribution for a treatment effect given two datasets in which confounding is present in one dataset due to missing covariates. This method demonstrates comparable performance to Bayesian MCMC approaches while significantly improving computational efficiency, making it practical for evolving LHS environments. Lastly, we conducted a scoping review of methodologies for combining randomized and observational data to estimate Conditional Average Treatment Effects (CATE), and evaluated the performance of three representative approaches via simulation, highlighting their unique strengths and limitations depending on confounding levels, covariate overlap, and sample sizes. The scoping review and these comparisons provide practical guidance for researchers seeking to combine diverse data sources effectively in precision medicine research.
DEVELOPING STATISTICALMETHODOLOGIES TO SUPPORTEMBEDDING CLINICAL TRIALSWITHIN LEARNING HEALTHSYSTEMSbyDenghuang ZhanB.Eng., Huazhong University of Science and Technology, 2014M.E.Sc., Western University, 2017A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe Faculty of Graduate and Postdoctoral Studies(Population and Public Health)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)April 2025© Denghuang Zhan 2025The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:Developing statistical methodologies to support embeddingclinical trials within learning health systemssubmitted by Denghuang Zhan in partial fulfillment of the requirementsfor the degree of Doctor of Philosophyin Population and Public HealthExamining Committee:Hubert Wong, Associate Professor, School of Population and Public Health,UBCSupervisorFidel Vila-Rodriguez, Associate Professor, Department of Psychiatry, UBCCo-supervisorMohammad Ehsanul (Ehsan) Karim, Assistant Professor, School of Popu-lation and Public Health, UBCSupervisory Committee MemberJoel Singer, Professor, School of Population and Public Health, UBCUniversity ExaminerRollin Brant, Professor, Department of Statistics, UBCUniversity ExaminerGeorge Tomlinson, Senior Scientist, Toronto General Hospital Research In-stitute, University Health NetworkExternal ExamineriiAbstractA learning health system (LHS) is a framework designed to generate and ap-ply evidence for improving healthcare decisions and outcomes. In practice,bridging the gap from data generation to evidence is challenging, often con-strained by the lack of statistical methodologies to utilize available LHS dataoptimally. This dissertation contributes to methodologies for embeddingclinical trials into learning health systems, addressing challenges in adaptivetrial designs, real-world data integration, and personalized medicine.First, we developed an R package, BayesAET, to evaluate Bayesian adap-tive enrichment designs via simulation, allowing for early termination, flex-ible randomization, and incorporation of external information. This toolsupports efficient trial designs tailored to subpopulations, facilitating morepersonalized treatment recommendations.Second, we investigated the properties of Bayesian group-sequential non-inferiority trials, highlighting trade-offs in power, sample size, and inter-pretability compared to Frequentist designs. Using plasmode simulationsbased on a real-world dataset, the study demonstrated Bayesian group-sequential designs reduce expected sample sizes significantly but careful con-sideration is required to plan the number of interim analyses to optimize thespecific design. We found that in our test scenarios, response adaptive ran-domization (RAR) resulted in more participants being randomized to theinferior treatment in group-sequential non-inferiority trials, emphasizing theneed for careful evaluation of RAR’s appropriateness in such settings.Third, we introduced a novel method for efficiently constructing a priordistribution for a treatment effect given two datasets in which confounding ispresent in one dataset due to missing covariates. This method demonstratescomparable performance to Bayesian MCMC approaches while significantlyiiiimproving computational efficiency, making it practical for evolving LHSenvironments.Lastly, we conducted a scoping review of methodologies for combiningrandomized and observational data to estimate Conditional Average Treat-ment Effects (CATE), and evaluated the performance of three representativeapproaches via simulation, highlighting their unique strengths and limita-tions depending on confounding levels, covariate overlap, and sample sizes.The scoping review and these comparisons provide practical guidance forresearchers seeking to combine diverse data sources effectively in precisionmedicine research.ivLay SummaryThis work develops mathematical tools to help design clinical trials thatare embedded in a learning health system. It focuses on 3 challenges toimproving a trial. First, how can we change a trial while it is in progress toreduce the number of participants needed to answer the research question.Second, how can we use data from outside the trial? Third, how can wedesign the trial to help us identify the best treatment for a given individual?We developed software to evaluate how well trial designs that address thesequestions. We conducted simulations to identify guidelines for improvingtrial designs. We also developed a novel method for combining datasets whensome of the datasets are missing important variables. Finally, we comparedexisting methods for combining randomized trial data with observationaldata. These results will help in choosing what treatment works best for anindividual.vPrefaceThis dissertation was completed under the supervision of Dr. Hubert Wongand Dr. Fidel Vila-Rodriguez at the University of British Columbia, Van-couver Campus. No generative AI was involved in any part of this research.The incorporation of external information into trials within a learninghealth system was motivated in part by work I published previously on en-hancing trial efficiency through the use of informative priors on time effectsin stepped-wedge cluster randomized trials (“Zhan, D., Ouyang, Y., Xu,L., & Wong, H., 2021. Improving efficiency in the stepped-wedge trial de-sign via Bayesian modeling with an informative prior for the time effects.Clinical Trials, 18(3), pp.295-302”). This dissertation addresses broadermethodological issues encountered in the embedding of clinical trials intolearning health systems.A version of Chapter 2 has been submitted for peer review as Zhan, D.,Ouyang, Y., Vila-Rodriguez, F., Karim, M.E. & Wong, H. Bayesian adap-tive enrichment design in multi-arm clinical trials: The BayesAET packagefor R users. I was the first and corresponding author, responsible for all as-pects of the manuscript, including drafting and revising the text, developingand evaluating the package code, and managing and testing the software.Dr. Hubert Wong and Dr. Fidel Vila-Rodriguez helped me develop theideas. My committee member, Dr. Mohammad Ehsanul Karim, assistedin creating the framework for developing the Shiny app. Dr. YongdongOuyang assisted with code evaluation. Dr. Hubert Wong, Dr. MohammadEhsanul Karim, and Dr. Yongdong Ouyang provided valuable feedback andcontributed to the revisions.A version of Chapter 3 will be submitted for peer review as Zhan, D.,Ouyang, Y., Vila-Rodriguez, F., Karim, M.E. & Wong, H. Impact of de-visign choices on power and sample size in a Bayesian group-sequential non-inferiority trial: An illustration using plasmode simulation. I was the firstand corresponding author and was responsible for conducting simulations,drafting, and revising the manuscript. Dr. Hubert Wong and Dr. FidelVila-Rodriguez helped me develop the ideas. Dr. Hubert Wong, Dr. Fi-del Vila-Rodriguez, Dr. Yongdong Ouyang and my committee member, Dr.Mohammad Ehsanul Karim provided valuable feedback and contributed tothe revisions. Dr. Fidel Vila-Rodriguez was also the PI for the exampletrial.I was the lead author for the work presented in Chapter 4, responsiblefor deriving formulas, conducting simulations, and drafting and revising thechapter. Dr. Hubert Wong initiated the research idea, assisted with formulaverification, and contributed to the revisions. My committee member, Dr.Mohammad Ehsanul Karim, provided valuable suggestions.I was the lead author for the work presented in Chapter 5 and wasresponsible for conducting the literature review, drafting and revising thechapter. Dr. Hubert Wong and Dr. Fidel Vila-Rodriguez initiated theresearch idea, and contributed to the revision. Dr. Ouyang Yongdong andMy committee member, Dr. Mohammad Ehsanul Karim, provided valuablesuggestions.I was the lead author for the work presented in Chapter 6, responsiblefor conducting the literature review, conducting simulations, drafting andrevising the chapter. Dr. Hubert Wong and Dr. Fidel Vila-Rodriguezinitiated the research idea and contributed to the revisions. My committeemember, Dr. Mohammad Ehsanul Karim, provided valuable suggestions.viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . xxAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Overview of embedding clinical trials within learning healthsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.1 Definition and core principles of learning health sys-tems . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.2 Addressing challenges in bridging data to evidencewithin LHS: A case study perspective . . . . . . . . . 11.1.3 Towards a practical framework for adaptive trials inLHS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Study rationale and thesis outline . . . . . . . . . . . . . . . 3viii1.2.1 Design and evaluate Bayesian adaptive trials withinLHS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2.2 Constructing a prior distribution for a treatment ef-fect from two observational datasets when known con-founders are not measured in one dataset . . . . . . . 71.2.3 Combining RCT and observational data for individu-alized treatment effect (or CATE) . . . . . . . . . . . 82 Bayesian adaptive enrichment design in multi-arm clinicaltrials: The BayesAET package for R users . . . . . . . . . 112.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 BayesAET package description . . . . . . . . . . . . . . . . . 142.2.1 Overview of the design logics and adaptations . . . . 142.2.2 Model framework . . . . . . . . . . . . . . . . . . . . 202.3 Practical example . . . . . . . . . . . . . . . . . . . . . . . . 282.3.1 Simulation settings . . . . . . . . . . . . . . . . . . . 282.3.2 Simulation outputs . . . . . . . . . . . . . . . . . . . 332.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 Impact of design choices on power and sample size in aBayesian group-sequential non-inferiority trial: An illustra-tion using plasmode simulation . . . . . . . . . . . . . . . . . 433.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.2.1 Plasmode simulation based on the THREE-D study . 453.2.2 Key design features for Bayesian adaptive trial . . . . 463.3 Simulation settings and results . . . . . . . . . . . . . . . . . 493.3.1 Fixed maximum sample size . . . . . . . . . . . . . . 513.3.2 Fixed target power . . . . . . . . . . . . . . . . . . . 523.3.3 Impacts of responsive adaptive randomization (RAR)strategies . . . . . . . . . . . . . . . . . . . . . . . . . 543.3.4 Impact of prior information under balanced random-ization (BR) . . . . . . . . . . . . . . . . . . . . . . . 55ix3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584 Constructing a prior distribution for a treatment effect fromtwo observational datasets when known confounders are notmeasured in one dataset . . . . . . . . . . . . . . . . . . . . . 624.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 624.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.2.1 Problem setup and derivations of some key estimators: 644.3 Simulation setups and results . . . . . . . . . . . . . . . . . . 714.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 755 Combining RCT and observational data to estimate condi-tional average treatment effect—a scoping review . . . . . 785.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 785.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 805.2.1 Searching . . . . . . . . . . . . . . . . . . . . . . . . . 805.2.2 Study selection and data extraction . . . . . . . . . . 815.2.3 Notations and assumptions . . . . . . . . . . . . . . . 825.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835.3.1 Assuming no unmeasured confounding in the OBSdataset . . . . . . . . . . . . . . . . . . . . . . . . . . 855.3.2 Allowing unmeasured confounding in the OBS dataset 875.3.3 Performance comparisons . . . . . . . . . . . . . . . . 925.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 936 Comparing methods for combining trial data and observa-tional data to estimate conditional average treatment effects 966.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 966.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 976.2.1 Notations and assumptions . . . . . . . . . . . . . . . 976.2.2 A review of several existing methods . . . . . . . . . 996.3 Simulation experiments . . . . . . . . . . . . . . . . . . . . . 1056.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1086.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112x7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1147.1 Summary of findings and implications . . . . . . . . . . . . . 1147.1.1 R Package Development: Bayesian Adaptive Enrich-ment Design for LHS . . . . . . . . . . . . . . . . . . 1157.1.2 Impact of design choices on power and sample size inBayesian group-sequential non-inferiority trial . . . . 1167.1.3 Prior Correction Method for Combining ObservationalData from the Same Underlying Population . . . . . 1187.1.4 Combining Observational and Trial Data for CATEEstimation and Impact on Precision Medicine . . . . 1187.2 Suggestions for future work . . . . . . . . . . . . . . . . . . . 1217.3 Knowledge Translation . . . . . . . . . . . . . . . . . . . . . 123Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124AppendicesA Supporting information for Chapter 2 . . . . . . . . . . . . . 143A.1 The input and output descriptions of the major functions . . 143A.1.1 Example run of function BAET.sim . . . . . . . . . . 143A.1.2 Details of function Multi.BAET . . . . . . . . . . . . 145A.2 Data Generative Process . . . . . . . . . . . . . . . . . . . . 149A.3 Modeling binary and count outcomes through multivariatenormal joint prior . . . . . . . . . . . . . . . . . . . . . . . . 151B Supporting information for Chapter 3 . . . . . . . . . . . . . 153B.1 Details of the proposed Bayesian adaptive design . . . . . . . 153B.1.1 Bayesian adaptive design features and inputs . . . . . 153B.1.2 The Bayesian adaptive design outputs . . . . . . . . . 155C Supporting information for Chapter 4 . . . . . . . . . . . . . 157C.1 Derivations of Var(µc) . . . . . . . . . . . . . . . . . . . . . . 157C.2 Derivations of Var(µw) . . . . . . . . . . . . . . . . . . . . . 161C.3 Additional simulation results . . . . . . . . . . . . . . . . . . 162xiD Supporting information for Chapter 6 . . . . . . . . . . . . . 165D.1 More results of percentage differences between the three meth-ods and the true CATE . . . . . . . . . . . . . . . . . . . . . 165D.2 Simulation results for 3 R-learner methods . . . . . . . . . . 167xiiList of Tables2.1 Input arguments and descriptions for function BAET.sim . . . 162.2 Outputs and descriptions for function BAET.sim . . . . . . . . 192.3 The data generation related parameters inputs . . . . . . . . 292.4 The design related parameter inputs . . . . . . . . . . . . . . 302.5 Simulation scenarios with different combinations of prior mean,prior covariance, and randomization strategy . . . . . . . . . 313.1 The percentage of simulated trials that stopped after eachinterim analysis, along with the percentage that stopped dueto declaring non-inferiority or (incorrectly) declaring inferior-ity, when conducting interim looks at every 100 samples (aninterim look of 3) with maximum sample size of 385. Theupper threshold U is adjusted to be 0.9828 to maintain theoverall Type I error rate at 0.05. . . . . . . . . . . . . . . . . 503.2 After tuning the U to get a desired overall Type I error rate(0.05), the corresponding Treatment effect estimation, thepower, and the average sample size of different designs. Theresults are from 30000 runs. . . . . . . . . . . . . . . . . . . 513.3 After tuning the maximum sample size and U to get a desiredoverall Type I error rate (0.05), the target power (0.91), thecorresponding Treatment effect estimation, and the averagesample size of different designs. The results are from 30000runs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53xiii3.4 Effect estimation, sample size (SS), power, and Type I er-ror rate under responsive adaptive randomization (RAR) andbalanced randomization (BR) with 3 interim looks (every 100samples). The upper threshold U is adjusted to 0.9811 tomaintain the overall Type I error rate at 0.05 under RAR. . . 553.5 Effect estimation, sample size (SS), power, and Type I errorrate with varying prior variance from 100 to 4 using balancedrandomization (BR) with 3 interim looks (every 100 samples).The prior mean is correctly specified and assumes no correla-tion between the two treatments. . . . . . . . . . . . . . . . . 573.6 Effect estimation, sample size (SS), power, and Type I errorrate with varying prior covariance from 0 to 3 using balancedrandomization (BR) with 3 interim looks (every 100 samples).The prior mean is correctly specified, and the prior varianceis fixed at 4 (i.e., σ11 = σ22 = 4). . . . . . . . . . . . . . . . . 574.1 Simulation performances of the 3 methods when there is 90%of old data, a = b = 0.2 . . . . . . . . . . . . . . . . . . . . . 724.2 Simulation performances of the 3 methods when there is 50%of old data, a = b = 0.2 . . . . . . . . . . . . . . . . . . . . . 724.3 Simulation performances of the 3 methods when there is 10%of old data, a = b = 0.2. . . . . . . . . . . . . . . . . . . . . . 734.4 Simulation performances of the 3 methods when there is 90%of old data, a = 0.8, b = 0.2. . . . . . . . . . . . . . . . . . . . 745.1 Summary of each paper in terms of the assumptions and thekey approach/motivation. . . . . . . . . . . . . . . . . . . . . 856.1 Summary of the major differences among the three approaches1006.2 The averaged RMSE for the 3 methods based on (S1a) pro-cedure to generate RCT samples while the OBS sample sizem is 500 with the corresponding RCT sample size n is 200.The results are based on 5000 simulation runs. . . . . . . . . 110xiv6.3 The averaged RMSE for the 3 methods based on (S1a) pro-cedure to generate RCT samples while the OBS sample sizem is 2000 with the corresponding RCT sample size n is 200.The results are based on 5000 simulation runs. . . . . . . . . 1106.4 The averaged RMSE for the 3 methods based on (S1b) pro-cedure to generate RCT samples, with n = 200 and m = 500.The results are based on 5000 simulation runs. . . . . . . . . 1116.5 The averaged RMSE for the 3 methods based on the secondprocedure to generate RCT samples while the OBS samplesize m is 500, 1000, and 2000 with the corresponding RCTsample size n fixed at 200. The results are based on 5000simulation runs. . . . . . . . . . . . . . . . . . . . . . . . . . . 111A.1 Outputs and descriptions for function Multi.BAET . . . . . . 147C.1 The performance of bias-corrected estimator under simulationsetting S1–S3 where a = b = 0.2 . . . . . . . . . . . . . . . . . 162C.2 Simulation performances of the 3 methods when there is 90%of old data, a = 0.2, and b = 2. . . . . . . . . . . . . . . . . . 162C.3 Simulation performances of the 3 methods when there is 50%of old data, a = 0.2, and b = 2. . . . . . . . . . . . . . . . . . 162C.4 Simulation performances of the 3 methods when there is 10%of old data, a = 0.2, and b = 2. . . . . . . . . . . . . . . . . . 163C.5 Simulation performances of the 3 methods when there is 90%of old data, a = 0.2, and b = 20. . . . . . . . . . . . . . . . . 163C.6 Simulation performances of the 3 methods when there is 50%of old data, a = 0.2, and b = 20. . . . . . . . . . . . . . . . . 163C.7 Simulation performances of the 3 methods when there is 10%of old data, a = 0.2, and b = 20. . . . . . . . . . . . . . . . . 164C.8 Simulation performances of the 3 methods when there is 90%of old data, a = 1.2, and b = 0.2. . . . . . . . . . . . . . . . . 164xvD.1 The averaged RMSE for the 3 R-learner methods based on(1a) procedure to generate RCT samples while the OBS sam-ple size m is 500 with the corresponding RCT sample size nis 200. The results are based on 5000 simulation runs. . . . . 168D.2 The averaged RMSE for the 3 R-learner methods based on(1a) procedure to generate RCT samples while the OBS sam-ple size m is 2000 with the corresponding RCT sample size nis 200. The results are based on 5000 simulation runs. . . . . 168D.3 The averaged RMSE for the 3 R-learner methods based on(1b) procedure to generate RCT samples, with n = 200 andm = 500. The results are based on 5000 simulation runs. . . . 168D.4 The averaged RMSE for the 6 methods based on the secondprocedure to generate RCT samples while the OBS samplesize m is 500, 1000, and 2000 with the corresponding RCTsample size n fixed at 200. The results are based on 5000simulation runs. . . . . . . . . . . . . . . . . . . . . . . . . . . 169xviList of Figures1.1 An illustration of a learning health system (LHS). LHS func-tions as a continuous cycle, converting practice into data,data into evidence, and evidence back into practice, all whilesupporting ongoing capacity-building efforts. . . . . . . . . . 22.1 Overview of the adaptive enrichment logic and proceduresembedded in the ‘BayesAET ’ R package. . . . . . . . . . . . 152.2 The simulation results under 4 different settings with the cor-relation between different treatments within each subpopu-lation changing from 0 to 0.6. (A) and (B) show the besttreatment (treatment 3) effect size estimation in Non-benzosubpopulation and Benzo subpopulation (the trajectory of“Ms-m, BR” overlaps with that of “Ms-m, RAR”). (C) and(D) show the subpopulation power in Non-benzo subpopula-tion and Benzo subpopulation. (E), (F) and (G) show the ex-pected sample size in Non-Benzo subpopulation, Benzo sub-population and overall population, respectively. . . . . . . . 382.3 Overall sample size distribution for simulation scenario S1. . 393.1 The tuning process for controlling the overall Type I error rateand target power in Bayesian group-sequential design. Thereare three key design features that could affect the power andthe Type I error rate of our Bayesian adaptive design. . . . . 48xvii3.2 The power and average sample size with different number ofinterim looks while fixing the maximum sample size at 385.The red curve indicates the power while blue curve indicatesthe expected sample size. . . . . . . . . . . . . . . . . . . . . 523.3 Maximum sample size and the corresponding expected samplesize with the change of the number of interim looks whenpower is fixed at around 0.91. The red curve indicates themaximum sample size required while the blue curve indicatesthe expected sample size accordingly. . . . . . . . . . . . . . 544.1 The computation speed of the proposed HP method comparedto MCMC when the missing rate is at 90%. . . . . . . . . . 755.1 PRISMA flow chart of identification, screening, eligibility,and inclusion of studies. . . . . . . . . . . . . . . . . . . . . . 846.1 Percentage differences between the three methods and thetrue CATE under simulation setting S1a with OBS size m =500 and b = 0.5. In this scenario, X2 is held constant at 1. . . 109A.1 A snapshot of the main function ‘BAET.sim’ and the requireduser inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143A.2 A snapshot of the corresponding ‘BAET.sim‘ outputs usingthe input values - Part 1. . . . . . . . . . . . . . . . . . . . . 144A.3 A snapshot of the corresponding ‘BAET.sim‘ outputs usingthe input values - Part 2. . . . . . . . . . . . . . . . . . . . . 145A.4 A snapshot of the wrapper function ‘Multi.BAET’ and therequired user inputs . . . . . . . . . . . . . . . . . . . . . . . 146A.5 The corresponding outputs with the input values from thewrapper function ‘Multi.BAET’ . . . . . . . . . . . . . . . . 148D.1 Percentage differences between the three methods and thetrue CATE under simulation setting S1a with OBS size m =500 and b = 1. In this scenario, X2 is held constant at 1. . . . 166xviiiD.2 Percentage differences between the three methods and thetrue CATE under simulation setting S1a with OBS size m =500 and b = 2.5. In this scenario, X2 is held constant at 1. . . 167xixList of Abbreviations• ATE: average treatment effects• BR: balanced randomization• CATE: conditional average treatment effects• HFL: high-frequency left-sided stimulation• HP: honest precision posterior• HRSD-17: Hamilton Rating Scale for Depression• IPW: inverse probability weighting• LHS: learning health system• MAMS: multi-arm multi-stage• MCMC: Markov Chain Monte Carlo• MI: multiple imputation• MSE: mean squared error• NP: Na¨ıve posterior• OBS: observational datasets• RAR: response adaptive randomization• RCT: randomized controlled trial• RMSE: root mean squared errorxx• rTMS: repetitive transcranial magnetic stimulation• SES: semiparametric efficiency score• SS: sample size• TBS: theta burst stimulationxxiAcknowledgmentsThis dissertation research would not have been possible without the unwa-vering support of my supervisors, Dr. Hubert Wong and Dr. Fidel Vila-Rodriguez.I would like to express my heartfelt gratitude to Dr. Hubert Wong for hiscareful guidance, mentorship, continuous encouragement, insightful sugges-tions, and immense patience throughout my academic journey since 2018.His support enabled me to transition seamlessly from an engineering back-ground to the field of statistics, shaping my career and, most importantly,my personal growth. I would also like to express my immense gratitudeto Dr. Fidel Vila-Rodriguez for his invaluable guidance, unwavering sup-port, and encouragement. He provided an excellent research environmentand numerous opportunities for me to explore and apply my work, helpingme grow as a researcher. I am profoundly grateful to Dr. Hubert Wongand Dr. Fidel Vila-Rodriguez for fostering this exceptional theoretical andclinical research environment; their guidance and encouragement have beeninstrumental in shaping my journey and bringing me to where I am today.I am deeply thankful to my supervisory committee member, Dr. Moham-mad Ehsanul Karim, for his invaluable assistance and insightful suggestionsthroughout my studies. His timely feedback and insights significantly en-riched and refined my research. I am also deeply grateful for the enrichinginteractions and collaborations with my passionate and insightful colleague,Dr. Yongdong Ouyang.Besides, I wish to thank all of my colleagues and friends, for their tremen-dous support, patience, and encouragement throughout my graduate studies.I would like to acknowledge that the projects in Chapters 2 and 3 arefunded by Canadian Institutes of Health Research (CIHR) and CanadianxxiiConsortium of Clinical Training Platform – Clinical Trials Trainings Pro-gram (CANTRAIN-CTTP). Chapters 2, 3, 4, and 6 were enabled in partby computational resources provided by the Digital Research Alliance ofCanada (https://alliancecan.ca/en).Lastly, my deepest gratitude goes to my parents and my girlfriend, whoseunconditional love and unwavering support have always encouraged me topursue my aspirations.xxiiiDedicationThis dissertation is dedicated to my parentsfor their unconditional love and support.xxivChapter 1Introduction1.1 Overview of embedding clinical trials withinlearning health systems1.1.1 Definition and core principles of learning healthsystemsA learning health system (LHS), is defined as one “designed to generateand apply the best evidence for the collaborative healthcare choices of eachpatient and provider; to drive the process of discovery as a natural out-growth of patient care; and to ensure innovation, quality, safety, and valuein healthcare” by the Institute of Medicine [1, 2]. LHSs are emerging aroundthe world and aim to leverage advancements in science, technology and prac-tice to improve health system performance at lower cost [3, 4]. As indicatedin Figure 1.1, LHS operates on a cyclical process that transforms practiceinto data, data into evidence, and evidence back into practice, with ongoingcapacity-building activities.1.1.2 Addressing challenges in bridging data to evidencewithin LHS: A case study perspectiveDespite the promise of LHSs, bridging the gap from data generation to ev-idence remains a significant challenge in practice, especially as efforts toachieve these goals are often constrained by the lack of statistical method-ologies that can make optimal use of available data sources within the LHS[4].This thesis is motivated by the therapeutic challenges faced by patientswith treatment-resistant depression (TRD) [5–8] and examines the role of1Figure 1.1: An illustration of a learning health system (LHS). LHS functionsas a continuous cycle, converting practice into data, data into evidence, andevidence back into practice, all while supporting ongoing capacity-buildingefforts.repetitive transcranial magnetic stimulation (rTMS) in addressing thesechallenges. rTMS is a non-invasive, non-pharmacological treatment thatuses electromagnetic induction to stimulate targeted brain regions, poten-tially modifying brain activity to alleviate symptoms [9, 10]. Despite theavailability and the ongoing development of multiple rTMS protocols, theoptimal protocol for individual patients remains uncertain, leading to severalcritical questions regarding its “real-world” application: For instance, is the2concurrent use of certain medications (e.g., benzodiazepines) during rTMStreatment detrimental? Is comorbid anxiety an indication to use right-sidedstimulation? Do patient preferences vary based on treatment type (e.g., left-vs. right-sided stimulation)?1.1.3 Towards a practical framework for adaptive trials inLHSIdeally, these questions would be addressed in the context of a learninghealth system, using a large-scale adaptive platform trial design while lever-aging existing data from other valuable sources of empirical observationssuch as population databases or large cohort studies with all ‘key’ variables(such as confounders and effect modifiers) measured, to identify the optimalrTMS protocol for different patient subgroups. However, this is not a real-istic goal in the short term due to high costs, long durations, large samplesizes, and the fact that the questions can change during the trial in responseto new information. Due to these concerns, we need to build a more prac-tical framework that allows us to make the best use of the data availablewithin the LHS to support embedding adaptive clinical trials within LHS.1.2 Study rationale and thesis outlineGiven the need for advanced statistical methodologies to design adaptiveclinical trials and efficiently leverage available data within Learning HealthSystems (LHS), the objective of this thesis is to develop statistical method-ologies that support embedding Bayesian adaptive trials within LHS, whichinvolves the following 3 key components:1. Design and evaluation of Bayesian adaptive trials (Chapters2 and 3): These chapters focus on the development and assessmentof Bayesian adaptive trial designs that align with LHS objectives, en-abling flexibility, efficient decision-making, and enhanced patient out-comes through adaptive responses to accumulating evidence. Specifi-cally, Chapter 2 focuses on constructing an R package that can flexibly3design Bayesian adaptive enrichment trials with predefined subgroups.Chapter 3 is a “tutorial” using simulation to illustrate key considera-tions in the design of a Bayesian group-sequential non-inferiority trial.2. A method to construct a prior distribution for a treatmenteffect from two observational datasets when one dataset ismissing known confounders (Chapter 4): This chapter aims toextend existing statistical models to incorporate richer datasets overtime for observational datasets (e.g., commercial claims data, elec-tronic health records, and national registries, etc.) that come fromthe same population, through correcting for treatment effect estima-tion bias in the dataset missing known confounders.3. Synthesize and evaluate the performance of emerging method-ologies for combining observational data and clinical trialsdata to estimate conditional average treatment effects (CATE,Chapters 5 and 6): Chapter 5 carries out a scoping review thatfocuses on combining observational data and trial data to estimateCATE, and Chapter 6 focuses on comparing the performances of threetypical methodologies addressing this problem.Together, these components address key statistical challenges encoun-tered when conducting Bayesian adaptive trials within LHS, which seek toleverage both trial and observational data to address practical challengesand enhance the application of evidence-based practices in healthcare.1.2.1 Design and evaluate Bayesian adaptive trials withinLHSRandomized Controlled Trials (RCTs) are widely regarded as the “gold stan-dard” in clinical research, offering the most robust evidence to evaluate theefficacy and effectiveness of treatments [11, 12]. An adaptive design in RCTsis characterized as a design that allows adaptations to trial and/or statisticalprocedures of the trial after its initiation without undermining the validityand integrity of the trial [13, 14]. This flexibility of designs includes but is4not limited to adaptive randomization design, group sequential design, andenrichment design. [14, 15]. The concept of conducting adaptive clinicaltrials aligns well with the demands of a Learning Health System (LHS). Itpermits adjustments to the trial protocol based on interim data analyses,thus enhancing the trial’s efficiency and relevance to ongoing patient care.This dynamic approach in RCTs ensures that the trials remain responsiveto emerging data, as expected of evidence-based healthcare practices withina LHS. [4].Most randomized trials are designed to determine average treatmenteffects for a target population. However, when important differences inbenefits or harms exist across subpopulations, treatment recommendationsfrom the standard trial designs may be suboptimal, leading to poor patientoutcomes and wasting healthcare resources [16]. Thus, the adaptive enrich-ment design, which can adaptively change enrollment criteria during a trial,is desired [17]. The adaptive enrichment design can provide improved infor-mation about which subpopulations benefit from new treatments and canrestrict the enrollment of only such subpopulations to the trial if early datademonstrates the complementary population is not benefiting, which lies atthe heart of LHS to deliver personalized treatment [18, 19].Applying a Bayesian approach to adaptive clinical trial designs has be-come appealing recently. Compared to the Frequentist method, the Bayesianmethods have some advantages. Most importantly and conceptually, theFrequentist approach does not allow incorporating external information whilethe Bayesian can include all available evidence by applying informativepriors [20, 21]. Generally, the decision rules for different adaptations aremade based on posterior probability conditioning on observed events, whichhas more straightforward interpretations than Frequentist probability state-ments which involve consideration of events that never occurred [22]. Forthese reasons, it can be argued that the Bayesian framework is more suitablethan the Frequentist framework in building a LHS.In Chapter 2, we utilized the R code of the Highly Efficient ClinicalTrials (HECT) simulator [23] as a scaffold, and developed a new softwareBayesAET package to enable designing Bayesian adaptive enrichment trials.5Essentially, HECT included key adaptive features such as: (1) early drop-ping of the futile treatments among multiple treatments, (2) early stoppingof the trial once the best treatment is selected among multiple treatments,and (3) response adaptive randomization (RAR) to allocate more patientsto the more promising treatment arms. Our work extended these capabili-ties to allow for a categorical effect modifier that splits the whole populationinto separate subpopulations with potentially different responses to treat-ment and enabled additional design logics: (1) drop futile treatments incertain subpopulations, (2) enriching: stop recruiting patients from certainsubpopulations if all the treatments failed to show efficacy in those sub-populations, (3) identify the best treatment for each subpopulation, and(4) early termination of the trial if all subpopulations have reached a con-clusion. In addition, we also allow for incorporating a multivariate jointprior on the mean treatment outcomes to increase the efficiency of the de-sign when external prior information is available. We showcase the softwareBayesAET through a case study of a feasibility trial evaluating rTMS fordepression and anxiety. However, the utility of the BayesAET packageextends far beyond the context of rTMS and treatment-resistant depres-sion, showcasing its adaptability in diverse real-world health scenarios. Forinstance, in oncology, BayesAET could be used to design adaptive enrich-ment trials for identifying subgroups of cancer patients who respond bestto specific immunotherapies [24–28]. The package is available on GitHub(https://github.com/zdhjeff/BayesAET).In Chapter 3, we demonstrated the impacts of design choices in Bayesiangroup-sequential non-inferiority trials on power/sample size, illustrated us-ing plasmode simulations based on data collected in the THREE-D trial.Specifically, we compared the performance of Bayesian group-sequential de-sign to the Frequentist fixed design, and demonstrated how varying thenumber of interim analyses, the maximum sample size, whether responseadaptive randomization is used, and the use of informative priors influ-ence trial performance, in terms of the expected sample size and statisticalpower in a non-inferiority trial. The findings from this study have practicalreal-world applications and guidance when designing clinical trials for new6therapies, especially in resource-limited settings or for rare diseases, whereparticipant recruitment can be challenging.By developing software for designing Bayesian adaptive enrichment trialsand providing valuable insights into optimizing Bayesian group-sequentialnon-inferiority trial performance, these two studies directly support the LHScycle, enabling faster and more efficient translation of clinical trial resultsinto healthcare practice and policy.1.2.2 Constructing a prior distribution for a treatmenteffect from two observational datasets when knownconfounders are not measured in one datasetIn a LHS, observational datasets (OBS) are extensively collected and serveas a vital resource for evidence generation. Richer information becomesavailable over time as new datasets are generated. These new datasets of-ten include additional observed covariates, particularly confounders, thatmay not have been measured in older datasets. This evolution necessitatesdeveloping methods to effectively combine such datasets and extend exist-ing statistical models to address new research questions. Different methodshave been developed to address this issue such as selecting a sufficient setof pre-treatment covariates [29] and conducting sensitivity analysis [30–32].Alternatively, newly observed confounders can be treated as missing valuesin the old datasets and multiple imputation (MI) could be used to recoverthese missing values for inference [33, 34]. Bayesian Markov Chain MonteCarlo (MCMC) can be used as a type of multiple imputation (MI) approach,which models missing values as latent variables, with MCMC methods usedto estimate the posterior distribution of the parameters of interest [35]. Inthe Bayesian framework, a natural extension is to transform informationfrom the existing datasets into posterior distributions and incorporate thisinformation as priors in models that integrate the new data. However, thisapproach remains under-explored in causal inference models, and there isa lack of studies systematically comparing the performance of MCMC, andBayesian prior-based methods in estimating treatment effects.7In Chapter 4, we propose novel prior correction methods designed toleverage both new and existing datasets, a common scenario in LearningHealth System (LHS) environments. These methods include theoreticalderivations for the mean and variance of the “bias-corrected prior” as thefirst step, which incorporates information from new data to address potentialbias in priors derived from older datasets, and based on this, we introducean “Honest Precision posterior” (HP), which combines the bias-correctedprior and the new data using a weighted average. The weights are optimallychosen to minimize the variance of the treatment effect estimate while ac-counting for the dependence between the two estimators, as both rely onthe new data. This methodology is specifically developed for Gaussian out-comes.1.2.3 Combining RCT and observational data forindividualized treatment effect (or CATE)While Chapter 4 focuses on combining OBS data that come from the samepopulation to estimate the population average treatment effect (ATE), oneof the ultimate goals of building a LHS is to identify the conditional aver-age treatment effect (CATE), to provide personalized prediction/health caresuggestions for individuals given their characteristics, which is often referredto as “precision medicine” [36, 37].While randomized controlled trials (RCTs) are considered the gold stan-dard for assessing causal effects of interventions, they come with significantlimitations. RCTs are costly and often recruit samples that differ substan-tially from the broader population eligible for the intervention. This islargely due to restrictive inclusion and exclusion criteria, which can limitthe generalizability of their findings. As a result, conclusions drawn fromRCTs frequently lack external validity in medical research [38–40]. In con-trast, OBS data are collected through non-systematic epidemiological de-signs, offering broader coverage of the target population at a lower costthan RCTs. However, the reliability of OBS data is often questioned due tothe absence of rigorous controls, which introduces confounding biases that8hinder valid causal inference [41, 42]. Combining both data sources becomesa natural solution, and the combination is expected to leverage the internalvalidity of RCT and complement it with the richness and external validityof the OBS data. However, the primary challenge lies in effectively address-ing unmeasured confounding in OBS data during the integration process[43, 44]. To deal with the unmeasured confounding, solutions such as sen-sitivity analysis [45, 46], or using an assumption on secondary outcomes orsurrogates [47] have been proposed, but with some restricted assumptions. Awidely favored method, which makes fewer assumptions, involves employinga bias/confounding function to eliminate the impact of hidden confounders[48–51]. Though having these emerging methods, a systematic comparisonof these methods in terms of their performance on estimating CATE underdifferent scenarios is however lacking.In Chapter 5, we conducted a scoping review synthesizing the currentavailable methodologies of combining observational data and trial data withthe aim of estimating CATE, by searching through Google Scholar, whichindexes research articles across all disciplines, including but not limited tomedicine, as well as preprint articles, from inception to October 14th, 2023.The review was structured around the key assumptions required by eachmethod, with particular focus on whether and how the methods account forunmeasured confounding in the observational data. This study provides asummary of methodologies, offering valuable insights into their applicabilityunder different scenarios.In Chapter 6, we compared the performances for 3 state-of-the-art method-ologies: Yang’s elastic combination approach [52], Kallus’s experimentalgrounding approach [48], and Wu and Yang’s integrative R-learner approach[51], through simulations that varied key factors, including the degree ofunmeasured confounding, the sample size ratio between RCT and OBSdatasets, and the structure of the confounding function. This study aimsto provide a comprehensive evaluation of some typical cutting-edge meth-ods for CATE estimation. Together with the scoping review of differentmethodologies of estimating CATE, we aim to provide a “toolbox” to pro-vide practical guidance for researchers integrating RCT and OBS data in9precision medicine, and to support adaptive decision-making in LHS by con-tinuously updating evidence and informing trial designs.The dissertation is concluded in Chapter 7, where we first summarizethe findings and implications of the work and make suggestions for futurework. Finally, we describe knowledge translation activities, completed andproposed, from this dissertation.10Chapter 2Bayesian adaptiveenrichment design inmulti-arm clinical trials: TheBayesAET package for Rusers2.1 IntroductionRandomized controlled trials (RCTs) are the gold standard for determin-ing the safety and efficacy of medical interventions, and are required forregulatory approval (i.e. pivotal trials). The inferences are made on theaverage treatment effect in a selected target population. However, benefitsand harms may vary substantially among subpopulations. For example, inoncology, it is broadly acknowledged that the molecular makeup of nearlyall tumors displays heterogeneity [53, 54], and that this variation can impactthe efficacy of an intervention. The potential consequence is that treatmentrecommendations from pivotal trials using standard trial designs may besuboptimal, when implemented in broader populations of patients in real-world situations, leading to suboptimal patient outcomes and less efficienthealthcare resource utilization [16].Adaptive enrichment design is a strategy used in clinical trials to iden-tify and focus on subpopulations of patients who are more likely to benefitfrom the treatment. These trials begin with a broad patient population, and11then analyze data as outcome data accrues to identify subgroups with betterresponses[14, 17, 18, 55]. In 2019, the Food and Drug Administration (FDA)updated and released the final guidance for the industry entitled “Enrich-ment Strategies for Clinical Trials to Support Demonstration of Effectivenessof Human Drugs and Biological Products”, which defined several types ofenrichment strategies and provided some examples of potential clinical trialdesigns [56].Bayesian clinical trial design has gained popularity recently. Unlike theFrequentist method, the Bayesian approach formally incorporates historicalinformation from previous studies in the form of a prior distribution for thetreatment effects into the analysis of data from the current study to ob-tain a posterior distribution that describes our updated knowledge aboutthe treatment effects [20, 21]. Although priors are often considered subjec-tive, several studies have aimed to improve the quality of prior elicitation[57–59]. Adaptive enrichment design under Bayesian framework has beenstudied. Simon and Simon [60] applied Bayesian methods to optimize thedecisions of how to restrict enrollment criteria when there are two treatmentarms; Xu et al. [61] proposed a Bayesian subgroup-identification enrichmentdesign, which can simultaneously search for predictive biomarkers, identifythe subgroups with differential treatment effects, and modify study entrycriteria at interim analyses when justified. The FDA supports adaptive de-signs and the use of prior information in pivotal trials but requires carefulplanning to ensure trial integrity and validity. Key considerations includepre-specifying adaptations, controlling Type I error rates, and implement-ing robust data monitoring. Detailed guidance is provided in the FDA’s“Adaptive Design Clinical Trials Guidance” (2019) and “Bayesian StatisticsGuidance for Medical Devices” (2010) [62, 63].Recently, several systematic reviews reported on available software foradaptive trial design [64], and multi-arm multi-stage platform trial design[65], which included both Bayesian and Frequentist approaches. While com-mercial (closed-source) software such as ‘FACTS’ [66] exist, various R pack-ages are also available. ‘seDesign’ [67] models continuous, binary and sur-vival outcomes for two subpopulations, while ‘GSED’ [68] supports adap-12tive enrichment design using group sequential methods. The package ‘asd’[27, 69] facilitates the design of seamless adaptive enrichment trials with co-primary analyses in both a prespecified subgroup and the full population.Additionally, ‘ASSISTant’ [70] can be used to design a 3-stage adaptiveenrichment trial using group sequential methods. However, most of thesepackages employ Frequentist methods, probably due to historical precedent,regulatory acceptance, and computational simplicity; Software employingthe Bayesian approach is pivotal but lacking.HECT (Highly Efficient Clinical Trial) is an online R Shiny app thatallows users to perform simulations of an individually-randomized, parallelmultiple arm, Bayesian adaptive trial to evaluate the performance of can-didate trial designs [23]. It supports adaptations such as early stoppingfor benefit or futility and allows for flexible treatment arm modificationsbased on posterior probabilities. However, HECT assumes a homogeneoustreatment effect across the target population. In this work, we describe anew R package ‘BayesAET ’ that allows for heterogeneous treatment effects.We utilize some of the adaptation logic and code in HECT and extend thesoftware to handle adaptive enrichment designs with multiple interventions.To showcase the capabilities of BayesAET, we evaluate the performanceof an adaptive enrichment trial under various adaptive randomization strate-gies and informative priors for the treatment effects. The R package can beinstalled online through GitHub: https://github.com/zdjheff/BayesAET.The paper is structured as follows: in section 2, we will introduce thedetails of the design logic and adaptations embedded in the package, alongwith detailed descriptions of the inputs and outputs of the two main simula-tion functions. In section 3, we will use a real trial example currently in theplanning phase to demonstrate the capacity of the package. We end with asummary and discussion in section 4.132.2 BayesAET package description2.2.1 Overview of the design logics and adaptationsBayesAET combines the features of Bayesian multi-arm multi-stage (MAMS)and enrichment designs. The objective of BayesAET is to assist researchersto design trials with multiple subpopulations and treatments with the goalof identifying the optimal treatment for each subpopulation. This designuses Bayesian adaptive design logic. Patients from different subpopulationsare recruited to the trial and randomized to different treatment arms. Ateach interim analysis, posterior distributions for the mean outcome for eachtreatment within each sub-group are obtained via simulation. Based onthese posterior distributions, any of the following design changes/decisionscan be applied:1. Drop a treatment in a subpopulation if the probability of that treat-ment being the best treatment for that subpopulation is low.2. Stop recruiting patients in a subpopulation if in that subpopulation,the probability of efficacy, compared to an absolute threshold, is lowfor all treatments (enrichment).3. Stop recruiting patients in a subpopulation and declare a winner forthat subpopulation if the best treatment has been identified.4. Change the randomization probabilities to assign more patients to themore promising treatment arms within each subpopulation (responseadaptive randomization – RAR).5. Terminate the trial early, if a conclusion has been reached in everysubpopulation.The flowchart for the full decision process is shown in Figure 2.1.14Figure 2.1: Overview of the adaptive enrichment logic and procedures em-bedded in the ‘BayesAET ’ R package.The two primary functions for evaluating a design are BAET sim, whichsimulates a single trial, and Multi BAET, a wrapper function for BAET simthat enables multiple trial simulations. In the following sections, we presentthe decision logic used for adaptive design and discuss the key input pa-rameters used in these decisions. A complete list of input parameters andoutputs for the BAET sim function are provided in Table 2.1 and Table 2.2,15respectively. The example run for the BAET sim function can be found in Ap-pendix Figure A.1. The details of Multi BAET function including exampleruns can be found in Appendix A.1.Table 2.1: Input arguments and descriptions for function BAET.simBAET.sim Input ar-gumentsDescriptionsnt The number of treatment arms.ns The number of subpopulations.maxN The maximum sample size.ss.interim.es A list containing g vectors, with the g-th vectorindicating the accumulated sample sizes at eachinterim analysis for the g-th subpopulation.response.type The outcome type, one of “gaussian”, “binary”or “count”.mean.response A nt × ns matrix of the assumed true meanresponses for each treatment (row number) ineach subpopulation (column number). For bi-nary and count outcomes, the inputs are theprobabilities and rates, respectively.sig.e The standard deviation of the error term whengenerating a Gaussian outcome. The defaultvalue is 1. This parameter is only relevant whenthe response.type is set as “gaussian”.shape The shape parameter for the gamma distri-bution for the standard deviation of the er-ror term, i.e., sig.e. The default value is0.001. This parameter is only relevant when theresponse.type is set as “gaussian”.Continued on next page16Table 2.1 – Continued from previous pageBAET.sim Input ar-gumentsDescriptionsrate The rate parameter for the gamma distributionfor the standard deviation of the error term, i.e.,sig.e. The default value is 0.001. This param-eter is only relevant when the response.type isset as “gaussian”.prob.subpopulation A vector of length ns indicating the probabilitiesof a participant belonging to each of the subpop-ulations. The default is an equal probability foreach subpopulation.prob.ut.arm A vector of length nt indicating the (initial)probability of a participant being assigned toeach treatment arm. The default is an equalprobability to each treatment arm.upper A vector of length ns indicating the posteriorprobability threshold above which a treatmentwill be declared best (superiority) for each sub-population. The default value is 0.90.lower A vector of length ns indicating the posteriorprobability threshold below which a treatmentwill be dropped for each subpopulation. Thedefault value is 0.10.rar A logical indicator of whether using responseadaptive randomization (RAR). The default set-ting is false, i.e., balanced randomization.MOR A vector of length ns indicating the minimumoutcome requirement threshold for each subpop-ulation. The default value is −∞.Continued on next page17Table 2.1 – Continued from previous pageBAET.sim Input ar-gumentsDescriptionsprob.MOR A vector of length ns indicating the posteriorprobability threshold below which a treatmentwill be dropped due to the outcome not achiev-ing the MOR for each subpopulation. The defaultvalue is 0.10.N.MCMC The number of MCMC samples (excluding burn-in samples) drawn from a posterior distribution.The default is 3000.prior.cov A (nt × ns) × (nt × ns) square matrix forthe prior covariance of a multivariate normalprior for the treatment effects. The default isa diagonal matrix with “100” on the diagonal.prior.mean A vector of length nt × ns for prior mean ofa multivariate normal prior for the treatmenteffects. The default is a vector of zeros.18Table 2.2: Outputs and descriptions for function BAET.simBAET.sim Output Descriptionsn.analysis The total number of analyses, i.e., the numberof interim analyses plus the final analysis.interim.sub A vector indicating the sequence of subpopula-tions that reach the specified sample size thresh-old and trigger interim analyses.n.terminate The total sample size consumed when the trialended.ss.sub A vector of length ns indicating the sample sizeconsumed in each subpopulation.est A list of length ns with each component summa-rizing the posterior distribution for the treat-ment effects within a subpopulation at trialtermination. Each component is a nt × 4data frame with rows representing treatmentsand columns displaying the posterior mean, thelower and upper bound of the 95% Credible In-terval, and the SD of the posterior distribution,respectively.powerind A vector of length ns indicating whether thebest treatment arm was correctly selected ineach subpopulation at trial termination. Thevalue is ‘1’ if correctly selected and ‘0’ other-wise.y A vector for the simulated outcomes.trt.sub A two-column matrix with the first column dis-playing the treatment each participant receivedand the second column displaying the subpopu-lation in which the participant belongs.Continued on next page19Table 2.2 – continued from previous pageBAET.sim Output Descriptionsprob.sup.minoutcome A list of length ns with each component sum-marizing the probability of superiority over thefixed MOR within a subpopulation. Each com-ponent is a matrix with rows representing treat-ments and columns displaying the probability ofsuperiority over the fixed MOR at each interimlook.prob.superiority A list of length ns with each component summa-rizing the probability of superiority among dif-ferent treatments within a subpopulation. Eachcomponent is a matrix with rows indicatingtreatments and columns displaying the proba-bility of superiority for each treatment at eachinterim look.2.2.2 Model frameworkDesign features and inputsSuppose there are M treatments and G subpopulations. In the package,this is specified by the user through the two arguments: nt for the numberof treatments and ns for the number of subpopulations. For each subpop-ulation g, suppose the recruiting probability is kg, g = 1, 2, . . . , G. Theuser can adjust the expected recruitment probability through the argumentprob.subpopulation, which is a vector of length G, with each entry havinga default value of 1/G. The data models are parameterized using “full-rank coding”, in which the model parameters reflect subpopulation meanresponses rather than mean responses relative to a reference treatment arm.To use our package in conventional settings where the comparative treatmenteffect is estimated relative to a reference (placebo) arm, we set the meanresponse of the reference/placebo arm to 0. The mean responses of other20treatments are then adjusted by subtracting the original mean response ofthe reference arm. In this setting, all other treatment input values representcomparative treatment effects.Suppose θmg indicates the mean response of treatmentm (m = 1, 2, . . . ,M)in the subpopulation g (g = 1, 2, . . . , G). Then for a Gaussian outcome, theindividual-level response Yi for patient i (i = 1, 2, . . . , N) who belongs tosubpopulation g and is receiving treatment m is modeled as:Yi = θmg + ϵi,where ϵi ∼ N (0, σ2).In the software, the assumed input values for θmg are specified by theuser through the argument mean.response, an M × G matrix where the[m, g]-th entry of this matrix indicates the mean response of treatment min subpopulation g. Since we model the mean outcome directly, covariateadjustment is not supported in this package. Details on the data generationprocess used in the simulations are given in Appendix A.2. In this package,it is assumed throughout that a larger value for θmg and Yi corresponds toa better outcome.Modeling and estimation framework Denote the θ as the vector ofthe true mean responses with length M × G, with its (M × (g − 1) +m)-th component indicating the mean response of treatment m in the g-thsubpopulation θmg, i.e.,θ = [θ11, θ21, . . . , θM1, θ12, . . . , θM2, . . . , θ1G, . . . , θMG].Let Θ denote the prior distribution for θ. When external informationis available a priori, informative joint priors can be incorporated to increasethe efficiency of the design. We then consider a multivariate joint priorstructure to account for the correlations between the parameters of themean responses. This is achieved by incorporating this information into21the variance-covariance matrix of the prior:Θ ∼MVN (µ0,Σ),where µ0 is the prior mean vector with length M × G and Σ is a (M ×G) × (M × G) prior variance-covariance matrix. These two arguments arespecified by the user through the arguments prior.mean and prior.cov,respectively.While analytic expressions for the joint posterior distribution for θ areavailable for Gaussian outcomes, we compute the posteriors empirically fromthe MCMC samples drawn by Gibbs sampling through software RJAGS toenable straightforward extension to non-Gaussian outcomes. The number ofMCMC draws (after burn-in) is specified by the user through the argumentN.MCMC. Since the models employed in this package for comparing arms arestraightforward, the model convergence issues are generally not expected tobe encountered.We have similar setups for binary and count outcomes and the θmg repre-sents the probability and rate, respectively. The type of outcome is specifiedby the user through the argument response.type, with options of “gaus-sian”, “binary”, and “count”. The implementation details for binary andcount outcomes are given in Appendix A.3.Interim analysis and adaptations As an adaptive design, interim anal-yses will be scheduled based on the batch sample size, Rbg, which de-scribes the specific (accumulated) sample sizes at b-th interim evaluationsfor subpopulation g. This is specified by the user through the argumentss.interim.es, which is a list containing the g vectors, with g-th vectorindicating the accumulated subpopulation sample size for each interim anal-ysis for g-th subpopulation. The maximum sample size recruited for the trialis denoted Rmax and is specified by the user through the argument maxN.As the trial progresses, different decisions and adaptations may be madebased on the interim data analysis, and the decision rules are based onthe posterior superiority probabilities. Note that each interim analysis will22utilize all available data (from all subpopulations) but make the decisionsonly for the subpopulation that has reached the user-specified sample sizethreshold.Specifically, the posterior superiority probability of the treatment m inthe subpopulation g, Smg, is defined as the probability of the treatment mbeing the best treatment among M treatments in the subpopulation g afterobserving the cumulated data:Smg = P (θmg − θhg > 0 ∀h ̸= m|data),and Smg can be estimated empirically by:Sˆmg =1NN∑i=11(θmg,i = max(θ1g,i, . . . , θMg,i)),where θmg,i is the i-th MCMC sample from the posterior distribution ofθmg. N is the total number of MCMC iterations, and 1(A) is the indicatorfunction that takes the value 1 when A is true and 0 otherwise.With the Smg updated after each interim look, we design the trial allow-ing the following four major adaptations:1) Dropping a treatment arm within subpopulation g A treat-ment m will be dropped if it is judged unlikely: (1) to be the best treatmentin that subpopulation g, or (2) to achieve a minimum mean outcome. Specif-ically, this will happen if either (1) the probability that this treatment is thebest one falls below a prespecified threshold Lg; that is,Smg < Lg,or (2) the posterior probability that the mean outcome for treatment m islarger than a minimum clinically acceptable value, Cg, falls below a mini-mum specified threshold Tg; that is,P (θmg > Cg|data) < Tg, m = 1, . . . ,M.23Lg is specified by the user through the argument lower, which is a vectorof length G. Cg and Tg are specified by the user through the arguments MORand prob.MOR, respectively. Both of these arguments are vectors of lengthG.2) Dropping the whole subpopulation g If all M treatments aredropped in the subpopulation g, no treatment will be declared as the winnerin this subpopulation and patient enrollment in this subpopulation will bestopped. This is how adaptive enrichment is realized in this trial design.3) Declaring the best treatment within subpopulation g Identi-fying the best treatment within each subpopulation g is the main objectiveof the trial design. A treatment m will be declared as the best treatment insubpopulation g if it simultaneously meets both the following two criteria:1. The posterior probability of a treatment m, Smg, exceeds a specifiedthreshold Ug for declaring superiority; that is,Smg > UgThe Ug is specified by the user through the argument upper, which isa vector of length G.2. The posterior probability that the mean response of treatment m ex-ceeds a clinically minimum outcome requirement (MOR), Cg, achievesa minimum acceptable certainty Tg, for the subpopulation g:P (θmg > Cg | data) ≥ TgOnce the best treatment within subpopulation g is identified, the enroll-ment of participants from subpopulation g is stopped.4) Termination of the whole trial The whole trial may stop whenreaching the maximum sample sizes or stop early if all the G subpopula-tions have reached a conclusion, where the conclusion can be either the besttreatment is selected for that subpopulation or all the M treatments in thatsubpopulation are dropped.24Response adaptive randomization The design can use either balancedrandomization (BR) or response adaptive randomization (RAR). Let ρm|gdenote the conditional probability of being assigned to treatment m fora participant belonging to g-th subpopulation. The initial probability isspecified by user through the argument prob.trtarm, which is a vector oflength M , with each entry having a value of 1/M .For the first batch of sample size R1, all participants from G subpopu-lations are assigned to the M treatments using a balanced randomizationscheme, which is:ρ1|g = · · · = ρM |g =1Mfor all g = 1, 2, . . . , GAfter the first batch of samples, the randomization strategy can continueto use BR or switch to RAR, in which greater probability is assigned to themore promising treatments within subpopulation g. While various proba-bility adjustment criteria have been proposed [71, 72], we adopt the criteriaused in HECT [23], which adjusts the assignment probability to be propor-tional to the square root of the posterior superiority probability Smg giventhe cumulated data, i.e., the treatment assignment probability for patientin subpopulation g in the next batch is specified as:ρm|g =√Smg∑g Smg.However, it is worth noting that under RAR, there is a possibility that anarm continues in the trial but having a low probability of superiority, re-sulting in limited patient allocation to that arm during the trial. Therefore,in the future, a lower threshold for allocation probability might be recom-mended for RAR to address this issue in the package.Whether the design utilizes BR or RAR after the burn-in period is con-trolled by the logical input parameter rar.25Design outputsThe primary focus when evaluating a trial design are its statistical perfor-mance properties, including mean response estimation, power, and the finalaveraged sample size. Thus, in this section, we will focus on the outputsof function Multi.BAET. For the outputs of a single run using the functionBAET.sim, please refer to Table 2.2, Figure A.2, and Figure A.3 for detailedexplanations and examples.This information can be obtained from the outputs of the following:1. Mean response estimation The final mean response of treatment min subpopulation g at the end of the trial, Ef (θmg), is estimated empiricallyby running the simulation T times and taking the mean of the T posteriormeans from each simulation run:Ef (θmg) =1TT∑t=1E(θmg|data)t,where E(θmg|data)t is the posterior mean of θmg at t-th simulation run.This output is stored in the est.mean variable, which is a G×M matrix,with [g,m]-th entry showing the m-treatment estimation in g-th subpopu-lation. Similarly, the mean standard deviation of the posterior distributionfor each treatment in each subpopulation is stored in the est.sd variable,which is also a G×M matrix, with [g,m]-th entry showing the m-treatmentposterior distribution standard deviation in g-th subpopulation.2. Power Both power and Type I error are Frequentist concepts. Ingeneral, a trial is often designed with a certain sample size calculated under apredefined Type I error rate (e.g., 0.05) and a power value (e.g., 0.8). Undera Bayesian framework, we can still have similar concepts determined bythe posterior probability of the treatment effects. Specifically, since we aremaking comparisons and decisions independently within each subpopulationg, and there are threshold values for each subpopulation g, there will thusbe subpopulation-specific power for each subpopulation g, Powerg. In our26setting here, we define subpopulation power, Powerg, as the probabilitythat a treatment m is successfully identified as the best treatment amongM treatments in subpopulation g (given m is in truth the best treatment),i.e.,Powerg = P (Smg > Ug and P (θmg > Cg|data) ≥ Tg).From the above equation, it is evident that power (and Type I error)can be controlled by adjusting threshold values such as: Lg, Cg, Tg, andUg. This output is stored in the power.sub variable, which is a vector of Gcomponents indicating the proportion of times the best treatment arms arecorrectly selected in each subpopulation.If users are interested in the Type I error of the design, they can obtainType I error with this function by setting the true treatment effect to zero.Detailed information about this function will be provided in Section 3.3. Sample size distribution The sample size at trial termination is animportant criterion for the efficiency of the adaptive designs. Intuitively, ifthe effect size/statistical power is high, the trial will often end early witha relatively small number of samples; if the effect size/power is low, thetrial will require a larger number of samples before identifying the optimaltreatment or will reach the maximum sample size without making the finalconclusions. We can visualize the sample size distribution and estimate themean sample size, Eˆ(SS), of a specific design through simulating the trial Ttimes by:Eˆ(SS) =1TT∑t=1Rt,where Rt is the total sample size in t-th simulation.The overall averaged sample size consumed is stored in the variabless.t.mean, and the averaged sample size consumed in each subpopulationis stored in the variable ss.sub.mean.272.3 Practical exampleWe illustrate the application of this package to the PORT (Patient-OrientedRandomized pragmatic feasibility Trial, NCT05028738) study of repetitivetranscranial magnetic stimulation (rTMS) in depression and anxiety. rTMSis a non-pharmacological intervention that relies on the principle of electro-magnetic induction to stimulate focal areas in the brain and induce lastingchanges in brain activity [73].rTMS is an approved intervention for treatment-resistant depression, butthere exist multiple rTMS protocols, and identifying which protocol is bestand for whom is an active research topic. For instance, previous studies sug-gest that concurrent use of benzodiazepines (Benzo) may negatively impactthe efficacy of rTMS therapy [9].PORT is currently a feasibility study intended to support the designof a full-scale trial to identify which of the rTMS treatment protocols isbest in each of two subpopulations (Benzo or Non-Benzo). Individuals wererecruited from each subpopulation and randomized to receive one of thethree interventions: (1) left DLPFC rTMS, (2) right DLPFC rTMS, and (3)the combination of both left and right DLPFC rTMS.The primary outcome is the reduction of the depression score measuredby the 17-item Hamilton Depression Rating Scale (HRSD-17) [74], whichconventionally is treated as a continuous outcome. Our purpose is to illus-trate the use of this package, not to identify the best design, and thereforewe present results from only a limited number of settings for demonstrationpurposes.2.3.1 Simulation settingsIn PORT, there are three rTMS treatment protocols (M = 3) and twosubpopulations (G = 2), which are No Benzo concurrent usage population(Non-Benzo, g = 1) and Benzo concurrent usage population (Benzo, g = 2).The Benzo usage was found to interact with the rTMS treatment according28to previous studies [9]. Thus, the treatment effect vector is:θ = [θ11, θ21, θ12, θ22, θ13, θ23],and the outcome Y is modeled as:Y =Xθ + ϵ,where ϵ ∼ N (0, σ2).We illustrate how the software is used to obtain results from differentpriors and randomization strategies. The details of the data generation-related parameters and other design-related parameters can be found inTable 2.3 and Table 2.4, respectively. These parameters are also the keyinput arguments for the main simulation function BAET.sim. Notice thatin Table 2.4, even though we have information about the minimal outcomerequirement of the rTMS treatments, we use the true effect sizes of thethree treatments as derived from our best approximation for the sample sizecalculation and simulations [75].Table 2.3: The data generation related parameters inputsAssumed parameters fordata generationInput valuesAssumed mean responses(θ11, θ21, θ12, θ22, θ13, θ23)mean.response = (8, 12, 14, 5, 8,11)Standard deviation σ sig.e = 10Recruitment probabilities fora Non-Benzo or Benzo patient(k1, k2)prob.subpopulation = (0.6, 0.4)29Table 2.4: The design related parameter inputsTrial Design Parameters Input ValuesMinimum acceptable superiorityrate (L1, L2)lower = (0.0, 0.0)Clinically minimum acceptablevalue of the mean response (C1, C2)MOR = (6, 6)Minimum acceptable rate (T1, T2) ofbeing larger than (C1, C2)prob.MOR = (0.5, 0.5)Minimum acceptable superiorityrate of being the best treatment(U1, U2)upper = (0.9, 0.9)Sample size at each interim look fortwo subpopulations (Rb1, Rb2)ss.interim.es = (60, 120,180, 220, 390) and (50, 110,140, 200, 390)The maximum sample size (Rmax) maxN = 390In this study, the third rTMS treatment protocol is the combination ofthe treatments from the first and second protocols, so it is plausible that theefficacy of the third treatment is correlated with the efficacy of each of thefirst two treatments. Therefore, the Bayesian model will incorporate thiscorrelation information.We will use the following prior structure:B ∼MVN (µ0,Σ),where:µ0 =µ1µ2µ3µ4µ5µ6and Σ =25 0 σ13 σ14 0 00 25 σ23 0 σ25 0σ31 σ32 25 0 0 σ36σ41 0 0 25 0 σ460 σ52 0 0 25 σ560 0 σ63 σ64 σ65 25Specifically, the off-diagonal numbers including σ13, σ23, σ46, and σ56(and their corresponding symmetrical entries in the matrix) represent the30covariance between mean responses within each subpopulation, while σ14,σ25, σ36 (and their corresponding symmetrical entries in the matrix) indicatethe covariance for the same treatments across two subpopulations.For simplicity in demonstration, we assume treatment effects are notcorrelated across subpopulations (i.e., σ14 = σ25 = σ36 = 0), though this isallowed in general.To illustrate how the various design inputs impact trial performance, weevaluated the 16 scenarios shown in Table 2.5:Table 2.5: Simulation scenarios with different combinations of prior mean,prior covariance, and randomization strategyScenario#Prior mean(µ1, µ2, µ3, µ4, µ5, µ6)Prior co-variance(σ13, σ23, σ46, σ56)RandomizationstrategyS1 (8, 12, 14), (5, 8, 11) (0, 0, 0, 0) BRS2 (8, 12, 14), (5, 8, 11) (2.5, 2.5, 2.5,2.5)BRS3 (8, 12, 14), (5, 8, 11) (10, 10, 10, 10) BRS4 (8, 12, 14), (5, 8, 11) (15, 15, 15, 15) BRS5 (11, 11, 11), (8, 8,11)(0, 0, 0, 0) BRS6 (11, 11, 11), (8, 8,11)(2.5, 2.5, 2.5,2.5)BRS7 (11, 11, 11), (8, 8,11)(10, 10, 10, 10) BRS8 (11, 11, 11), (8, 8,11)(15, 15, 15, 15) BRS9 (8, 12, 14), (5, 8, 11) (0, 0, 0, 0) RARS10 (8, 12, 14), (5, 8, 11) (2.5, 2.5, 2.5,2.5)RARS11 (8, 12, 14), (5, 8, 11) (10, 10, 10, 10) RARS12 (8, 12, 14), (5, 8, 11) (15, 15, 15, 15) RAR31Scenario#Prior mean(µ1, µ2, µ3)Prior co-variance(σ13, σ23, σ46, σ56)RandomizationstrategyS13 (11, 11, 11), (8, 8,11)(0, 0, 0, 0) RARS14 (11, 11, 11), (8, 8,11)(2.5, 2.5, 2.5,2.5)RARS15 (11, 11, 11), (8, 8,11)(10, 10, 10, 10) RARS16 (11, 11, 11), (8, 8,11)(15, 15, 15, 15) RARBasically, scenarios 1 through 4 (S1-S4) focus on correctly-specified priormean (Cs-m) and balanced randomization strategy (BR), with (Pearson)correlation changing from 0 to 0.6; scenario 5 to scenario 8 are focusedon mis-specified prior mean (Ms-m) and equal randomization strategy, withcorrelation changing from 0 to 0.6; scenarios 9 through 12 focus on correctly-specified prior mean (Cs-m) and responsive adaptive randomization strat-egy (RAR), with correlation changing from 0 to 0.6; scenarios 13 through16 focus on mis-specified prior mean (Ms-m) and responsive adaptive ran-domization strategy, with correlation changing from 0 to 0.6. We perform5,000 MCMC iterations and simulate each trial scenario 50,000 times.To manage these multiple simulations for assessing trial properties, wedeveloped a wrapper function called Multi.BAET. This function shares thesame parameters as BAET.sim and introduces two additional arguments:n.sim, which specifies the number of simulation runs, and n.cores, whichindicates the number of cores used for parallel computing. By default,Multi.BAET utilizes half of the available cores on the device. Figure A.4 inthe Appendix illustrates an example of the function Multi.BAET.322.3.2 Simulation outputsAs described, we utilize the Multi.BAET function to simulate trial proper-ties. This function outputs: (1) the mean of the posterior means and meanof the posterior standard deviations for each treatment across each subpop-ulation. These outputs are stored in the variables est.mean and est.sd,respectively. Both variables are structured as G×M matrices, where the gthrow corresponds to the gth subpopulation and the mth column correspondsto the mth treatment. (2) The sample size for each subpopulation across allruns, as well as the average sample size for each subpopulation, are recordedin the variables ss.sub.dist and ss.sub.mean, respectively. (3) Similarly,the total sample size for each run, along with the average total sample size,are recorded in ss.t.dist and ss.t.mean. (4) The power calculated foreach subpopulation is recorded in the variable power.sub. (5) Lastly, thecomputation time utilized is recorded in the variable computation.time.The details of Multi.BAET function including example runs can be found inAppendix A.1.Treatment effect estimation and biasFor the 16 scenarios described in section 3.1, Figure 2.2 (A) and 2.2 (B)shows the treatment estimation (mean response) of the best treatments (i.e.,treatment 3 in this example) under different prior mean settings and ran-domization strategies, when the correlation is changing from 0 to 0.6, forthe Non-Benzo subpopulation and Benzo subpopulation, respectively, withthe black dash line indicating the true effect size. Overall, for both Non-Benzo and Benzo subpopulations, for the scenarios with mis-specified priormean, the treatment estimation tends to be less biased when the correlationis 0, but tends to be very biased when the correlation is high (0.6); for thescenarios with correctly-specified prior mean, the treatment estimation isslightly over-estimated.33The subpopulation powerThe power is an important performance evaluation criterion of a trial de-sign. Figures 2.2 (C) and 2.2 (D) show subpopulation power under differ-ent prior mean settings and randomization strategies, when the correlation ischanging from 0 to 0.6, for the Non-Benzo subpopulation and Benzo subpop-ulation. For both Non-Benzo and Benzo subpopulations, as the correlationincreases, for the scenarios with mis-specified prior mean, the power tendsto drop dramatically; while for the scenarios with correctly specified priormean, the power tends to increase slightly. The loss in power occurs be-cause the mis-specification of the prior mean leads to an underestimation ofefficacy in the best treatment arm, making it difficult for the trial to selectthe best treatment arm.34353637Figure 2.2: The simulation results under 4 different settings with the cor-relation between different treatments within each subpopulation changingfrom 0 to 0.6. (A) and (B) show the best treatment (treatment 3) effect sizeestimation in Non-benzo subpopulation and Benzo subpopulation (the tra-jectory of “Ms-m, BR” overlaps with that of “Ms-m, RAR”). (C) and (D)show the subpopulation power in Non-benzo subpopulation and Benzo sub-population. (E), (F) and (G) show the expected sample size in Non-Benzosubpopulation, Benzo subpopulation and overall population, respectively.Expected sample size and sample size distributionsAnother important evaluation criterion of an adaptive trial design is theexpected sample size. Figure 2.2 (E), 2.2 (F) and 2.2 (G) show the expectedsample size in Non-Benzo subpopulation, Benzo subpopulation and overallpopulation, respectively, under 4 different settings with correlation changingfrom 0 to 0.6. As the correlation increases, for the scenarios with correctlyspecified prior mean, the expected sample size does not change apparently,though with a slightly increased power; for the scenarios with mis-specifiedprior mean, the expected sample size tends to increase greatly but with a38decreased power. This is due to the high correlation that pulls Treatment 3towards the ’null hypothesis’(i.e., all three treatments are the same), makingit more challenging for treatment 3 to demonstrate superiority.Sample size distribution can provide an overview of the sample size con-sumed in each trial simulation. Though the expected sample size is animportant evaluation of the trial performance, unlike fixed design, it maynot be reliable to guide the adaptive design, as this expected sample sizeis estimated given a maximum sample size. Figure 2.3 demonstrated thesample size distribution for simulation scenario S1. As demonstrated in thefigure, a noticeable portion (around 50%) of the runs reach the maximumsample size, suggesting that some trial runs may still not reach conclusivefindings within this limit and a larger maximum sample size may be neededin practice.Figure 2.3: Overall sample size distribution for simulation scenario S1.392.4 DiscussionIntroduction to BayesAETWe developed an R package BayesAET to allow adaptive enrichment designsthat can identify which pre-defined subpopulations benefit from treatments.To the best of our knowledge, this is one of the first R packages that enablesusers to assess multiple treatments for each subpopulation in enrichmentdesigns.Key Features and CapabilitiesThis package supports early termination of trials for efficacy or futilitybased on posterior superiority probabilities, enabling faster identification ofoptimal treatments for each subpopulation. Users can choose between equalor responsive adaptive randomization based on their preferences. Addition-ally, when external prior information about treatment interactions is avail-able, the package allows for a multivariate joint prior structure, enhancingthe design’s efficiency.Performance and Example ApplicationsIn our demonstration, we used a planned trial as an example to showcasethe package’s capabilities. We examined typical scenarios involving differentcorrelations between treatments in the prior covariance structure and variedrandomization strategies.Type I Error Rate and Multiple TestingIn our example, we have deliberately ignored assessment and discussionof Type I error rates for several reasons. First, the multiplicity of tests aris-ing from comparing multiple treatments in multiple subpopulations makes itdifficult to specify a single null hypothesis that would meaningfully representthe real-world questions of interest. (We note that if in fact one were inter-ested in this “global” null hypothesis, the corresponding Type I error could40be evaluated using our software by obtaining the power when the treatmenteffects are identical for all treatments and subpopulations. However, correc-tions for multiple-testing in multi-arm trials remains a controversial topic[76].) Second, although one may seek to evaluate Type I error rates in orderto adjust the decision criteria to “maintain control” of these rates, doingso negates in general any gain in power obtained through using informativepriors [77]. As a result, focusing on Type I error rates is not productive ifone wishes to make use of prior information. Finally, we believe that thedecision thresholds used in a Bayesian approach are clinically more intuitiveand easier to interpret than Type I error rates. We acknowledge, however,that potential Type I error rate inflation resulting from early stopping forefficacy in a Bayesian adaptive design can be a concern [78].While only a few settings are examined in our simulations for demonstra-tion purposes, one should notice that the trial performance is expected tochange greatly when varying other design parameters, for instance, changingthe minimum acceptable superiority rate (L1, L2) to a higher level and/orchange the minimum acceptable superiority rate of being the best treatment(U1, U2) to a lower level will speed up the trial with a higher power, a smallersample size, but with a risk of higher Type I error rate. Though we onlyconsidered the Gaussian outcome here, we would expect similar patterns forthe binary and count outcomes.Future Directions and ExtensionsOur proposed adaptive enrichment design aims to provide ‘personalizedmedicine’ where the treatments are tailored to patients based on their char-acteristics. When subpopulations become complex, more Bayesian modelscan be considered. For example, emerging work includes applying Bayesianhierarchical models [79], using random partitions and semiparametric Bayesianmodels [80], and using non-parametric cluster hierarchical models [81] toidentify the treatment heterogeneity in subpopulations. There is also emerg-ing research focusing on finding the personal treatment index through propos-ing new criteria measuring the benefits and losses for individuals both in-side and outside of the clinical trial [82], using variable selection to identify41a covariate sub-vector that identifies treatment-sensitive patients and de-termines a personalized benefit index [83], and including a latent subgroupmembership variable that can be used to exploit to adaptively collapse ho-mogeneous subgroups or identify heterogeneous subgroups [84].ConclusionBayesAET provides a comprehensive tool for designing and analyzing Bayesianadaptive enrichment trials. By enabling the identification of treatment-sensitive subpopulations and incorporating prior information, BayesAETenhances the flexibility and efficiency of clinical trials. The package is avail-able for installation on GitHub https://github.com/zdhjeff/BayesAET,providing an accessible solution for researchers seeking to implement adap-tive enrichment designs in their studies.42Chapter 3Impact of design choices onpower and sample size in aBayesian group-sequentialnon-inferiority trial: Anillustration using plasmodesimulation3.1 IntroductionClinical trials, often regarded as the gold standard in clinical research, havelong relied on the Frequentist fixed design in which the total sample sizeis pre-specified and data are analyzed only at the end of the trial usingFrequentist null hypothesis testing, p-values, and confidence intervals. Thefixed design remains widely utilized due to its simplicity and robustness inplanning and analysis [85–88]. Compared to the fixed design, adaptive de-signs are intended for continual learning as the data accumulate throughoutand allow for modifications to key design components, including early stop-ping, changes to the allocation ratio or to eligibility criteria, and droppingor adding treatment arms [14, 15, 89].To further enhance trial flexibility, Bayesian methods increasingly havebeen utilized in adaptive designs [90–94]. Unlike the Frequentist approach,Bayesian methods utilize probability distributions to represent uncertainties43in treatment effect estimates. Bayesian methods offer a formal mechanismto update estimates as new data become available, making them ideallysuited for conducting interim analyses with accumulating information [95].Moreover, Bayesian methods can enhance the relevance and precision of in-ferences by incorporating information from outside the trial (such as datafrom previous studies and clinician judgments) through integrating this in-formation into a prior distribution [95, 96].In this paper, we present a simulation study that shows how varyingthe number of interim analyses, the maximum sample size, whether re-sponse adaptive randomization is used, and the use of informative priorsin a Bayesian group-sequential design influence trial performance for a non-inferiority trial, and compare this performance to that of the Frequentistfixed design. In particular, we will focus on the expected sample size andstatistical power. To ensure comparability between the Bayesian group-sequential and Frequentist fixed designs, we will specify certain design re-quirements (e.g., Type I error rate), when appropriate.This demonstration will use simulations based on data from the THREE-D study [10], which was a blinded, randomized, multicenter trial to estab-lish non-inferiority of intermittent theta burst stimulation (TBS) repetitivetranscranial magnetic stimulation (rTMS) compared to conventional 10 Hzhigh-frequency left-sided (HFL) rTMS for treating individuals with majordepressive disorder (MDD). The focus on a non-inferiority design in thischapter is deliberate to highlight its specific applications and nuances, andit is less studied compared to superiority trials under Bayesian adaptivesettings.The paper is organized as follows: in section 2, we describe the Bayesianadaptive design logic and how we control the overall Type I error rate inBayesian adaptive design. In section 3, we illustrate the performance undervarying numbers of interim looks in each of three sets of simulation sce-narios: the first set fixes the maximum sample size (which leads to varyingpower), the second set fixes the power (which leads to changes in the requiredmaximum sample size), the third set evaluates the impact of randomizationstrategies, and the fourth set evaluates the impact of prior information. We44conclude our findings with discussions in section 4.3.2 Method3.2.1 Plasmode simulation based on the THREE-D studyThe original THREE-D study enrolled 414 patients in order to achieve 90%power to demonstrate non-inferiority of TBS to HFL, assuming that bothtreatments worked equally well, and with the non-inferiority margin set at2.5 points on the HRSD-17 reduction (i.e., the TBS benefit can be up to 2.25points less than that for HFL). However, only 385 participants contributeddata to the entire study. The study found that HRSD-17 scores improvedfrom 23.5 to 13.4 in the HFL group and from 23.6 to 13.4 in the TBS group,with an adjusted difference of 0.103 and a lower bound of the 95% confidenceinterval of -1.16 (p = 0.0011). These results indicate the non-inferiority ofTBS compared to HFL [10]. Our aim is to illustrate how this trial might haveunfolded if we had used a Bayesian adaptive design allowing for stoppingearly at one of the pre-specified interim looks at the data.We will use the plasmode simulation [97, 98] approach wherein simu-lated datasets are sampled from a real, observed dataset which is treated asthe source population. In plasmode simulation, the original data structure,relationships, and distributions are preserved as much as possible, and canbetter mimic real-world scenarios. Thus, in this work, we assume the dataobserved in the THREE-D study represents the true population.The outcome measure is the reduction in the Hamilton Rating Scale forDepression (HRSD-17) score (continuous outcome) from baseline to finalfollow-up at week 6. In the THREE-D dataset, based on the completed 385participants, the mean reduction in HRSD-17 scores were nearly equal at10.23 for TBS and 10.1 for HFL. Based on these values, the power basedon a Frequentist fixed design is 91% (rather than the 90% obtained fromassuming the benefits from the two treatments are exactly equal). For thesimulations, participants are sampled from this dataset with equal proba-bility, with replacement.453.2.2 Key design features for Bayesian adaptive trialBayesian methods in clinical trials use probability to quantify uncertaintyabout treatment effects, allowing researchers to update their beliefs about atreatment’s efficacy as data are collected. Using Bayes theorem, pre-existinguncertainty (the prior distribution) and new data (described by the likeli-hood function) are combined to produce the posterior distribution. Thisposterior distribution represents an updated belief about the treatment ef-fect after considering the observed data. In our Bayesian adaptive trial, weallow for early termination at predetermined interim analyses. At each in-terim look, decisions are made based on the posterior distribution, which isupdated with the data accumulated at each sample size milestone. In addi-tion to the non-inferiority margin mentioned in the previous section, thereare three key design features that could affect the power and the Type I errorrate of our Bayesian trial: the upper threshold of probability for claimingnon-inferiority (U), the lower threshold of probability for claiming inferior-ity (L), and the maximum sample size. The trial will be terminated duringan interim look if the probability for claiming non-inferiority is larger thanU or smaller than L. The trial will also be terminated when the maximumsample size is reached. The details of how to construct decision rules usingposterior probability for early stops due to non-inferiority can be found inAppendix B.1 and Chapter 2.To facilitate comparison between the Frequentist and Bayesian approachesin Section 3.1 below, we will adjust the upper (U) and lower (L) decisionthresholds to control the Type I error rate. Once these thresholds are fixedto maintain the desired Type I error rate, we will then assess the power ofthe Bayesian approach using these specific U and L values. This is oper-ationalized in our simulations by creating a ‘null dataset’ which shifts themean reduction in HSRD-17 scores for TBS from 10.23 to 7.85, so that thedifference in mean reduction in HSRD-17 scores between the two arms isexactly the non-inferiority margin -2.25. Using this ‘null dataset,’ we willfix the lower threshold, L, to be 0.1, and then tune the upper threshold, U,to keep the Type I error rate as 0.05 (within a margin of ±0.001) through ex-46tensive simulations (30000 simulation runs, Monte Carlo SE of 0.001). Usingthis tuned upper threshold U, we assess the other trial operating character-istics, for different numbers of pre-specified interim looks and pre-specifiednumbers of participants through simulation using the original dataset. ThisType I error rate tuning process is summarized in Figure 3.1 from Step 1to Step 4. In section 3.2, we will need to adjust U and maximum samplesize to control both Type I error rate and target power, and this involves 3additional adjustment steps as shown in Figure 3.1: step 0, propose/adjustmaximum sample size, step 5, check if the power is around the target power,and step 6, record the expected sample size. A detailed explanation will beintroduced in section 3.2. Unless otherwise specified, vague priors were usedfor simulations.47Figure 3.1: The tuning process for controlling the over-all Type I error rate and target power in Bayesian group-sequential design. There are three key design features thatcould affect the power and the Type I error rate of ourBayesian adaptive design.48We will also compare the efficiency of the conventional Frequentist fixeddesign and Bayesian adaptive design, in terms of their treatment estimations,the power, and the sample size consumed. For the Bayesian adaptive design,the details of calculating the power, average sample size needed, and TypeI error rate can be found in Appendix B.1 and Chapter 2. The power of theconventional Frequentist fixed design is calculated by the online calculatorhttps://clincalc.com/stats/Power.aspx.3.3 Simulation settings and resultsBy adjusting the upper (U) and lower (L) decision thresholds to controlthe Type I error rate, we align the conventional Frequentist design with theBayesian design. A straightforward example demonstrating this alignmentis seen in the power comparison: after fixing the Type I error rate, the powerfor the original THREE-D conventional fixed design is 0.910, which is com-parable to the Bayesian design’s power of 0.908 when there are no interimlooks. Controlling the Type I error rate also facilitates comparison amongdifferent Bayesian adaptive designs, as demonstrated in the subsequent sec-tions.Before moving to the general results, we present an example simulationwith 3 interim looks to illustrate the process. For this case, simulations usingthe ‘null dataset’, where the null hypothesis assumes the effect size equals thenon-inferiority margin of -2.25, revealed that an upper threshold U of 0.9828is required to maintain the overall Type I error rate at 0.05. Using this upperthreshold, we then conducted simulations under the alternative hypothesis,where the true effect size is assumed to be 0.12, based on the original dataset,to evaluate the design’s performance. Table 3.1 shows the percentage ofsimulated trials that stopped after each interim analysis and whether thestop was for declaring non-inferiority or (incorrectly) declaring inferiority.Across all simulated trials, the average estimated treatment effects for HFLand TBS were 9.86 and 10.45 respectively, the average sample size was 239and the power was 0.849.Several points are worth mentioning here: (1) Around 15%, which is49Table 3.1: The percentage of simulated trials that stopped after each in-terim analysis, along with the percentage that stopped due to declaringnon-inferiority or (incorrectly) declaring inferiority, when conducting interimlooks at every 100 samples (an interim look of 3) with maximum sample sizeof 385. The upper threshold U is adjusted to be 0.9828 to maintain the over-all Type I error rate at 0.05.InterimLookInterimsamplesizePercentageof trialsstoppedStoppedfor: Non-inferiorityInferiority#1 100 25.1% 25.0% 0.1%#2 200 29.7% 29.7% 0%#3 300 18.4% 18.4% 0%Final 385 26.8% 11.8% 0%Overall NA 100% 84.9% 0.1%a non-negligible portion of the trials not being able to draw a conclusionwhen reaching this maximum sample size, (2) while the average sample sizeof 239 is much lower than 385, the power also has been reduced from 91%to 85%. Thus the sample size/power of a Frequentist fixed design is notdirectly comparable to a Bayesian design with the same maximum sample.Additionally, it means that if maintaining power is required, the maximumsample size needs to be increased.In the next two subsections 3.3.1 and 3.3.2, we will evaluate a Bayesianadaptive trial design under two settings, while maintaining a fixed overallType I error rate:1. assessing the power and expected sample size with varying numbers ofinterim analyses, keeping the maximum available sample size fixed at385; and2. evaluating the expected sample size and the maximum sample size un-der varying numbers of interim analyses while maintaining the targetpower at 0.91.In subsections 3.3.3 and 3.3.4, we will evaluate the impacts of randomiza-tion strategies and prior information on trial performances, respectively. In50these two subsections, we will fix the upper threshold U and lower thresholdL, acknowledging that the overall Type I error rate is not guaranteed to bemaintained at the nominal level.3.3.1 Fixed maximum sample sizeFor the first set of simulations, we will investigate the designs with interimanalyses after sample sizes of 385, 193, 130, 100, or 70 have accumulated(i.e., with a possible number of interim looks of 0, 1, 2, 3, or 5 respectively).Similarly to the simulation runs in the basic setup, we tune the U to geta desired overall Type I error rate (0.05) for different number of interimlooks scenarios, and with these determined upper thresholds, we evaluatethe treatment effect estimation, the power, and the average sample size ofdifferent designs. The results are demonstrated in Table 3.2. We furtherplot the number of interim looks vs the expected sample sizes and power,as shown in Figure 3.2. We observed that the trend of the power and thecorresponding average sample size as the number of interim looks increases:the more frequently one conducts interim analysis, the average sample sizewill drop, but so will the power. The treatment effect estimation for botharms tends to be biased: the HFL tends to be underestimated while the TBStends to be overestimated, making the overall effect size overestimated.Table 3.2: After tuning the U to get a desired overall Type I error rate(0.05), the corresponding Treatment effect estimation, the power, and theaverage sample size of different designs. The results are from 30000 runs.Sample sizeper interimlook(Tuned) Up-per thresholdto claim non-inferiority(U)SimulatedType IerrorrateTreatmentestimation(HFL, TBS)Power Averagesamplesize385 (no interimlook)0.9522 0.0501 (10.09, 10.2) 0.908 385193 (1 interimlook)0.9730 0.0499 (9.99, 10.34) 0.873 274130 (2 interimlooks)0.9805 0.0500 (9.89, 10.40) 0.855 251100 (3 interimlooks)0.9828 0.0501 (9.86, 10.45) 0.849 23970 (5 interimlooks)0.9862 0.0497 (9.80, 10.52) 0.839 22951Figure 3.2: The power and average sample size with differentnumber of interim looks while fixing the maximum samplesize at 385. The red curve indicates the power while bluecurve indicates the expected sample size.3.3.2 Fixed target powerTo obtain a more straightforward comparison between the fixed design andthe Bayesian adaptive design, we will fix both the overall Type I error rateat 0.05 and a target power of 0.91 (with a margin of ±0.01) but allow themaximum sample size in the Bayesian design to vary. Specifically, we willadjust the maximum sample size and then tune the U to get the desiredoverall Type I error rate (0.05), and then with this tuned U, we simulate thepower. We then adjust the maximum sample size again to make the targetpower to around 0.91 if needed. This process is shown in Figure 3.1 fromstep 0 to step 6. With this process, we again evaluate different number ofinterim looks of 0, 1, 2, 3, and 5, and the results are shown in Table 3.3.Figure 3.3 shows the maximum sample size and the corresponding expectedsample size with the change of the number of interim looks. It can be seenthat with more interim looks, the required maximum sample size increases,but the expected sample size decreases. The treatment effect estimation52for both arms tends to be biased, similar to what was observed in the firstsimulation setting.Table 3.3: After tuning the maximum sample size and U to get a desiredoverall Type I error rate (0.05), the target power (0.91), the correspondingTreatment effect estimation, and the average sample size of different designs.The results are from 30000 runs.Samplesize perinterimlook(Tuned) Up-per thresholdto claim non-inferiority(U)SimulatedType IerrorrateTreatmentestimation(HFL,TBS)Power AveragesamplesizeRequiredMaximumsamplesize385 (nointerimlook)0.9517 0.0501 (10.09,10.22)0.908 385 385212 (1interimlook)0.9726 0.0499 (9.99, 10.33) 0.907 293 424145 (2interimlooks)0.9808 0.0500 (9.90, 10.41) 0.904 275 435116 (3interimlooks)0.9827 0.0498 (9.87, 10.44) 0.906 264 46480 (5interimlooks)0.9865 0.0497 (9.81, 10.51) 0.903 256 48053Figure 3.3: Maximum sample size and the corresponding ex-pected sample size with the change of the number of interimlooks when power is fixed at around 0.91. The red curveindicates the maximum sample size required while the bluecurve indicates the expected sample size accordingly.3.3.3 Impacts of responsive adaptive randomization (RAR)strategiesFor the third set of simulations, we will evaluate the responsive adaptiverandomization (RAR) strategy. Response-adaptive randomization (RAR) isa strategy used in clinical trials where the probability of assigning a patientto a particular treatment arm changes over time based on accumulating dataabout treatment responses. Unlike balanced randomization (BR), whichassigns patients to treatment arms with fixed probabilities, RAR adjuststhese probabilities to favor treatments showing better efficacy or safety asthe trial progresses [99]. While various probability adjustment criteria havebeen proposed [71, 72], we adopt the criteria used in HECT [23], whichadjusts the assignment probability based on the square root of the posteriorsuperiority probability given the cumulating data.We first fix the upper threshold U to be 0.9828 and the sample size at54each interim look to be 100 and apply RAR. The simulation results showthat the power is 0.789 while the Type I error rate is only 0.0470. In or-der to make it comparable to BR design, we tune the upper threshold Ufrom 0.9828 to 0.9811 to maintain the Type I error rate at around 0.05. Asshown in Table 3.4, compared to BR, applying RAR significantly impactedtrial outcomes, especially in terms of sample size and power. Specifically,RAR required more samples but resulted in lower power. Additionally, coun-terintuitively, the treatment TBS, which was expected to perform (slightly)better, received fewer patients compared to HFL.Table 3.4: Effect estimation, sample size (SS), power, and Type I error rateunder responsive adaptive randomization (RAR) and balanced randomiza-tion (BR) with 3 interim looks (every 100 samples). The upper threshold Uis adjusted to 0.9811 to maintain the overall Type I error rate at 0.05 underRAR.RandomizationstrategyTreatmentestimation(HFL, TBS)SS (HFL, TBS) Power Type I errorrateRAR (9.83, 10.33) 240 (147, 93) 0.798 0.0500*BR (9.86, 10.45) 239 (119, 120) 0.849 0.0501* This Type I error rate is adjusted to 0.050 by tuning upper threshold U from 0.9828 to 0.9811.3.3.4 Impact of prior information under balancedrandomization (BR)Note that the prior variance reflects the strength of our belief about thetreatment effects: a larger variance indicates greater uncertainty, while asmaller variance reflects stronger confidence in the expected effects. Further-more, since both treatments are based on rTMS technology, it is reasonableto expect them to behave similarly—for example, when observing that HFLleads to reductions in participants’ HRSD-17 scores, we would also expectTBS to show a similar trend. This anticipated similarity suggests a correla-tion between the treatment effects, which can be incorporated into the priorby assigning a reasonable covariance structure between the mean responses55of the two treatments.Specifically, denote µ1 and µ2 as the prior mean for the two treatments,σ11 and σ22 as the prior variance for HFL and TBS, respectively, and σ12 =σ21 as the covariance for the two treatments. The correlation of the twotreatments is simply ρ12 =σ12√σ11·σ22 . We will apply a joint prior to modelingthe data. The exact format of the joint prior distribution can be found inAppendix B.1.1.For the fourth set of simulations, we will again fix the maximum samplesize at 385, the upper threshold U to be 0.9828, and sample size at eachinterim look to be 100. We will fix the prior mean as correctly specified(i.e., µ1 = 10.1, µ2 = 10.23) and vary the prior covariance structure. Notethat with U fixed at 0.9828, the Type I error rate is no longer maintained at0.05 when applying an informative prior [100]. Thus, if we strictly controlthe Type I error rate by adjusting U , for example, when using a weaklyinformative prior with σ11 = σ22 = 25, the U needs to be increased from0.9828 to 0.9835 to maintain the Type I error rate at 0.05. With this newthreshold U and this informative prior, the power and average sample sizeare 0.848 and 238, respectively, matching the results shown in Table 3.2,where a vague prior was used.We first fix the prior correlation as 0 (i.e., σ12 = σ21 = 0), and thenchange the variance of the prior distribution for the HFL and TBS (σ11and σ22) from 100 to 4, transitioning from a vague prior to a relativelystrong informative prior. The results are shown in Table 3.5. As shownin Table 3.5, increasing the prior precision (by reducing the prior variance)improves efficiency by increasing power with a smaller sample size. However,this comes at the cost of an inflated Type I error rate.56Table 3.5: Effect estimation, sample size (SS), power, and Type I error ratewith varying prior variance from 100 to 4 using balanced randomization(BR) with 3 interim looks (every 100 samples). The prior mean is correctlyspecified and assumes no correlation between the two treatments.Prior variance(σ11 = σ22)Treatmentestimation(HFL, TBS)SS (HFL, TBS) Power Type I errorrate100 (9.87, 10.45) 238 (119, 119) 0.848 0.05025 (9.88, 10.44) 236 (118, 118) 0.856 0.05310 (9.86, 10.46) 231 (115, 116) 0.864 0.0584 (9.93, 10.40) 221 (110, 111) 0.884 0.068We then vary the prior information by incorporating the correlation be-tween treatment HFL and treatment TBS into the prior covariance struc-ture. We fix the variance of two treatments at 4 (σ11 = σ22 = 4, i.e., rela-tively strong informative), and vary the prior covariance (σ12 = σ21 = 0, 1, 2,and 3). The results are demonstrated in Table 3.6. As observed in Table 3.6,under the simulation setting where the prior mean is correctly specified andthe prior variance is low, the correlation between treatments significantlyimproves trial performance by reducing sample size and increasing power.However, as expected, this efficiency gain comes at the cost of an inflatedType I error rate.Table 3.6: Effect estimation, sample size (SS), power, and Type I error ratewith varying prior covariance from 0 to 3 using balanced randomization(BR) with 3 interim looks (every 100 samples). The prior mean is correctlyspecified, and the prior variance is fixed at 4 (i.e., σ11 = σ22 = 4).Covariance(σ12) (correla-tion)Treatmentestimation(HFL, TBS)SS (HFL, TBS) Power Type I errorrate0 (0) (9.93, 10.40) 221 (110, 111) 0.884 0.0681 (0.25) (9.94, 10.38) 215 (107, 108) 0.897 0.0792 (0.5) (9.97, 10.35) 203 (101, 102) 0.915 0.1023 (0.75) (10.01, 10.24) 163 (131, 132) 0.951 0.141573.4 DiscussionUsing plasmode simulation based on the THREE-D study, we comparedthe performance between the conventional Frequentist design and Bayesianadaptive design, and, typically, showed within Bayesian adaptive designs:(1) how the power and sample size change with the number of interim lookswhen the maximum sample size is fixed, (2) how the maximum sample sizeand expected sample size change with the number of interim looks when thepower is fixed, (3) how the randomization strategies (i.e., RAR) and (4) theprior information would impact the trial performances.Compared to a Frequentist fixed sample size design, a Bayesian group-sequential design can reduce the expected number of participants neededto reach a definitive conclusion. However, maintaining the same power andType I error rate as the fixed design requires an increase in the maximumsample size for the Bayesian group-sequential design. When Type I errorrate control is essential, the efficiency gains of the Bayesian group-sequentialdesign arise primarily from conducting interim analyses, rather than fromswitching from a Frequentist to a Bayesian framework. Frequentist group-sequential designs can also achieve similar reductions in sample size or in-creases in power through interim analyses. However, they lack a mechanismto incorporate prior information, which can be a key advantage of Bayesianmethods. Additionally, the statistical summaries produced by Bayesian ap-proaches, such as posterior probabilities, are often more intuitive and inter-pretable for stakeholders, making them more appealing for communicatingtrial results.It is important to note that, although we controlled the overall Type Ierror rate at 0.05 in the first two sets of simulations, this was done purely tofacilitate comparison of the adaptive Bayesian design with the Frequentistdesign. However, a fixed value for the Type I error rate may not be rele-vant in real-world practice. For example, in comparative effectiveness trials(where both treatments may be equally likely to be the better one), trialresults are often interpreted with a “balance of evidence” perspective ratherthan with a goal of rejecting one treatment outright, which suggests differ-58ent decision criteria would be more appropriate. In the THREE-D trial,an upper threshold U of 0.9, say, may be deemed more interpretable andclinically relevant, and in this case, one might choose to evaluate the trialperformance with U = 0.9 fixed in the decision rule and report the resultingType I error rate as a supplementary measure rather than adjust the designto maintain a Type I error rate of 0.05. The use and interpretation of TypeI error rates become even more problematic when multiple testing is con-ducted, and arguments have been made in support of ignoring adjustmentsfor Type I error rates [101].In the second set of simulations, we showed that the power lost dueto including interim analyses can be recovered by increasing the maximumsample size, and that the expected sample size continues to be substan-tially below ( 25%-35% reduction) the original maximum sample size. Therequired increase in sample size was modest ( 10%) in this study when asingle interim analysis was added, but was more substantial ( 25%) whenfive interim analyses were included. Hence, if in practice there is a sufficientsample size available, an adaptive design would be more efficient on average,but attention should be given to whether it is feasible when the availablenumber of participants is limited. luckily, unlike standalone trials, LHStrials benefit from being embedded within healthcare systems, facilitatingaccess to diverse patient populations and streamlined recruitment processesthrough direct database access and established provider relationships.The greatest changes in performance occur with the addition of a singleinterim analysis. Increasing the number of interim looks beyond 3 in thiscase yielded small incremental changes in performance. These trade-offsare key considerations in Bayesian adaptive designs: while more interimanalyses provide opportunities to stop the trial sooner—benefiting resourceefficiency, this increased flexibility may lead to a higher chance of missinga true effect, thereby lowering power. In real-world settings, this trade-offis significant because trial designers must balance the ethical and logisticaladvantages of reducing sample size with the scientific goal of maintainingadequate power to detect meaningful effects. As more interim looks typicallyare associated with greater logistical complexity and costs, the appropriate59number of interim looks depends on how the “pay-off” is judged.In the third set of simulations, we also applied Response-Adaptive Ran-domization (RAR) to examine its performance in a non-inferiority trial,where ethical considerations are often cited as a primary argument for usingRAR in clinical trial design [102, 103]. In superiority trials, it has beenpointed out that while there is a risk that more participants will be allo-cated to an inferior treatment in any given trial, but on average across manytrials, more participants will receive the better treatment [104]. However,we observed that in our scenarios, on average more participants received theinferior treatment, i.e., RAR in a non-inferiority trial can on average allocatefewer participants to the treatment that in truth works better. Specifically,in our setting, where the two treatments had very similar effects but with anegative non-inferiority margin (-2.25), RAR tended to randomize more par-ticipants to the treatment that initially appeared more effective (HFL), evenwhen TBS was slightly better. This imbalance occurred because RAR basesrandomization on treatment effect estimates, and the adaptive trial’s inher-ent early termination often ends runs that already show sufficient evidenceof non-inferiority. Consequently, trials where HFL appeared to outperformTBS were allowed to continue longer, resulting in more participants beingrandomized to HFL. Therefore, when designing a non-inferiority adaptivetrial, caution should be exercised when considering the use of RAR.In the fourth set of simulations, we evaluated trial performance whenapplying informative priors by incorporating information into the prior vari-ance and covariance structures, respectively. Under our simulation set-ting, where the prior mean was correctly specified, using informative pri-ors—either through prior variance or covariance—resulted in increased powerand reduced sample size requirements [20]. However, applying these infor-mative priors also led to an inflated Type I error rate. Furthermore, if theType I error rate was strictly controlled by further tuning U under this sim-ulation setting, no efficiency gains were observed, consistent with findingsfrom a previous study [105].Lastly, the early termination feature of adaptive designs inherently car-ries a risk of biased estimation. If an interim analysis shows efficacy that60meets a predefined threshold, the trial may be stopped early, reducing theopportunity to gather further data for accurate estimation. As demonstratedin both sets of our simulations, the more interim analyses conducted, thegreater the bias in effect size estimates. Therefore, when analyzing datafrom trials that stop early, adjustments to treatment effect estimates shouldbe pre-planned (as recommended by the FDA [62]), though at this time,methods for doing so are not well-developed. However, if the goal of anadaptive design is decision-making rather than treatment effect estimation,such as in Chapter 2 where the objective is to identify the optimal treatmentarm from multiple options, the bias observed in this scenario may not posea significant concern.61Chapter 4Constructing a priordistribution for a treatmenteffect from two observationaldatasets when knownconfounders are notmeasured in one dataset4.1 IntroductionWithin a learning health system (LHS), as data collection progresses, richerinformation will be acquired over time, meaning new datasets may containnewly observed covariates (especially confounders) that the earlier datasetsdid not have. Thus, the handling of missing data when estimating the effi-cacy of a treatment is a very important task in building a learning healthsystem (LHS) [4], since trials embedded within an LHS can be more ef-ficient if existing data can be used to inform the construction of a priordistribution for the treatment effects being investigated. Relying solely onearlier datasets can lead to biased treatment effect estimates due to the lackof adjustment for these confounders. While newer datasets, which containmore comprehensive information, may yield less biased or even unbiased es-timates, the earlier datasets still hold valuable information that can improveestimation precision if appropriately incorporated. Therefore, it is essential62to consider methods that effectively integrate both earlier and more recentdatasets to enhance the overall efficiency and accuracy of estimation. Inthis paper, we will refer to datasets that include newly observed covariatesas “new data” and datasets collected earlier that do not capture these co-variates as “old data”. We propose and evaluate various Bayesian methodsfor analyzing the old and new data combined for more efficient treatmenteffect estimations. The objective of these methods is to construct posteriordistributions for the treatment effect, which can ultimately serve as a priordistribution for the trial.One of the most common approaches to handle missing data is theBayesian Markov Chain Monte Carlo (MCMC) imputation approach, inwhich missing data are treated as unknown parameters whose values can beimputed using a model that incorporates prior information and the likeli-hood of the observed data [106–109]. Specifically, MCMC is a computationalmethod used to approximate the posterior distribution of the parameters ofinterest and is usually considered as a gold standard approach when the im-putation model is correctly specified. Yet its practical application remainschallenging, primarily due to the complexity involved in randomly drawingfrom the posterior distribution, a process that is complex and computation-ally demanding [110]. In our context, we will apply the Bayesian MCMCimputation approach to analyze the old and new data combined by treatingthe missing confounders of the old dataset as missing random variables.Alternatively, in adherence to the spirit of Bayesian inference, it is nat-ural to consider converting the information obtained from the old datasetsinto a prior distribution which would then be used as an informative prior inthe analysis of the new data. However, the prior distribution obtained fromold datasets may be subject to bias due to potential missing confounders. Toovercome this problem, we thus first propose a bias-corrected prior, whichutilizes some information from the new data to de-bias the prior from theold. We then can obtain a “Na¨ıve posterior” (NP) by analyzing the newdata together with the bias-corrected prior. However, although the “Na¨ıveposterior” is bias-corrected, it may overestimate precision because it involvesre-use of the new data. Noting that both the de-biased prior and the new63data (analyzed using a diffuse prior) provide unbiased estimators for thetreatment effect, we further propose to combine these two estimators intoan “Honest Precision posterior” (HP) derived by taking a weighted aver-age of the two estimators in which the weights are chosen to minimize thevariance (of the treatment effect), while appropriately accounting for thenon-independence of these two estimators due to both relying on the newdata.The paper will consist of the following 3 sections: in section 2, we willderive the mathematical formulae for the bias-corrected prior, and then forthe NP and the HP. In section 3, we will set up simulations to comparethe statistical performance of our proposed NP and HP approaches to theBayesian MCMC imputation approach in terms of the: (a) bias, (b) pre-cision, (c) coverage, and (d) computational burden. In section 4, we willsummarize our findings, discuss the simulation results patterns, and addressthe limitations.4.2 Methods4.2.1 Problem setup and derivations of some keyestimators:In the following paragraphs, we will use different colors to distinguish typ-ical symbols and clarify whether they were derived based on new data orold data. Specifically, red will be used to indicate the components usingnew data, blue for old data, and normal black color will be used for thecombined data and other general expressions. This color coding will only beapplied in cases where the representation of the notations is not straightfor-ward to distinguish, rather than universally, to ensure clarity while avoidingunnecessary visual complexity. We assume both the old and the new dataare observational data, and come from the same population, meaning theyshare the same underlying data generating model. We illustrate our methodby considering a simple scenario where we have one old dataset and one newdataset as mentioned above. Suppose we have a full (true) linear model that64can be fit to the new dataset:Y = Xβ + Zγ + ϵ1, ϵ1 ∼ N (0, σ21) (1)and a reduced model that omits variable Z, representing the model thatcan be fit to the old dataset and new data:Y = Xβ∗ + ϵ2, ϵ2 ∼ N (0, σ22) (2)Y = Xβ∗ + ϵ12, ϵ12 ∼ N (0, σ212) (2’)When Z is correlated with X, equations (2) and (2’) will generate biasedestimators of β∗. Our purpose is to use the information from the new datato correct the bias introduced to β∗. We will address this within the contextof the Bayesian framework.First, in the full model (1), if we further defineα =[βγ], H = [X,Z] (i.e., Y = Hα+ ϵ1)and we put conjugate priors on α:α ∼MVN (µ0,Σ0)then the posterior isα|Y ∼MVN (µ,Σ)whereΣ =(1σ21H⊤H +Σ−10)−1(3)andµ = Σ(1σ21H⊤Y +Σ−10 µ0)(4)Assumption 1:Assume we start from a very diffuse prior, i.e., where the diagonal elements65of Σ−10 approaches zero.Then the posterior covariance matrix and mean will take the followingforms:Σ = σ21(H⊤H)−1 (5)µ = (H⊤H)−1H⊤Y (6)which have the same forms as least square (or maximum likelihood)estimators.Our focus is to extract the mean and the covariance parts only for theX components, i.e., for β. Replace matrix H with the partitioned matrix[X,Z] in (5), we will have:Σ =[ΣXX ΣXZΣZX ΣZZ]= σ21[X⊤X X⊤ZZ⊤X Z⊤Z]−1Denote:P = X(X⊤X)−1X⊤,S = Z⊤Z − Z⊤X(X⊤X)−1X⊤Z = Z⊤(I − P )ZThe covariance for the X component is:ΣXX = σ21[(X⊤X)−1 + (X⊤X)−1X⊤ZS−1Z⊤X(X⊤X)−1] (7)Similarly, we can extract the posterior mean part for the X component:pluginµ =[µXµZ]into (6), we obtain the mean structure for X components:µX = (X⊤X)−1X⊤Y + (X⊤X)−1X⊤ZS−1Z⊤(X⊤X)−1X⊤Y− (X⊤X)−1X⊤ZS−1Z⊤Y (8)Equations (7) and (8) are the analytical formulae for the posterior co-66variance and posterior mean of β, under assumption 1 (i.e., starting from avague prior on β).If we go back to the reduced model (2), under assumption 1, the posteriorof β∗ when omitting Z is:β∗|Y ∼MVN (µ∗,Σ∗)where:Σ∗ = σ22(X⊤X)−1 (9)µ∗ = (X⊤X)−1X⊤Y (10)Taking the difference between (8) and (10) yields the bias due to omit-ting Z from the model. We thus propose the following data combinationapproach by first constructing a bias-corrected mean estimator, µc:µc = µ∗ + µX − µ∗ (11)where µ∗ represents the mean structure of (10) from the reduced model byusing old data, µ∗ represents the mean structure of (10) from the reducedmodel by using new data, and µX is the unbiased mean structure (8) fromthe full model by using new data. It can be proved easily that (11) is anunbiased estimator for the true posterior mean. Simplifying (11), we get:µc = (X⊤X)−1X⊤Y + (X⊤X)−1X⊤ZS−1Z⊤(P − I)Y . (12)Note thatµX − µ∗ = (X⊤X)−1X⊤ZS−1Z⊤(P − I)Y ,The key step to derive the variance Var(µc) is to derive the variance ofthis bias-correction term, Var(µX − µ∗).Suppose Z is an n×m matrix, X is an n× p matrix, we regress Z on X67by the following multivariate regression:Z = XΦ+ ϵ,where:• Φ is a p×m matrix (coefficients),• ϵ is an n×m error matrix.We model the error terms in ϵ with the following block structure:vec(ϵ) ∼MVN (0, In ⊗W ),where:• W is an m×m covariance matrix capturing the relationships betweenthe m response variables (columns of Z),• In is the n × n identity matrix, indicating independence across the nobservations (rows of Z).Then Var(µX − µ∗) can be approximated as:Vˆar(µX − µ∗) = (γˆ⊤Wˆ γˆ)(X⊤X)−1 + σˆ12(X⊤X)−1X⊤ZS−1Z⊤X(X⊤X)−1Finally, we have the approximated theoretical variance of the estimatorµc as:ˆVar(µc) = σˆ22(X⊤X)−1+(γˆ⊤Wˆ γˆ)(X⊤X)−1 + σˆ12(X⊤X)−1X⊤ZS−1Z⊤X(X⊤X)−1(13)The details of the derivation steps can be found in Appendix C.1.We thus have this bias-corrected prior, βbcp:βbcp ∼MVN (µc,Var(µc)) (14)A straightforward approach is to apply this bias-corrected prior to an-alyzing new data: we first augment βbcp with a diffuse prior for the extra68components γ (i.e., the missing confounders in old dataset) in the full modelto have an augmented bias-corrected prior αa,bcp:αa,bcp ∼MVN (µa,c,Var(µa,c)) (15)Where µa,c = [µc,0], and Var(µa,c) =[Var(µc) 00 Σ0], in which Σ0 canbe a diagonal matrix with large numbers on diagonals. Then based on (3)and (4), we can derive the “Na¨ıve posterior” (NP), αNP easily:αNP|new data ∼MVN (µNP,ΣNP) (16)Where:ΣNP =(1σ21H⊤H +Var(µa,c)−1)−1(17)AndµNP = ΣNP(1σ21H⊤Y +Var(µa,c)−1µa,c)(18)As mentioned previously, to make full use but avoid overuse of the in-formation from new data, we further propose a fully efficient estimator, µw,by weighting the estimator of (11) and the estimator from the new data:µw = w1µc + w2µX (19)where w1 + w2 = 1.It can be again shown easily that this weighted estimator (18) is unbi-ased.The theoretical variance of this weighted estimator is:Var(µw) = Var(w1µc + w2µX),and it can be approximated by:ˆVar(µw) = w21σˆ22(X⊤X)−1 + (1− w1)σˆ12(X⊤X −X⊤Z(Z⊤Z)−1Z⊤X)−169+ (w21 − w1)σˆ122(X⊤X)−1 + w1(γˆ⊤Wˆ γˆ)(X⊤X)−1+σˆ12(X⊤X)−1X⊤ZS−1Z⊤X(X⊤X)−1 (20)The details of the derivation of equation (20) can be found in AppendixC.2.Suppose the treatment is the i-th column in the matrix X, we wish toselect w1 to minimize the variance with respect to the treatment effect inthe form of (20), and this gives:w1,min =σˆ12(X⊤X −X⊤Z(Z⊤Z)−1Z⊤X)−1ii2σˆ22(X⊤X)−1ii + σˆ122(X⊤X)−1ii+σˆ122(X⊤X)−1ii − ((γˆ⊤Wˆ γˆ)(X⊤X)−1)ii2σˆ22(X⊤X)−1ii + σˆ122(X⊤X)−1ii− σˆ12((X⊤X)−1X⊤ZS−1Z⊤X(X⊤X)−1)ii2σˆ22(X⊤X)−1ii + σˆ122(X⊤X)−1ii(21)Plug equation (21) into equation (20) to obtain the minimized varianceVar(µw)min.We thus have this “Honest Precision posterior” (HP) βHP:βHP ∼MVN (µw,Var(µw)min) (22)Notice that in all the equations derived above, the variance of the twoerror terms σ21 and σ22 will be estimated through the least square methodand in the simulation in section 3, the value of the two variance termswill be replaced by point estimates, meaning the uncertainty of the twoestimators will be ignored. The potential consequences will be discussed inthe discussion section.The details of deriving the variance forms (13), (20), and (21) can befound in Appendix C.1 and C.2.In the following section, we use simulations to evaluate the proposedmethods by comparing the following 3 distributions:(a) the “Na¨ıve posterior” (NP), αNP (16),70(b) the “Honest Precision posterior” (HP), βHP (22), and(c) the Bayesian MCMC imputation approach treating Z as missing ran-dom variable.We will focus on the mean of the treatment effect distributions, thestandard deviations of the treatment effect distributions, and the coveragewith respect to a significant level of 0.05.We treat approach (c) Bayesian MCMC imputation as a benchmarkmethod and we hypothesize that approach (b) would give a too conservativecoverage since it overused the information from the new dataset. Approach(b) “Honest Precision posterior” would be as efficient as approach (c) as itfully used the rest information from the new dataset.Notice βBCP (14) is only considered as an intermediate step to propose(16) NP and (22) HP, it would be less efficient compared to (b) or (c) sinceit is an intermediate step and only uses part of the information from thenew dataset.4.3 Simulation setups and resultsFor demonstration purpose, we assume a univariate Z. We set up the sim-ulation parameters as follows:Z ∼ N (0, 1),Treatment indicator Itrt ∼ Bern(π), logit(π) = 0.1 + a ∗ Z, (23)Y = 1 + 2 ∗ Itrt + b ∗ Z + ε, ε ∼ N (0, 1). (24)We will assume a total sample size of 1000, but with the following 3scenarios indicating different percentages of the old data:• S1: 900 old data and 100 new data, a = b = 0.2.• S2: 500 old data and 500 new data, a = b = 0.2.• S3: 100 old data and 900 new data, a = b = 0.2.71We will evaluate the performances of the above methods in terms of bias,MSE, theoretical SE, empirical SE, coverage, and relative computation speedto the MCMC based on 50,000 simulation runs.The simulation results are shown in Table 4.1, Table 4.2, and Table 4.3.With 50,000 runs, the Monte Carlo simulation ensures that the estimationerrors for all metrics are kept below 1%, ensuring a high degree of precisionfor simulation results [111, 112].Table 4.1: Simulation performances of the 3 methods when there is 90% ofold data, a = b = 0.2Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC 0.002 0.006 0.081 0.079 0.958 1HonestPrecisionposterior (HP)0.001 0.006 0.080 0.080 0.956 11386Na¨ıveposterior (NP)0.001 0.006 0.079 0.080 0.951 11423∗ The computation speed is calculated by treating the speed of MCMC as one unit.Table 4.2: Simulation performances of the 3 methods when there is 50% ofold data, a = b = 0.2Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC -0.001 0.004 0.065 0.065 0.949 1HonestPrecisionposterior (HP)-0.001 0.004 0.065 0.065 0.949 6524Na¨ıveposterior (NP)-0.001 0.004 0.065 0.065 0.949 6420∗ The computation speed is calculated by treating the speed of MCMC as one unit.72Table 4.3: Simulation performances of the 3 methods when there is 10% ofold data, a = b = 0.2.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC -0.001 0.004 0.064 0.064 0.952 1HonestPrecisionposterior (HP)-0.001 0.004 0.064 0.064 0.952 3204Na¨ıveposterior (NP)-0.001 0.004 0.064 0.064 0.952 3352∗ The computation speed is calculated by treating the speed of MCMC as one unit.The mean of w1 are 0.861, 0.481, and 0.097 for the S1 ∼ S3 simulationscenarios, respectively. It can be seen from the table that in all simulationscenarios, the performances of the“Na¨ıve posterior” αNP and the “HonestPrecision posterior” βHP are comparable to the standard approach MCMC,and all methods maintain the coverage at the nominal level. The precisionof the bias-corrected estimator decreased when the percentage of the olddata dropped, as can be seen in Table C.1 in Appendix C3.While in practice the missing covariate Z is not likely to be a strongconfounder, we also changed the coefficient of Z in the simulation model(24) from 0.2 to 2, and 20 to reflect stronger association between Z and theoutcome Y , and the performance of the 3 methods are still comparable, asdemonstrated in Tables C.2 ∼ Tables C.7 in Appendix C3.Additionally, we will evaluate the performance when the association be-tween the treatment effect and the missing confounder Z is strong so wewill change the log-odds ratio from 0.2 to 0.8 in equation (23) to obtainsimulation S4, with 900 old data and 100 new data, a = 0.8, b = 0.2The simulation results for S4 are shown in Table 4.4. It can be seen fromTable 4.4 that the HP method is still comparable to MCMC approach, butNP method is however too optimistic with the coverage of 0.926. The NPis overly precise with its Theoretical SE smaller than its Empirical SE (trueSE), due to the over-use of the new data. The coverage is smaller whenincreased a to 1.2, as shown in Table C.8 in Appendix C3. If relying solelyon old data, the bias of the prior mean is 2.46 in S1 when a=0.2, and the73bias increases to 3.12 in S4 when a =0.8, both indicating a severe bias.Table 4.4: Simulation performances of the 3 methods when there is 90% ofold data, a = 0.8, b = 0.2.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC 0.006 0.011 0.105 0.105 0.952 1HonestPrecisionposterior (HP)0.002 0.011 0.107 0.107 0.952 11495Na¨ıveposterior (NP)0.004 0.012 0.098 0.109 0.926 11250∗ The computation speed is calculated by treating the speed of MCMC as one unit.From the above tables, it is noticeable that the computation speed ismuch faster for the proposed HP method, especially when the missing rateis high. We further evaluated the computation speed of the HP methodcompared to MCMC as the sample size increased. As shown in Figure4.1, increasing the total sample size from 500 to 5000 with a 90% missingrate demonstrated that the computational advantage of HP became morepronounced as the sample size grew.74Figure 4.1: The computation speed of the proposed HP method comparedto MCMC when the missing rate is at 90%.4.4 DiscussionWe introduced an “Honest Precision posterior” that combines old and newdata, by weighting information from both “bias-corrected prior” and estima-tors derived from new data. The weighting aims to optimize the use of bothold and new data, ensuring their effective utilization without overrelianceon either, and the primary goal is to minimize the variance of this weightedestimator. Our findings highlight its statistical performance, showing it tobe comparable to that of the MCMC method across various proportions ofold data. This approach is particularly valuable in scenarios where compu-tational efficiency is critical, such as real-time decision-making in evolvingdatasets.Most importantly, we demonstrated that our method offers superior com-putational efficiency over MCMC, especially with large datasets. Whenhandling missing data, MCMC typically uses data augmentation, treating75missing values as latent variables that are iteratively sampled along withthe model parameters. This process requires additional sampling steps, asMCMC must draw values not only for model parameters but also for eachmissing data point at every iteration. As sample size increases, the numberof missing values also grows, and each missing value contributes to the com-putational load in a non-linear manner due to interdependencies within theposterior distribution [113]. Also, unlike MCMC imputation, our methodavoids the need for a correctly specified imputation model, making it moreuser-friendly.We also demonstrated that using the “Na¨ıve posterior” derived by em-ploying the “bias-corrected prior” for analyzing new data. This “Na¨ıveposterior” has a comparable bias to “Honest Precision posterior”, however,it yields overly optimistic results when the association between the missingconfounder and the treatment becomes stronger due to underestimating thevariance. This strong association leads to a high correlation between µnewand µ∗new, and furthermore, resulted in a high correlation between µnew andµc. This high correlation between µnew and µc will make the NP estimatoroveruse the information from the new data, leading to overly optimistic cov-erage. While in practice we usually expect the missing covariate is unlikelyto be a strong confounder, when it is not a strong confounder, it would beappropriate to use either “Honest Precision posterior” or “Na¨ıve posterior”.This work provides an alternative method of dealing with missing dataother than using imputation approaches and offers a statistically and com-putationally efficient solution. In this work, we used a simple two-datasetexample to demonstrate the combining process. Our goal is to extend thisapproach to combine multiple datasets by using the (Honest Precision) pos-terior from less informative datasets as the prior for richer datasets. Thisproposed method aims to provide an efficient prior for estimating treatmenteffects in trials conducted within a Learning Health System (LHS). However,to combine additional datasets using the HP posterior as the prior for richerdatasets, the partitioned components in equations (7) and (8) would need tobe modified, potentially increasing their complexity. Future work will focuson expanding and refining this process.76Additionally, our proposed method is based on Gaussian outcome andwe derived an analytical form for the mean and variance of “Honest Preci-sion posterior”. Dealing with binary or count outcomes is challenging sincethere is no conjugate prior for the coefficients of the regression. Thus, it isimpossible to get the exact expression of “bias-corrected prior” and “HonestPrecision posterior”. However, it may still be possible to derive closed-formapproximations for the posterior mean structure of the observed covariateregression coefficients, for both new and old data, using Laplace approxima-tion. By following similar principles and steps, we can then further constructthe mean and variance structures of the “bias-corrected prior” and the “Hon-est Precision posterior”. Developing and formalizing this approach will bethe focus of our future work.Lastly, while our proposed method effectively handles datasets with miss-ing known confounders, it does not address scenarios where missing covari-ates may act as effect modifiers. Additionally, since data acquisition in LHSoccurs gradually, the method does not fully explore the variability of treat-ment effects over time influenced by confounding variables, which can occurin practice.77Chapter 5Combining RCT andobservational data toestimate conditional averagetreatment effect—a scopingreview5.1 IntroductionConditional average treatment effects (CATEs), also known as heteroge-neous treatment effects [114, 115], or in some contexts, individualized treat-ment effects [116, 117] (though this can be debated, see [117]), refers tothe average treatment effects (ATEs) conditional on (i.e., within subpop-ulations defined by) values of one or more covariates. Estimating CATEsis valuable across various fields, including marketing, economics, and epi-demiology. For instance, in the medical field, estimating CATEs is key toprecision medicine, as clinicians need to understand the effect of a treatmenton individual patients to make personalized treatment decisions.Randomized Clinical Trials (RCTs) are considered the gold standard forcomparing the causal effects of different treatments. Randomization enablesunbiased comparisons of different treatments by ensuring the characteristicsof the two treatment groups are similar in expectation. However, RCTstraditionally have been designed to estimate ATE in the whole populationand are often unsuitable for estimating CATEs for two main reasons: (1)78the high cost and effort to conduct RCTs lead to limited sample sizes thatmay lack the statistical power to detect treatment effect heterogeneity acrosspatients and high uncertainty in CATE estimates; (2) the strict inclusionand exclusion criteria used in an RCT result in a highly selected samplethat may under-represent subpopulations of interest [118–120]. Observa-tional data (OBS), also referred to as real-world data, typically is collectedpassively from larger, more representative populations than those in trials,with a common example being data collected from electronic health records.However, due to the lack of randomization, treatment groups may not besimilar, so confounding adjustment is necessary when estimating the dif-ference in treatment effect. Even after adjustment, estimates may still bebiased due to unmeasured confounding [121].Various methods have been proposed for estimating CATE using obser-vational data, and new methods continue to emerge [122–128]. However,these methods often rely on strong assumptions (such as no unmeasuredconfounding or the availability of instrumental variables), and as a result,the estimates may still suffer from confounding bias. To address these limi-tations, a natural approach is to combine RCT and OBS data to improve theprecision of the estimates. This process, sometimes referred to as data inte-gration [129] or data fusion [130], has gained considerable attention in recentyears. The combination of RCT and OBS data leverages the internal valid-ity of RCTs and complements it with the richness and external validity ofobservational data. In recent years, both the U.S. Food and Drug Adminis-tration (FDA) and the European Medicines Agency (EMA) have recognizedthat real-world data and RCT data can complement each other [131, 132].They have also provided guidelines and granted regulatory approvals forstudies employing rigorous methods to combine RCT and real-world data[133–135].Several recent reviews on data integration methodologies have been pub-lished, including those by Colnet et al. [43], Yang and Wang [136], andBrantner et al. [137]. These reviews explore various purposes for combiningdatasets, with one focus being the estimation of CATE. While our reviewincorporates some of the papers identified by these reviews, we also include79some other most recent works. The main difference of our review is that wesummarize the method each individual paper was using, especially with theassumptions that the methods are based on or motivated by. We wish toprovide a toolbox of methods for estimating CATE using combined datasets,together with the assumptions the corresponding papers are required for in-terested readers to refer to.5.2 MethodsWe conducted this review in three stages, based on the scoping review guide-lines by Arksey et al. [138], by: (1) identifying the relevant papers by elec-tronic databases, reference lists, and hand searching of key journals; (2)selecting the relevant papers based on the inclusion criteria and extractingthe information of interest from the papers; (3) summarizing and reportingthe results.5.2.1 SearchingOne challenging part of searching is this topic has emerged in many researchareas: not only in health, statistics, or mathematics, but also in computerscience, economics, electronic engineering, and business. Also, this topicis emerging and getting more attention in the most recent three years, somost of the papers are still in preprint (e.g., [48]). This greatly limitedthe appropriate databases that could be used. We try to avoid limitingthe search to a certain area since the methods proposed are potentiallygeneralizable to different research areas as mentioned above. We thus useGoogle Scholar as the search engine as it captures research articles in allresearch areas, and it also indexes preprint articles. We searched GoogleScholar from inception to June 4th, 2023. The searches were limited todocuments written in English.The search keywords are made up of 4 parts: the first part is the key-words that describe the “conditional average treatment effect,” and thissometimes can also be used as “individualized treatment effect” or “treat-80ment effect heterogeneity” or “heterogeneous treatment effect”; the secondpart is the keywords that show the “combination” of the datasets, and wehave the keywords as “combination” or “combine” or “combining”; the thirdpart is the keywords to describe the OBS dataset, which is usually quite con-sistent across different areas, so we use “OBS”; the last part is the keywordsto describe clinical trial data, so we use “trial” or “experimental” to capturethat. Overall, the relevant papers should contain all of the above 4 aspects,so we have the following search keywords and logic:("conditional average treatment effect" OR"individualized treatment effect" OR "treatment effectheterogeneity" OR "heterogeneous treatment effect") AND("combination" OR "combine" OR "combining") AND "OBS" AND("trial" OR "experimental")5.2.2 Study selection and data extractionA paper was considered relevant if it focused on the methodology of esti-mating CATE by combining RCT and OBS datasets. Any publication typewas considered as long as the paper met the inclusion criteria. We, how-ever, excluded studies that referenced and applied pre-existing methodologyfor estimating CATE using combined datasets but did not investigate (e.g.,through analytical formula derivation or simulation) the statistical perfor-mance of such methods. We excluded papers that focused on estimatingATE, or estimating CATE using OBS or RCT datasets only. We also ex-cluded papers proposing methods based on a single type of dataset but laterapplied their proposed methods to a combined dataset of RCT and OBS.The reviewer (DZ) independently reviewed the title and abstract of eachpaper identified by the Google Scholar search engine based on the inclusioncriteria above. Any uncertainties were discussed with clarifications made tothe inclusion criteria as needed, as per scoping review methods. Full-textarticles were retrieved if the inclusion criteria were met or if the abstractdid not contain sufficient information; full texts were screened in duplicateby the independent reviewer, and the disagreements were resolved through81discussion including other authors (HW).Data were extracted by the reviewer (DZ). Extraction was guided bya template matrix developed for this review and approved by all authors,which included data on: citation details (authors, paper title, publicationyear, etc.), focus of the paper (key assumptions, key methodologies), avail-ability of the code the statistical models used, and the main findings.5.2.3 Notations and assumptionsIn this paper, we define the following notations:• Y : potential outcomes, where Y (0) and Y (1) are the potential out-comes for untreated and treated cases, respectively.• A: binary treatment indicator, with A = 0 for untreated and A = 1for treated.• X: vector of measured covariates.• E: vector of effect modifiers.• U : vector of unmeasured covariates.• S: indicator for patient belonging to RCT (S = 1) or OBS (S = 0).• e(X,S): propensity score of treatment for a patient with covariatepattern X in RCT (S = 1) or OBS (S = 0).• m: sample size of the OBS study.• n: sample size of the RCT.We adopt the following standard assumptions, most of which are com-monly used in estimating ATE in target populations [139]:1. Consistency of potential outcomes: Y = Y (a) if A = a, for a = 0 or 1.2. Unconfounded RCT: Y (a) ⊥ A | (X,S = 1) for a = 0 or 1.823. Treatment positivity (treatment overlap): 0 < P (A = a | X = x, S =s) < 1 for a = 0 or 1, almost surely.4. Selection exchangeability (no outcome modification): Y (a) ⊥ S | Xfor a = 0 or 1.5. Generalizable OBS study: P (S = 0 | X = x) = P (S = 0).6. Transportability of RCT: E(Y (1)− Y (0) | X = x, S = 1) = E(Y (1)−Y (0) | X = x).We also consider a second category of assumptions that are optional butimportant in practical applications:7. All confounders are measured in the OBS dataset: Y (a) ⊥ A | (X,S =0) for a = 0 or 1.8. Population overlap: 0 < P (S = s | X = x) < 1 for all x and s = 0 or1.9. Effect modifiers are measured: E ⊆ X.These assumptions are foundational for combining RCT and OBS dataeffectively. Some of these assumptions (e.g., assumptions 1 through 6) arestandard and typically implicitly used but may not always be explicitlystated. Other assumptions (e.g., assumptions 7 and 9) are essential forspecific methods and underpin their theoretical framework.5.3 ResultsFrom the main search of Google Scholar, 18 papers met the search criteria.Fig. 5.1 summarizes the results of the search process.The earliest publication was from 2016, and 14 (78%) appeared in 2022or later. Ten (56%) papers have been published in journals or conferences,while 8 (44%) are in preprint (ArXiv). Based on the key assumptions regard-ing whether there is unmeasured confounding in the OBS study (assumption83Figure 5.1: PRISMA flow chart of identification, screening, eligibility, andinclusion of studies.7), there are 3 (17%) papers assuming no unmeasured confounding in OBSdatasets, while 15 papers (65%) relaxed this assumption to allow unmea-sured confounding in OBS datasets.For the papers allowing unmeasured confounding, 7 (39%) papers areestimating CATE by combining separately estimated CATE from the OBSstudy and RCT, and 8 (44%) papers are estimating CATE by using RCTdata to correct/debias the unmeasured confounding in OBS datasets. Thekey information of each paper, including the requirement of the assumptionsand the key approach (or motivations), is summarized in Table 5.1.In the following subsections, we classify the papers mainly based on thekey assumption 7, i.e., whether assuming there is unmeasured confounding in84the OBS dataset, while in each subsection the papers may still have differentassumptions. Though some works [48, 51, 52, 140–144] have already beenreviewed and discussed in detail in the two review papers [43, 137], webriefly summarized the methods and assumptions of these papers, togetherwith other papers that are new. The readers can refer to these two reviewsor the original papers for details if interested.Table 5.1: Summary of each paper in terms of the assumptions and the keyapproach/motivation.PaperStandardassumptions(1–6)Confoundingall measuredin OBS(7)Covariatesoverlap inRCT & OBS(8)Effectmodifiersidentified(9)Keyapproach/motivation*Yang et al. 2023 [52] y n y y combineCao and Yu 2023 [145] y n y y combineZhao et al. 2023 [146] y n y y combineLin and Evans 2023 [147] y n y y combineRobertson 2023 [148] y y y y combineRosenman et al. 2023 [142] y n y y combineChalkou et al. 2023 [149] y n y y de-biasHatt et al. 2022 [141] y n n y de-biasCheng and Cai 2022 [140] y n y y combineLi et al. 2022 [150] y y y y combineRosenman et al. 2022 [143] y y n y combineWu and Yang 2022 [51] y n n y de-biasYang et al. 2022 [144] y n n y de-biasWu and Yang 2022 [151] y n y y combineLadhania et al. 2020 [152] y n y n de-biasFerna´ndez-Lor´ıa and Provost 2020 [153] y n n y de-biasKallus et al. 2018 [48] y n n y de-biasPeysakhovich and Lada 2016 [154] y n y y de-bias* Here, ‘combine‘ indicates combining separated estimators, ‘de-bias‘ indicates de-biasing OBS using RCT.5.3.1 Assuming no unmeasured confounding in the OBSdatasetSometimes it might be plausible to assume there is no unmeasured confound-ing in the OBS dataset when the covariates are observed extensively, withfew confounding bias left in the OBS study [150]. Under this assumption,the estimation bias is thus not a big concern and the methods proposed areusually focused on how to deal with the effect modifiers and how to combinethe CATE estimates across the subgroups in both RCT and OBS datasets,85especially when the covariate distribution between RCT and OBS is notoverlapped (i.e., assumption 8 is violated).Rosenman et al. investigates the CATE by estimating the stratifiedATE, i.e., the ATE within each subgroup [142, 143]. In their first work [143],they investigated the “spiked-in” method that directly inserts the RCT datainto the propensity score OBS subgroup, and estimate the subgroup-specificATE. They compared this “spiked-in” estimator with an estimator that isbased on data-driven convex combination of the variance of the RCT es-timator and the MSE of the OBS data estimator. They discover that the“spiked-in” estimator only performs well when the covariate distributions arevery similar across the two data sources (i.e., under assumption 8) and theirdynamic weighted average estimator has low bias regardless of the overlapof the covariate distributions between RCT and OBS study (i.e. when as-sumption 8 is violated). They also pointed out that the performance of theirdynamic weighted estimator is limited to the assumption of no unmeasuredconfounding in OBS dataset (i.e., under assumption 7).Similarly, Robertson et al. [148] also estimate CATE as a combination ofthe ATE in identified subgroups. They proposed a two-step pseudo-outcomeregression modeling method for estimating CATE: In the first step, theycreate the pseudo-outcome that is similar to Kennedy [155] to represent thedifferences between the trial and the target population stratified by a setof effect modifiers by incorporating the probability of trial participation,the expectation of the outcome, and the probability of treatment in thetrial. In the second step, the pseudo-outcome is regressed on the key effectmodifiers to estimate the CATE function. They also construct pointwiseconfidence intervals for the CATE at a specific value of the effect modifiersand uniform confidence bands for the CATE function. It should be notedthat their method is based on the assumption that the OBS data does nothave information on the treatment and outcome, only covariate informationis collected. The method is suitable for “transportability analysis” where theRCT is only a subpopulation of the target population (i.e. when assumption8 is violated).Li et al. [150] also proposed a two-step algorithm called Weighted Multi-86source Direct Learner (WMDL), firstly constructing a pseudo-outcome anda weight value for each data point by plugging in the estimators of nuisancefunctions, and then regressing the pseudo-outcomes on the covariates X us-ing a selected regress learner, solving it as a weighted least square problem.In their work, in contrast to the general assumption that the RCT is usu-ally a subpopulation of the population interest while the OBS study is arepresentation of the population of interest (assumption 5), they have thereversed assumption that the RCT population represents the populationof interest whereas the OBS study is a subpopulation of the RCT. Theirmethod is also robust when assumption 8 is violated.5.3.2 Allowing unmeasured confounding in the OBS datasetWhen relaxing the assumption to allow unmeasured confounding in the OBSdataset, methods to estimate CATE then start to focus on when and towhat extent the OBS dataset should be used, as the estimate is biased dueto unmeasured confounding. The combination strategy is usually based onthe biased-variance trade-off, i.e., to minimize MSE.However, there is a special case that if there is no outcome or treat-ment information but only covariate distribution information is obtained inthe OBS dataset (as Robertson et al. [148]). In this case, the unmeasuredconfounding assumption of assumption 7 becomes less important since theadvantage of using OBS data is to calibrate the RCT population to do a‘transportability’ analysis. An important assumption for this case is thatthere are no unmeasured effect modifiers to modify the outcomes (i.e., as-sumption 4 holds).Under these assumptions, Wu and Yang [151] proposed to use an inte-grative transfer learning method to calibrate the covariate distribution ofthe RCT data to that of the OBS dataset, of which the calibration weightis selected using a cross-validation procedure aiming to minimize the MSEof the resulting estimator. Similarly, Zhao et al. [146] also focused on cali-brating the covariate shift from sample distribution to the target populationby proposing a robust transfer learning framework for estimating the CATE87with right-censored survival data. Both papers require assumption 8 to holdeven though they allow the distributions to be different in two datasets.Estimating CATE by combining separately estimated CATEfrom OBS study and RCTWhen the OBS data contains all covariate, treatment and outcome informa-tion, the methods will then focus on how to utilize all this information moreefficiently. This section focuses on reviewing papers that estimate the CATEseparately in RCT and OBS study and combined in a weighted approach,where the weight is usually determined by the extent of the bias present inthe OBS study.In the second work of Rosenman et al. [142], they extended the assump-tion to allow for the unmeasured confounding in OBS dataset, and adap-tively shrink the biased OBS estimate toward the unbiased experimental es-timate based on the Stein shrinkage literature [156, 157]. Specifically, theyproposed two shrinkage estimators based on two shrinkage structures—theshrinkage by a constant factor, and the shrinkage by a variance-weighted fac-tor, with the aim of minimizing the squared loss of the subgroup ATEs. Bothworks are estimating stratified ATE but it can be extended for CATE eas-ily since the subgroup could be discovered by machine learning approaches[158, 159], which will be a function of the covariates (i.e., assumption 9 canbe relaxed). Similar to the shrinkage estimation method proposed by Rosen-man et al. [142], Cheng and Cai [140] also incorporates a similar shrinkageapproach, but accounting for the estimated degree of bias in the OBS es-timator to yield separate RCT and OBS CATE estimates that minimizeMSE. To estimate CATE functions in RCT and OBS studies, they use asimilar two-step regression approach as Robertson et al. [148], by firstlyconstructing doubly robust pseudo-outcomes based on Kennedy [155], andthen regressed using kernel regression on the potential effect moderators, toestimate the CATE in the RCT and OBS data separately. They also pointedout in theory when the combination should be allowed if the assumption ofall effect modifiers are measured (assumption 6) and/or the assumption of88all confounding are measured (assumption 7) is violated. Lin and Evans[147] proposed a proper likelihood approach to combine datasets, aimingto optimize the CATE estimates with regard to the Mean Squared Error(MSE) as the loss function.In their overall data likelihood function, they make full use of the RCTdata as it is unbiased but put an influence factor as the power of the likeli-hood of the OBS dataset to adjust how much information they would like toutilize. They use a data-adaptive procedure for maximizing the ExpectedLog Predictive Density (ELPD) to select the influence factor that best reg-ulates the information from the OBS data. Similar to Lin and Evans’ ap-proach, Yang et al. [52] also combine separate estimates of the CATE fromthe RCT and OBS data to minimize MSE, again, using a data-adaptiveapproach. However, before combining, they also construct a hypothesis testfirst to determine whether the assumption of unconfoundedness in the OBSdata is sufficiently met based on a “semiparametric efficient score” (SES) asthey defined in the paper. If this null hypothesis is rejected, the ultimateparameters for the CATE are determined only based on the RCT data; ifnot, an elastic integration of both the RCT and OBS data will be used forestimating CATE.Cao and Yu [145] proposed to estimate the CATE for survival outcome.They obtain the estimators from the RCT and OBS datasets using doublyrobust method first, and then form the combined estimator by leveragingthe correlation between the two estimators based on the large sample theory.It should be noted that even though they assume the OBS data have un-measured confounders, these confounders are however measured in the RCTdata so that this information can be used for the unmeasured confoundersin OBS data when the covariates distribution are overlapped (assumption8).89Estimating CATE by using RCT data to correct/debias theunmeasured confounding in OBS datasetThis section focuses on estimating the CATE using RCT data to reduce thepotential bias in the OBS dataset. With this motivation, some methods ex-plicitly constructed a confounding function, which is usually a function thatdemonstrates the bias caused by unmeasured confounding, and assumedparametric or non-parametric forms for this confounding function, and es-timated this confounding function with various modern methods. The finalCATE estimate is thus bias-reduced, but usually still be tuned to be optimalbased on the biased-variance trade-off.With this idea, Kallus et al. [48] assumed that the influence of hid-den confounding is the difference of the CATE between RCT dataset andOBS dataset, and assumed this confounding effect had a parametric struc-ture with low complexity and can be effectively modeled via “experimen-tal grounding.” Their method does not require the assumption 8 to hold.Yang et al. [144] proposed a confounding function which is defined in theOBS study as the difference in potential outcome means between treatmentgroups. They derived the semiparametric efficiency score (SES) and therate-doubly robust integrative CATE estimators under the assumption thatthe CATE function and confounding function both have a parametric form.Wu and Yang [151] then make a slight adjustment on the confounding func-tion proposed by Yang et al. [144] and propose an integrative R-learnerthat extends the framework to allow flexible machine learning methods forapproximating CATE and confounding function.Hatt et al. [141] propose a two-step representation learning approachto combine the OBS data and RCT data. They first use OBS data tolearn a shared structure and then use RCT data to learn the data-specificstructures. They also use a confounding function which has a similar formto Kallus et al. [48]. It should be noticed that all the above four papers donot need the population overlap assumption (assumption 8) to hold—thisis because the parametric structural assumptions are made on the CATEestimator and/or confounding function, which offers an alternative means90for causal generalization [144].Some other works implicitly utilized the “confounding function” concep-tion and proposed a two-step method, by first estimating the CATE withonly OBS data, which is prone to bias, and then using RCT data to de-biasthe CATE estimations.Peysakhovich and Lada [154] focus on time-series data that have re-peated measurements of individuals. They first use OBS time-series datato estimate a mapping from covariates to individual-level CATE. They thenuse RCT data to identify a monotonic, one-dimensional transformation fromOBS-predicted treatment effects to real treatment effects, assuming that thebias in treatment effect estimates from OBS data preserves unit-level rela-tive rank orderings. Ferna´ndez-Lor´ıa and Provost [153] propose a three-stepprocedure to combine OBS and RCT data to optimize the trade-off betweensupervised learning errors and confounding errors (MSE). They first learn acausal-effect model using OBS data which could suffer from high confound-ing bias, and then use both the OBS and RCT data to learn a tree-structuredmodel that estimates the confounding bias, i.e., the deconfounder tree. Fi-nally, the CATE estimations are the results of the predictions made by theOBS model subtracting the estimated confounding.Ladhania et al. [152] proposed a two-study approach: in their Study1, they apply either a parametric approach or non-parametric approach(ensemble supervised machine learning) to the OBS dataset to detect thesubgroups, and use a model-based recursive partitioning approach to identifythe high/low impact subgroups, and validate them by using sample-splittingwith RCT data. Due to Step 1, the assumption that all effect modifiersare identified (assumption 9) is not needed. The main objective of theStudy 1 is to rule out noise so that the bias in CATE is only caused bythe unobserved confounding. In their Study 2, they classify the RCT databased on subgroups learned in Study 1, and then use this classified RCTdata to test for the hypothesized causal relationships in the sub-groups.Chalkou et al. [149] considered a scenario where there exists a baseline riskvariable to be predictable to the final outcome in the dataset. They thusproposed a three-stage model: in the first stage they build a prognostic91model to estimate the baseline risk of the outcome using prognostic factorsas predictors, with OBS data only; in the second stage, they recalibratethese baseline risks using the RCT data. Finally, they use the calibratedbaseline risk as the effect modifier to estimate the CATE using a networkmeta-regression model.5.3.3 Performance comparisonsWhile most of the proposed methods have conducted simulations to evaluatetheir proposed methods, there are only a few works focused on comparingthe performances of the methods identified in this review, probably becausemost of the methods are very new and are not available to be compared.In the work of Lin and Evans [147], they compared their method to someexisting approaches: Green et al. [160], Rosenman et al. [142], and Kalluset al. [48]. In the simulation settings, they created 10 strata and theycompared the overall MSE of the CATE estimates of the 10 strata for eachmethod, using the overall MSE of the inverse probability weighting (IPW)estimates in RCT as the reference. The simulation results show that Linand Evans’s method [147] and Kallus et al.’s experimental grounding method[48] outperform Green et al.’s [160] and Rosenman et al.’s [142] shrinkageestimators. Lin and Evans’s method [147] tends to have the lowest overallMSE when the parametric model for estimating CATE of RCT is correctlyspecified.Wu and Yang [151] also compared the performance of their proposedintegrative R-learner method to the Kallus et al. [48], under two differentsimulation settings: the RCT and the OBS dataset have only partial overlapand the true confounding function is linear, and true confounding functionis non-linear and the strength of confounding bias can be adjusted. Underboth simulation settings, the integrative R learner method outperforms themethod of Kallus et al. [48].925.4 DiscussionThe aim of this paper is to perform a scoping review of the current methodsavailable for combining RCT and OBS data for estimating CATE, within allresearch areas since the methods will be highly shareable among differentresearch areas. This research topic is rapidly growing and most of the papers(78%) have been published since 2022, while almost half (44%) of the worksare still in the preprint stage.As can be seen from the identified papers, combining RCT with OBSto estimate CATE is inherently a bias-variance trade-off problem: by intro-ducing information from the large, but possibly biased OBS data to limited-scaled, but unbiased RCT data, the aim is to seek to improve the precisionof the causal estimator at the expense of increasing bias, and finally getsthe two optimized based on a certain criterion (e.g., mean squared error(MSE)).The main difference between our review and the other reviews is we arefocused on the assumptions each method requires, as we think this is the keyto the methods proposed in each paper. The assumptions are also the key forinterested readers to pick the suitable methods within the toolbox for theirown datasets under specific scenarios. Besides the 9 assumptions, we do havesome ‘special’ scenarios: for example, (1) in terms of the data structures,Peysakhovich and Lada [154] assumed repeated measurements in the OBSdata, while Cao and Yu [145], Wu and Yang [151], and Zhao et al. [146]focused on survival outcome; (2) some papers [146, 148, 151] have a scenariothat the OBS data contains only the covariate information but without theoutcome and treatment information. There are also many method-specificassumptions (e.g., in Yang et al. [144], the derivation of the semiparametricefficiency score (SES) needs to assume the CATE function and confoundingfunction both have a parametric form; Cheng et al.’s method [140] relieson the correct specification of propensity score or outcome working models;and Lin and Evans [147] approach relies on the correctly specification ofthe CATE model for RCT) within each paper and we omitted them in thisreview.93Because this research topic is growing rapidly and the methods discussedare new, many methods have not been thoroughly compared to one an-other in simulation studies. In terms of the performance comparisons, themethod-specific assumptions mentioned above become crucial. Apart fromthe 9 assumptions mentioned, simulations with different assumptions (usu-ally model specification related, for example, the comparison between Wuand Yang [151] and Kallus et al. [48]) need to be checked to have a validcomparison.We excluded papers that focused on ATE, but it should be noted thatthese methods proposed to use combined datasets to estimate ATE maypotentially be applied to estimate CATE as well. For example, Ilse et al.[161] proposed a causal reduction method in which they use a single latentconfounder that takes values in the same space as the treatment variable,to replace a number of possibly high-dimensional latent confounders, whilemaintaining the covariate distributions of the observational and RCT data.This idea of the dimension reduction of confounders could be utilized forestimating CATE as well. However, since the methods are not validatedfor CATE (i.e., by theoretical proof or simulation), we thus do not includethem. Readers interested in estimating ATE with combined datasets canrefer to these very up-to-date reviews [43, 129, 162].Finally, even though we excluded the papers that solely focused on OBSstudy, the methods for estimating CATE by combining both data sourcesmay not stand alone with the methods of using OBS study only. For exam-ple, most of the papers assumed the effect modifiers are already identified(assumption 9), there is one paper by Ladhania et al. [152] proposing touse either parametric or non-parametric approaches to identify the potentialeffect modifiers first from the OBS study, of which the idea is also mentionedin Rosenman et al. [142]. The first step is only based on OBS data, whichmeans the methods for detecting treatment effect heterogeneity and/or esti-mating CATE in OBS study [163] is a precursor, see [153] for a comprehen-sive review for listing and comparing machine learning (ML) methods forestimating CATE using OBS. Similarly, in section 3.2.2, people are usingRCT data to de-bias the CATE estimates obtained from OBS data because94of unmeasured confounding, the methods of dealing with unmeasured con-founding in OBS data solely can also be potentially useful, or be used as aprecursor as well. Interested readers can refer to Varga et al. [164] for anup-to-date review of methods for dealing with (unmeasured) confounding inOBS study.95Chapter 6Comparing methods forcombining trial data andobservational data toestimate conditional averagetreatment effects6.1 IntroductionWe conducted a review paper specifically focused on CATE in Chapter 5,where we classify the literature based on how each method addresses un-measured confounding in observational data. Given the rapid growth of thisresearch area and the novelty of many methods, thorough comparisons ofdifferent approaches in simulation studies are still lacking. In this Chapter,we compare the performance of three representative methods: (1) elasticcombination approach by Yang et al. [52], which estimates CATE by com-bining separately (and individually) estimated CATEs from OBS study andRCT; (2) experimental grounding approach by Kallus et al. [48]; and (3)integrative R learner approach by Wu and Yang [51], which uses RCT datato correct/debias the unmeasured confounding in the OBS dataset. Eventhough in Wu and Yang’s [51] work, some comparisons were already madeto the approach of Kallus et al. [48], a more comprehensive comparison isneeded while including the elastic combination approach by Yang et al. [52].This chapter is organized as follows. In Section 2, we introduce the nota-96tions and common assumptions within the causal inference framework andreview the key methodologies employed in the three selected papers. In Sec-tion 3, we set up simulations to evaluate the performance of these methodsunder various scenarios, including the strength of unmeasured confoundingand the ratio of OBS to RCT data. In Section 4, we summarize the perfor-mance of the three methods, along with some baseline methods, in terms ofthe average RMSE. We conclude with some final remarks in Section 5.6.2 MethodsThis section introduces the notations and common assumptions within thecausal inference framework and reviews key methodologies employed in theselected studies. In this section, we use the same notations as Chapter 5,but are reproducing them here for convenient reference.6.2.1 Notations and assumptionsIn this paper, we define the following notations:• Y : potential outcomes, where Y (0) and Y (1) are the potential out-comes for untreated and treated cases, respectively.• A: binary treatment indicator, with A = 0 for untreated and A = 1for treated.• X: vector of measured covariates.• E: vector of effect modifiers.• U : vector of unmeasured covariates.• S: indicator for patient belonging to RCT (S = 1) or OBS (S = 0).• e(X,S): propensity score of treatment for a patient with covariatepattern X in RCT (S = 1) or OBS (S = 0).• m: sample size of the OBS study.97• n: sample size of the RCT.We adopt the following standard assumptions, most of which are com-monly used in estimating ATE in target populations [139]:1. Consistency of potential outcomes: Y = Y (a) if A = a, for a = 0 or 1.2. Unconfounded RCT: Y (a) ⊥ A | (X,S = 1) for a = 0 or 1.3. Treatment positivity (treatment overlap): 0 < P (A = a | X = x, S =s) < 1 for a = 0 or 1, almost surely.4. Selection exchangeability (no outcome modification): Y (a) ⊥ S | Xfor a = 0 or 1.5. Generalizable OBS study: P (S = 0 | X = x) = P (S = 0).6. Transportability of RCT: E(Y (1)− Y (0) | X = x, S = 1) = E(Y (1)−Y (0) | X = x).We also consider a second category of assumptions that are optional butimportant in practical applications:7. All confounders are measured in the OBS dataset: Y (a) ⊥ A | (X,S =0) for a = 0 or 1.8. Population overlap: 0 < P (S = s | X = x) < 1 for all x and s = 0 or1.9. Effect modifiers are measured: E ⊆ X.These assumptions are foundational for combining RCT and OBS dataeffectively. Some of these assumptions (e.g., assumptions 1 through 6) arestandard and typically implicitly used but may not always be explicitlystated. Other assumptions (e.g., assumptions 7, 8, and 9) are essential forspecific methods and underpin their theoretical framework.986.2.2 A review of several existing methodsWe will begin by providing a brief overview of the three aforementionedmethods and then proceed to set up simulations to compare their perfor-mances across various scenarios.Overall, the three selected methods share common assumptions from(1) to (8), with all aiming to minimize the bias-variance trade-off in CATEestimation by combining RCT and OBS data. However, the methods differin their underlying principles of how they address unmeasured confoundingin the OBS data. Table 6.1 summarizes and the major differences amongthese methods.99Table 6.1: Summary of the major differences among the three approachesMethod Covariate X over-lap between RCTand OBSUnderlying principle toaddress unmeasured con-founding in OBSYang et al.[52]Overlap of X be-tween the RCT andOBS samples is re-quired in the prob-lem of generalizabil-ityConstruct a test to gauge thecomparability and reliabilityof the OBS data that are sub-ject to unmeasured confound-ing and decide whether touse the OBS data in an inte-grative analysis, and providea data-adaptive procedure toselect the test threshold thatpromises the smallest meansquare error (MSE)Kallus et al.[48]Support of X in theRCT has some over-lap with that in OBSUse RCT to correct OBS withhidden confounding in causaleffect models trained on OBSdata and estimate the (un-measured) confounding effectby assuming a linear relation-ship with the measured con-founders.Wu and Yang[51]Support of X in RCTis nested within oroverlapping withthat in the OBSDevelop an integrative R-learner for the CATE and con-founding function by leverag-ing RCT and OBS data by im-posing different regularizationterms for the CATE and con-founding functions.Uniformly, we define τ(X) as the CATE function, and τ(x) is the CATEof a patient with covariate pattern X = x:τ(x) = E(Y (1)− Y (0)|X = x) (6.1)100Yang et al. Elastic combinationThe motivation of Yang et al.[52] is to evaluate the comparability and reli-ability of the OBS data that are subject to unmeasured confounding. Theyconstruct a hypothesis test to determine whether the assumption of uncon-foundedness in the OBS data is sufficiently met based on a “semiparametricefficient score” (SES) [165, 166]. If this null hypothesis is rejected, the ul-timate parameters for the CATE are determined only based on the RCTdata; if not, an elastic integration of both the RCT and OBS data is usedfor estimating CATE.Specifically, they assume a linear CATE function τ(X) = ψ⊤X, andthey define the Hψ which mimics the potential outcome Y (0):Hψ = Y − τ(X) ∗A (6.2)Following Robins [166], they showed that E(Hψ | A,X, S) = E(Y (0) |A,X, S)From here, they introduced a semiparametric efficient score (SES) of theparameters ψ. The efficient estimator for ψ can be obtained by solvingm+n∑i=1ˆSESψ(Xi) = 0. (6.3)They proposed algorithms and applied non-parametric methods to ap-proximate nuisance functions involved in solving the above equation.This semiparametric efficient score is used in their hypothesis test todetermine whether there is unmeasured confounding in the OBS data witha null hypothesis H0 of E(SESψ | S = 0) = 0. They constructed a teststatistic T and they show that under H0, the test statistic T follows a Chi-square distribution with degrees of freedom p (where p is the number of theeffect modifiers, E): T ∼ χ2p. Let cγ = χ2p,γ be the 100× (1 − γ) percentileof χ2p, they adaptively select the value of γ with the aim of minimizing theMSE of ψˆ.Finally, if this null hypothesis is rejected, the ultimate parameters ψˆ for101the CATE are determined solely from the RCT. Otherwise, the parametersψˆ are solved from the below equation by combining the SES functions ofboth the RCT and OBS data:∑i∈m+n{Si ∗ ˆSESψ(Xi) + 1(T < cγ)(1− Si) ∗ ˆSESψ(Xi)}= 0. (6.4)We expect this method to perform well across varying strengths of un-measured confounding effects. However, it assumes a linear CATE function,which may result in efficiency loss due to model misspecification if the trueCATE model is non-linear with respect to the measured covariates or in-cludes higher-order terms.For the details of the parameters estimation and their asymptotic prop-erties, there are more discussions in Yang et al. [52] paper.Kallus et al. Experimental groundingKallus et al. [48] assumed that the influence of hidden confounding is thedifference of the CATE between RCT dataset and OBS dataset, and as-sume this unmeasured confounding effect had a parametric structure withlow complexity and can be effectively adjusted via “experimental ground-ing” by the measured covariates. Specifically, they denote the difference inconditional average outcomes in the OBS data as ω(x):ω(x) = E(Y |A = 1, X = x, S = 0)− E(Y |A = 0, X = x, S = 0). (6.5)Due to unmeasured confounding, ω(x) ̸= τ(x). However, the differenceof these two quantities is the so-called “confounding effect,” which can bedenoted as c(x):c(x) = τ(x)− ω(x), (6.6)and they assume the confounding function c(x) is linear:102c(x) = θ⊤x, (6.7)They further defined a signed re-weighting function q(x):q(x|S = 1) = Ae(x, S = 1)− 1−A1− e(x, S = 1) , (6.8)and they have shown that:E (q(x|S = 1) · Y (S = 1) | X = x,E, S = 1) = τ(x|S = 1), (6.9)which indicates q(x|S = 1) · Y (S = 1) is an unbiased estimate of τ(x) inRCT. And this can be used to learn the confounding function c(x) withinthe RCT data (S = 1) as follows:θˆ = argminθn∑i=1(q(xi)Yi − ωˆ(xi)− θ⊤xi)2(6.10)And finally, the CATE for each individual can be estimated as:τˆ(x) = ωˆ(x) + θˆ⊤x (6.11)Consequently, we expect this method to perform well when the measuredconfounding variables are not complexly associated with the confoundingeffect induced by unmeasured variables and this confounding effect can beaccurately modeled using linear models, but might be less efficient if theconfounding effect is complex and nonlinear.Wu and Yang, Integrative R learnerMotivated by the R-learner method proposed by Nie and Wager [167], Wuand Yang [51] further proposed an integrative R-learner method that al-lows flexible machine learning methods for approximating the CATE andconfounding function.Especially, they have the same definition of the confounding function asKallus et al. [48] showed in Equation (6.6) except they defined the function103inversely:c′(x) = ω(x)− τ(x) (6.12)One key step to identify and estimate the CATE and confounding func-tion of this work is introducing the residual ϵ:ϵ = Y − E(Y |X,A = 0, S)−A ∗ (τ(X) + (1− S) ∗ c′(X)) (6.13)And they showed the fact that E(ϵ|X,A, S) = 0.Define the conditional mean outcome µ(X,S) as:µ(X,S) = E(Y |X,S) (6.14)Then the conditional mean outcome µ(X,S) can be rewritten as:µ(X,S) = E(Y |X,A = 0, S) + (τ(X) + (1− S) ∗ c′(X)) ∗ e(X,S) (6.15)Combining (6.13) and (6.15), they have:ϵ = Y − µ(X,S)− (τ(X) + (1− S) ∗ c′(X)) ∗ (A− e(X,S)) (6.16)which motivates the mean squared error loss function minimization prob-lem to identify the CATE function and confounding function (τ(X), c′(X)).They then first proposed to use cross-fitting to estimate the nuisance func-tions µ(X,S) and e(X,S), resulting in the following empirical loss functionfor estimating τ(X) and c′(X):104{τˆ , cˆ′}= argminτ,c′[PN{Yi − µˆ−k(i)(Xi, Si)−(τˆ(Xi) + (1− Si)cˆ′(Xi))∗(Ai − eˆ−k(i)(Xi, Si))}2+ Λτ + Λc′](6.17)Where PN (Xi) = N−1∑i∈n+mXi,denotes the empirical measure overthe combined RCT and OBS data, Λτ and Λc′ are regularizers on the com-plexity of the function τ(x) and confounding function c′(x) to avoid overfit-ting.They subsequently described the integrative R-learner adopting seriesestimation of the CATE function τ(x) and confounding function c′(x). Theyalso showed that their proposed integrative R-learner method to combineRCT and OBS data is consistent and at least as efficient as the R-learnerbased only on RCT.Thus, this method tends to be more robust to complex confoundingeffects and flexible to the form of true CATE functions. It might also bemore efficient for out-of-sample prediction, as it employs regularization tech-niques to prevent overfitting. However, it might be less effective than a na¨ıvecombination when OBS data has weak influences from the unmeasured con-founding effects.For more information, one can refer to the paper [51] for details.6.3 Simulation experimentsWe will generate data based on two different procedures.The first set of simulations (S1) is similar to Wu and Yang [51] butwith some modifications: we generate OBS samples first and then generateRCT samples; when generating RCT samples, we generate the RCT selectionprobability based on the covariates and then sample from the rest population(i.e., after sampling OBS samples) based on the RCT selection probability.Specifically, we simulate data based on the following 3 steps:105Step 1: Generate population with counterfactuals(1) X1, X2, X3 ∼ N (0, 1), X1 and X2 are effect modifiers and X3 is theunmeasured confounder; (population size of N).(2) τ(X) = 1 +X1 +X2,(3) Y (a) = 1 +X1 +X2 +X3 + a ∗ (1 +X1 +X2) + e, which means:Y (0) = 1 +X1 +X2 +X3 + e, Y (1) = 1 +X1 +X2 +X3 + τ(X) + e.Step 2: Generate OBS samples(1) OBS samples are generated from the N population with sample sizem by simple random sampling.(2) Then generate the treatment assignment probability/propensity scoree(X,S = 0) for these m samples based on:logit(e(X,S = 0)) = −1−X1 +X2 + bX3 (6.18)The value of b reflects the strength of unmeasured confounding, withthe larger absolute b value indicating a larger confounding bias. Thetreatment indicator for OBS data is thus A|(S = 0) ∼ Bern(e(X,S =0)).Step 3: Generate RCT samplesFor the RCT samples generation, we investigated the following 2 gener-ating options:(1a) [S1a, patient characteristics in RCT and OBS data fully overlapped]generate RCT selection probability π to select RCT samples from theN −m population: logit(π) = −n0 −X1 +X2, and randomly samplen RCT samples from N − m population without replacement withselection probability π.This RCT generating process makes the RCT samples and OBS sam-ples fully overlap.106(1b) [S1b, patient characteristics in RCT data have poor overlap with theOBS data] randomly select n samples from the N−m population withsome inclusion criteria. Here we set X2 > 0 as inclusion criteria forRCT samples. This means the RCT samples and OBS samples areonly partially overlapped.(2) Then for the selected RCT samples, the treatment assignment proba-bility will be 0.5: e(X,S = 1) = 0.5.We will fix the total population size and OBS size at N = 100000, andvary b ∈ {0, 0.5, 1, 1.5, 2.5, 5} to reflect different strengths of unmeasuredconfounding. For the (S1a) procedure to generate RCT samples, we setn0 = 7 to generate the probability of selecting each RCT sample. We willvary m ∈ {500, 2000} and fix the RCT sample size n at 200. For the (S1b)procedure, we will fix m = 500 and RCT sample size at n = 200.In the second set of simulations (S2), we will use the same procedure asKallus et al. [48], where we assume a linear confounding function and withonly partial covariate overlap.Specifically, we assume X1 is an effect modifier and X2 is an unmeasuredconfounder, then we generate the RCT data with sample size n as follows:X1 ∼ Uniform[−1, 1], X2 ∼ N (0, 1), A|(S = 1) ∼ Bern(0.5).Next, we generate the OBS data with sample size m as follows: A|(S =0) ∼ Bernoulli(0.5), and the observed effect modifier X1 and the unobservedconfounderX2 are sampled from a bivariate Normal distribution with samplesize m:(X1X2)∣∣∣∣∣A|(S = 0) ∼ N((00),(1 A|(S = 0)− 0.5A|(S = 0)− 0.5 1)).For both RCT and OBS data, we generate the outcomes:Y = 1 +X1 + 0.5X12 +X2 +Aτ(X1) + 0.5e,107where τ(X1) = 1 + 2X1 + 0.75X12 and e ∼ N (0, 1). Then by the definitionof the confounding function, we can get c(X) = E(X2 | X1, A = 1, S =0) − E(X2 | X1, A = 0, S = 0) = X1. We will vary m ∈ {500, 1000, 2000}and fix the RCT sample size n at 200.Finally, for all three simulation settings (S1a, S1b and S2), we use theroot mean squared error (RMSE) as proposed in Wu and Yang [51] as thecriterion metric to evaluate the performance of these three methods:RMSE(τˆ) =√√√√ 1NN∑i=1(τ(X(i))− τˆ(X(i)))2 (6.19)6.4 ResultsWe evaluate the following 3 methods based on averaged RMSE: elastic ap-proach by Yang et al. [52] (“Elastic”), experimental grounding by Kallus etal. [48] (“Kallus”), Integrative R learner by Wu and Yang [51] (“R-int”).In addition, to better understand how the “R-int” approach works, we alsochecked the results from 3 other R learner calculations: R-learner based onRCT (“R-rct”), R-learner based on OBS (“R-obs”), and R-learner basedon naive combination of RCT and OBS assuming there is no unmeasuredconfounding in OBS (“R-na¨ıve”). The results can be found in Appendix D2.To illustrate the accuracy of the three approaches in estimating theCATE, we plotted the percentage difference between each approach andthe true CATE under simulation setting S1a with the OBS size of 500 andb = 0.5. In this scenario, X2 is fixed at 1, while X1 varies from -1.5 to 1.5,as shown in Figure 6.1.108Figure 6.1: Percentage differences between the three methods and the trueCATE under simulation setting S1a with OBS size m = 500 and b = 0.5. Inthis scenario, X2 is held constant at 1.As illustrated in Figure 6.1, when the predictor X2 is fixed at 1 and X1ranges from -1.5 to 1.5, Kallus’s approach consistently underestimates theCATE, whereas “Elastic” and “R-int” tend to overestimate it. However, interms of the percentage difference from the true CATE, Kallus’s approachdemonstrates the best overall performance in this specific scenario. Wefurther evaluated this setting under different values of b (1 and 2.5), ob-serving that the overall pattern remains similar for b=1 but differs notablywhen b=2.5, compared to b=0.5, as shown in Figure D.1 and Figure D.2,respectively. This variation highlights the impact of differing strengths ofunmeasured confounding on performance.To further assess the performance of the three methodologies, we system-atically evaluate the averaged RMSE across various scenarios. The averagedRMSE is calculated based on 5000 simulation runs (with simulation errorsin brackets).According to Table 6.2, when the covariate pattern between OBS and109RCT is fully overlapped, and the sample size ratio of OBS to RCT is at2.5:1 (m = 500, n = 200), Yang’s test-based elastic combination approachperformed best when there is moderate (b is from 0.5 to 1.5) unmeasuredconfounding effect, while Wu and Yang’s integrative R learner approachperformed best when there is no or severe (b > 2.5) unmeasured confoundingeffect; Kallus’s approach did not perform well due to its incorrectly specifiedmodel for the non-linear confounding function c(X).Table 6.2: The averaged RMSE for the 3 methods based on (S1a) procedureto generate RCT samples while the OBS sample size m is 500 with thecorresponding RCT sample size n is 200. The results are based on 5000simulation runs.b Elastic Kallus R-int0 0.41 (0.01) 0.69 (0.02) 0.34 (0.01)0.5 0.48 (0.01) 0.68 (0.02) 0.54 (0.01)1 0.45 (0.01) 0.68 (0.01) 0.59 (0.01)1.5 0.43 (0.01) 0.69 (0.02) 0.54 (0.01)2.5 0.42 (0.01) 0.68 (0.01) 0.44 (0.01)5 0.42 (0.01) 0.68 (0.01) 0.41 (0.01)Table 6.3: The averaged RMSE for the 3 methods based on (S1a) procedureto generate RCT samples while the OBS sample size m is 2000 with thecorresponding RCT sample size n is 200. The results are based on 5000simulation runs.b Elastic Kallus R-int0 0.40 (0.01) 0.68 (0.01) 0.28 (0.01)0.5 0.50 (0.01) 0.69 (0.01) 0.55 (0.01)1 0.47 (0.01) 0.69 (0.02) 0.60 (0.01)1.5 0.43 (0.01) 0.68 (0.01) 0.52 (0.01)2.5 0.42 (0.01) 0.68 (0.01) 0.48 (0.01)5 0.41 (0.01) 0.67 (0.01) 0.42 (0.01)When the sample size ratio of OBS to RCT is at 10:1 (m = 2000,n = 200), as shown in Table 6.3, the overall pattern is quite similar to thoseobserved in Table 6.2. When increasing the OBS data from 500 to 2000,110all three methods have better performances when there is no unmeasuredconfounding or severe confounding (b > 1.5). According to Table 6.2 andTable 6.3, in terms of each method, when increasing the OBS data, Yang’selastic approach and Kallus’s method demonstrate similar and relativelystable performance across different strengths of unmeasured confounding,while Wu and Yang’s integrative R-learner approach have noticeable effi-ciency gain when there is no unmeasured confounding.Table 6.4: The averaged RMSE for the 3 methods based on (S1b) procedureto generate RCT samples, with n = 200 and m = 500. The results are basedon 5000 simulation runs.b Elastic Kallus R-int0 1.50 (0.01) 1.66 (0.02) 1.60 (0.01)0.5 1.55 (0.01) 1.66 (0.01) 1.69 (0.01)1 1.61 (0.01) 1.64 (0.02) 1.71 (0.02)1.5 1.66 (0.02) 1.66 (0.01) 1.70 (0.01)2.5 1.67 (0.01) 1.65 (0.01) 1.62 (0.02)5 1.66 (0.02) 1.64 (0.01) 1.64 (0.01)When the RCT sample is a narrower sample where X2 can only bepositive, as shown in Table 6.4, none of the methods perform well.For the second set of simulations where the confounding function is linearand with only partial covariate overlap, the results are shown in Table 6.5.For different OBS sample sizes, the integrative R-learner approach performsthe best.Table 6.5: The averaged RMSE for the 3 methods based on the secondprocedure to generate RCT samples while the OBS sample size m is 500,1000, and 2000 with the corresponding RCT sample size n fixed at 200. Theresults are based on 5000 simulation runs.m Elastic Kallus R-int500 0.32 (0.01) 0.34 (0.01) 0.24 (0.01)1000 0.32 (0.01) 0.34 (0.01) 0.24 (0.01)2000 0.32 (0.01) 0.34 (0.01) 0.23 (0.01)1116.5 DiscussionWe compared three main approaches of combining RCT and OBS data forestimating CATE, i.e., test-based elastic combination approach by Yang etal. [52], experimental grounding approach by Kallus et al. [48], and integra-tive R learner approach by Wu and Yang [51]. The comparison primarilyused average RMSE as the metric. Overall, our results indicate that whencovariates are fully overlapping between RCT and OBS data and the con-founding function is nonlinear, Yang’s test-based elastic combination ap-proach performs best under moderate unmeasured confounding. Wu andYang’s integrative R learner approach is the most efficient when there iseither no or severe unmeasured confounding, while Kallus’s approach un-derperforms due to an incorrectly specified model for the non-linear con-founding function c(X). These methodologies provide valuable strategiesfor managing different circumstances. For example, the ability of Wu andYang’s integrative R learner approach to efficiently handle scenarios with se-vere unmeasured confounding is particularly beneficial in fields where criticalvariables are difficult to measure or are overlooked. This could include com-plex medical studies where not all patient characteristics are known or canbe controlled [168, 169].Notably, Yang’s elastic approach and Kallus’s approach are relativelystable with respect to the number of OBS samples [52], whereas the perfor-mance of Wu and Yang’s integrative R learner approach varies dependingon the OBS sample size: when the sample size of OBS is increasing, theperformance is better when there is no unmeasured confounding but theperformance will be worse when the confounding effect is strong. Thesevariations are closely related to how each method formulates and optimizesits loss function. Additionally, we observed a trend where the performanceof all three methods initially decreases and then increases as the strength ofunmeasured confounding increases. This is likely because, in these two caseswhere the presence or absence of unmeasured confounding is more evident,it is easier for the algorithm to determine the optimal combination process.In contrast, when moderate unmeasured confounding exists, it introduces112additional uncertainty in the combination process. This is especially chal-lenging for Yang’s elastic approach, which relies on a strict threshold todecide whether to combine the data.When the covariate patterns in RCT and OBS data have poor overlap, asin simulation setting S1b, none of the methods perform well, as they all relyon the transportability assumption (i.e., assumption (6)). In Yang’s test-based elastic approach, the OBS sample is rejected 95% of the time acrosssimulations, leading to results similar to using only RCT data. To achievepotential efficiency gains from OBS data, we assume that the support of Xin the RCT is nested within or overlaps with that in the OBS, as τ(X) isnot identifiable outside the support of X in the RCT.It’s important to note that in the first set of simulations, we assume alinear CATE function τ(X), which means Yang’s elastic combination ap-proach benefits from correct model specifications. In contrast, in the secondset of simulations when there is a quadratic term in the CATE function, Wuand Yang’s integrative R learner outperforms Yang’s elastic combinationapproach by using a series method to approximate τ(X) and c(X), offeringgreater flexibility across different functional forms.The primary metric used to evaluate the three approaches is the averageRMSE, which measures the mean CATE estimation error across all partic-ipants. Therefore, our results primarily reflect overall performance acrossthe entire participant population. However, performance can vary withinspecific ranges of predictor values. For example, in the simulation settingshown in Figure 6.1, Kallus’s approach performs best among the three ap-proaches when X2 is fixed at 1 while X1 ranges from -1.5 to 1.5. The varyingperformances of the three methods across specific value ranges may indicatepotential areas for algorithm refinement and improvement.Finally, all methods assume that all effect modifiers are known and mea-sured (i.e., Assumption 7). However, this assumption may be relaxed asemerging works [152] have developed techniques to detect potential unmea-sured effect modifiers.113Chapter 7ConclusionsIn this concluding chapter, Section 7.1 provides a summary of the key find-ings and implications of the dissertation. Section 7.2 offers suggestions forfuture research directions. Finally, Section 7.3 highlights the knowledgetranslation efforts undertaken as part of this dissertation.7.1 Summary of findings and implicationsThe primary contribution of this dissertation lies in the development of keymethodologies to support the embedding of clinical trials within LearningHealth Systems (LHS). While the work was motivated by challenges in rTMStechniques, the methodologies developed for building LHS are broadly ap-plicable to other health-related domains. This dissertation addresses thefollowing key aspects of LHS development:1. Developed an R package for simulating Bayesian adaptive enrichmentdesigns.2. Addressed critical considerations in the design and evaluation of Bayesiangroup-sequential non-inferiority trial.3. Derived a prior correction method to quickly and accurately combineobservational (OBS) datasets, which may be subject to unmeasuredconfounding, to obtain priors for treatment effects.4. Conducted a scoping review of available data combination methods forestimating Conditional Average Treatment Effects (CATE) and sys-tematically evaluated several key methodologies to provide practicalguidance for researchers and practitioners.1147.1.1 R Package Development: Bayesian AdaptiveEnrichment Design for LHSRandomized controlled trials (RCTs) often fail to assess treatment effect het-erogeneity across subpopulations, which can result in suboptimal treatmentrecommendations and inefficient use of healthcare resources. Addressing thislimitation is a key requirement in building Learning Health Systems (LHS),where the goal is to generate evidence that is directly applicable to diversepatient populations and adaptable to real-world clinical settings [170].To meet this need, Chapter 2 presents the development of an R pack-age, BayesAET, designed to facilitate Bayesian adaptive enrichment designswithin the LHS framework. BayesAET allows users to identify pre-definedsubpopulations that benefit most from treatments, enabling more preciseand efficient clinical trials. To the best of our knowledge, this is one of thefirst R packages that supports the evaluation of multiple treatments for eachsubpopulation in enrichment designs. The package provides several featurestailored to adaptive trial designs including:1. Early termination: treatments/trials can be stopped early for effi-cacy or futility based on posterior superiority probabilities, accelerat-ing the identification of optimal treatments and sensitive subpopula-tions.2. Flexible randomization: users can choose between equal random-ization or responsive adaptive randomization strategies.3. Incorporation of external information: when prior informationabout treatment interactions is available, BayesAET supports a mul-tivariate joint prior structure to enhance trial efficiency.BayesAET is publicly available for installation on GitHub (https://github.com/zdhjeff/BayesAET) and serves as a powerful tool for integrat-ing Bayesian adaptive enrichment designs into LHS.1157.1.2 Impact of design choices on power and sample size inBayesian group-sequential non-inferiority trialIn Chapter 3, we explored the impacts of design choices in Bayesian group-sequential non-inferiority trials, using plasmode simulations based on theTHREE-D study. Key comparisons were made among different Bayesiangroup-sequential designs and with conventional Frequentist designs, high-lighting trade-offs in power, sample size, and interpretability.Compared to a Frequentist fixed sample size design, a Bayesian group-sequential design reduces the expected number of participants required toreach a conclusive result. However, achieving the same power and Type I er-ror rate as the fixed design necessitates a larger maximum sample size in theBayesian group-sequential design. When the Type I error rate is controlled,the efficiency gains of the Bayesian group-sequential design arise from in-terim analyses rather than the transition from a Frequentist to a Bayesianframework. While Frequentist group-sequential designs can similarly reducesample size or increase power through interim analyses, they lack the abilityto incorporate prior information—a notable advantage of Bayesian methods.Furthermore, Bayesian approaches offer statistical summaries, such as pos-terior probabilities, that are often more intuitive and easier for stakeholdersto interpret, enhancing their appeal for communicating trial results.We demonstrated that the loss of power associated with including interimanalyses can be offset by increasing the maximum sample size. Despite thisadjustment, the expected sample size remained significantly lower (approxi-mately 25%-35% reduction) than the original maximum sample size. In thisstudy, adding a single interim analysis required only a modest ( 10%) in-crease in sample size, whereas adding five interim analyses required a moresubstantial ( 25%) increase. Therefore, while adaptive designs are gener-ally more efficient on average when sufficient sample size is available, theirfeasibility must be carefully considered when participant numbers are con-strained.The greatest changes in performance occur with the addition of a singleinterim analysis. Increasing the number of interim looks beyond 3 in this116case yielded small incremental changes in performance. These trade-offs arecritical in Bayesian adaptive designs: although additional interim analysesprovide more opportunities to stop the trial early, which enhances resourceefficiency, this flexibility can also increase the likelihood of missing a trueeffect, thereby reducing power. In real-world scenarios, trial designers mustweigh the ethical and logistical benefits of reducing sample size against thescientific imperative of ensuring adequate power to detect meaningful effects.Adjustments to treatment effect estimates should also be pre-planned [171].Since more interim looks generally entail greater logistical complexity andcosts, the optimal number of interim analyses ultimately depends on howthe trade-offs align with the goals and constraints of the trial.We also examined the performance of the Bayesian group-sequential non-inferiority trial under Response-Adaptive Randomization (RAR). RAR wasproposed primarily due to ethical concerns but has been criticized for its po-tential to randomize participants in the “wrong” direction, a concern oftenhighlighted in the context of superiority trials [104]. However, our findingsin Chapter 3 based on a non-inferiority trial setting suggest that this issue ismore predictable than random. This behavior highlights the challenges asso-ciated with RAR in non-inferiority trials. This imbalance occurred becauseRAR bases randomization on treatment effect estimates, and the adaptivetrial’s inherent early termination often ends runs that already show sufficientevidence of non-inferiority. Consequently, trial designers should carefullyevaluate the appropriateness of RAR in non-inferiority adaptive trials.It is worth noting that, although we controlled the overall Type I errorrate at 0.05 to facilitate a direct comparison between the adaptive Bayesiandesign and the Frequentist design, a fixed Type I error rate may not alwaysbe relevant in real-world practice. For instance, in comparative effectivenesstrials—where either treatment could be equally likely to be superior—resultsare often interpreted from a “balance of evidence” perspective rather thanwith the intent to definitively reject one treatment. This suggests thatalternative decision criteria may be more appropriate. Furthermore, theuse and interpretation of Type I error rates become increasingly complexwhen multiple testing is involved, with some experts arguing against making117adjustments for Type I error rates in such scenarios [172]. In this study,we also observed that adjusting decision criteria to strictly maintain theType I error rate negates the power gains achieved by using informativepriors [77, 105].7.1.3 Prior Correction Method for CombiningObservational Data from the Same UnderlyingPopulationWhen combining observational data from the same underlying population toestimate the population average treatment effect, conventional approachesoften address unmeasured confounders by treating them as missing data andapplying Bayesian Markov Chain Monte Carlo (MCMC) methods. Whileeffective, these methods can be computationally intensive.In Chapter 4, we introduced an “Honest Precision posterior” that com-bines information from both old and new data by weighting the contribu-tions of the “bias-corrected prior” and estimators derived from new data.The weights are designed to optimize the balance between the two datasources, minimizing the variance of the weighted estimator while avoidingoverreliance on either source. Our results show that this approach achievescomparable statistical performance to the MCMC method across variousproportions of old data, with the added advantage of significantly greatercomputational efficiency, particularly for large datasets. By efficiently in-tegrating historical and real-time data, the “Honest Precision posterior”enables continuous evidence generation and refinement. Its computationalefficiency makes it highly practical for LHS environments, where large-scaledatasets are constantly evolving, and timely decision-making is crucial.7.1.4 Combining Observational and Trial Data for CATEEstimation and Impact on Precision MedicineWhen data originates from different populations, particularly when combin-ing Randomized Controlled Trials (RCT) and observational (OBS) datasets,estimating the Conditional Average Treatment Effect (CATE) becomes cru-118cial for precision medicine within LHS. In Chapter 5, we conducted a scopingreview synthesizing the currently available methodologies of combining OBSdata and RCT data with the aim of estimating CATE, by searching throughGoogle Scholar from inception to June 4th, 2023. We identified in total 18relevant papers and most of the papers (14, 78%) have been published since2022, while almost half (8, 44%) of the works are still in the preprint stage.The review was structured based on the assumptions each method re-quired, especially whether assuming there is unmeasured confounding in theobservational study. The review revealed that existing methods often rely onstrong assumptions, and a comprehensive comparison of their performanceunder varying scenarios was notably absent.To address this gap, Chapter 6 compared three typical methods for es-timating CATE by combining OBS data and RCT data:1. Elastic Combination Approach by Yang et al. [52]: this methodestimates CATE by combining separately and individually estimatedCATEs from OBS studies and RCT.2. Experimental Grounding Approach by Kallus et al. [48]: thisapproach uses RCT data to correct or de-bias the effects of unmeasuredconfounding in OBS datasets when estimating CATE.3. Integrative R learner Approach by Wu and Yang [51]: similarto the experimental grounding approach, this method leverages RCTdata to adjust for unmeasured confounding in OBS datasets but usesan integrative R-learning framework to estimate CATE.Using average RMSE as the primary metric, we found that Yang’s Elas-tic Combination Approach performs best under moderate unmeasured con-founding when covariates overlap fully between RCT and OBS data. Itsperformance benefits from correct model specification, particularly withlinear CATE functions, but it struggles with uncertainty under moderateconfounding and relies heavily on a strict combination threshold. Wu andYang’s Integrative R Learner Approach excels in cases with no or severe un-measured confounding, offering flexibility for complex functional forms like119quadratic CATE functions. However, its performance varies with OBS sam-ple size—improving with larger samples under no confounding but deterio-rating when confounding effects are strong. Kallus’s Experimental Ground-ing Approach underperforms due to its reliance on an incorrectly specifiedmodel for nonlinear confounding. However, it can perform well in specificpredictor ranges, such as when certain covariates fall within narrow valuebands.All methods assume overlapping covariates between RCT and OBS datasets;when overlap is poor, their performance declines sharply due to the trans-portability assumption. Yang’s method, for example, rejects OBS data 95%of the time, effectively relying only on RCT data in such cases. Moreover,all approaches assume that effect modifiers are fully known and measured,though recent works suggest potential methods for relaxing this assumption.These findings emphasize that the choice of method depends on the degreeof unmeasured confounding, covariate overlap, and functional form of theCATE function.Each method offers unique strengths under specific conditions, high-lighting the need for careful evaluation when combining data in practice.The methodologies compared in this chapter provide practical strategiesfor seamlessly combining RCT and OBS data, enabling more precise andpersonalized treatment effect estimation while supporting the embedding oftrials into LHS through efficient evidence generation, real-time learning, andadaptive decision-making.In summary, the primary contribution of this dissertation lies in advanc-ing methodologies to support the seamless embedding of clinical trials withinLearning Health Systems (LHS). By addressing key challenges in integratingreal-world data and trial evidence, this work provides practical tools and in-sights for building adaptive and data-driven healthcare systems. Specifically,the dissertation develops an R package for simulating Bayesian adaptive en-richment designs, offering a flexible framework to optimize trial efficiencyand decision-making. It further evaluated some key impacts of design choiceson power and sample size in Bayesian group-sequential non-inferiority tri-als, providing insights and guidance on designing Bayesian group-sequential120trials. It also proposes a novel prior correction method to integrate observa-tional datasets, overcoming challenges such as unmeasured confounding, toenable accurate and computationally efficient treatment effect estimation.Additionally, the scoping review and the comparisons of methodologies forestimating CATE provides a comprehensive toolbox for researchers, sup-porting the synthesis of diverse data sources to inform precision medicine.7.2 Suggestions for future workThis dissertation has explored various statistical methodologies for buildingLearning Health Systems. However, further refinements and enhancementsare necessary to improve their practicality and applicability in real-worldsettings.Our proposed adaptive enrichment design, implemented in the developedR package, can be further extended to advance personalized medicine by tai-loring treatments to patients based on their characteristics. For more com-plex subpopulations, additional Bayesian models can be explored. Emergingapproaches include the use of Bayesian hierarchical models [79], random par-titions and semiparametric Bayesian models [80], and non-parametric clusterhierarchical models [81] to identify treatment heterogeneity in subpopula-tions. Further innovations focus on determining personalized treatment in-dices by proposing new criteria to measure individual benefits and losses [82],using variable selection to identify covariate sub-vectors for treatment-sensitivepatients [83], and incorporating latent subgroup membership variables toadaptively collapse homogeneous subgroups or identify heterogeneous sub-groups [84].Also, as observed in Chapters 2 and 3, the early termination featureof adaptive designs introduces a risk of biased estimation. When trials arestopped early due to efficacy thresholds, the opportunity to collect additionaldata for accurate effect size estimation is reduced. Simulations demonstratedthat increasing the number of interim analyses exacerbates this bias. To ad-dress this, adjustments to treatment effect estimates should be pre-planned,as recommended by the FDA [171]. However, current methodologies for im-121plementing such adjustments are still underdeveloped, highlighting an areafor future research to improve the reliability of adaptive enrichment designs.Chapter 4 introduces an alternative method for addressing missing vari-ables, offering a statistically and computationally efficient solution comparedto traditional imputation approaches. Our proposed method is specificallydesigned for Gaussian outcomes, with analytical derivations provided forthe mean and variance of the ”Honest Precision posterior.” Extending thismethod to binary or count outcomes will be more useful for real-world sce-narios in LHS, it however presents challenges due to the lack of a conjugateprior for the regression coefficients, making it impossible to derive exactexpressions for the “bias-corrected prior” and the “Honest Precision pos-terior”. Despite these limitations, it is still feasible to derive closed-formapproximations for the regression coefficients of both new and existing datausing Laplace approximation. By following similar principles and steps,we can then further construct the “bias-corrected prior” and the “HonestPrecision posterior”. This approximation could potentially extend the appli-cability of the method to more complex outcome types, warranting furtherinvestigation.In Chapter 6, we evaluated the three approaches using average RMSE,which measures the mean CATE estimation error across the entire partici-pant population. While this metric provides an overall assessment, it doesnot capture variations across specific predictor value ranges. For instance, ina simulation study evaluating the percentage differences between the threemethods and the true CATE as one covariate varied within a specific range,Kallus’s approach outperformed the other two methods within a particularrange. These variations underscore the importance of further investigationsto determine the most efficient approach for specific scenarios and covariatedistributions. Furthermore, all three methods assume that all effect modi-fiers are known and measured. However, emerging research [152] has begunto address this limitation by developing techniques to detect potential un-measured effect modifiers, offering a promising pathway for relaxing thisassumption and broadening the applicability of these approaches.1227.3 Knowledge TranslationChapter 2 has been submitted for peer review, and the BayesAET package,developed as part of this work, is now available as open-source software in R.The package is hosted on GitHub, enabling trialists to plan and implementvarious Bayesian adaptive enrichment trials. The methodological advance-ments from Chapters 3, 4, and 5 will be submitted soon for peer-reviewedpublication.The methodologies developed in Chapter 3 for Bayesian group-sequentialtrials will inform the design of future trials with similar objectives includ-ing but not limited to rTMS trials. The data combination approach usingcorrected priors from Chapter 4 can be applied as more real-world rTMSdata become available. Additionally, once ongoing trials like PORT andLeRNIT are completed, the methods evaluated in Chapter 5 will be appliedto combine trial data with observational rTMS datasets.123Bibliography[1] L. Olsen, D. Aisner, and J.M. McGinnis. The learning healthcaresystem: workshop summary. The National Academies Press, 2007.[2] D. Gracia. Institute of medicine (IOM). the learning healthcare sys-tem: Workshop summary. washington, dc: The national academiespress. Cr´ıtica, 39:89–91, 2013.[3] M. Menear, M.-A. Blanchette, O. Demers-Payette, and D. Roy. Aframework for value-creating learning health systems. Health ResearchPolicy and Systems, 17(1):1–13, 2019.[4] L. Tuzzio, L.R. Hanson, D.B. Reuben, R.R. Baier, J.H. Gurwitz, E.A.Bayliss, J. Williamson, J.R. Fraser, S.J. Sherman, and E.B. Larson.Transforming dementia care through pragmatic clinical trials embed-ded in learning healthcare systems. Journal of the American GeriatricsSociety, 68:S43–S48, 2020.[5] C. Otte, S.M. Gold, B.W. Penninx, C.M. Pariante, A. Etkin, M. Fava,D.C. Mohr, and A.F. Schatzberg. Major depressive disorder. NatureReviews Disease Primers, 2:1–20, 2016.[6] S.L. James, D. Abate, K.H. Abate, S.M. Abay, C. Abbafati, N. Abbasi,H. Abbastabar, F. Abd-Allah, J. Abdela, and A. Abdelalim. Global,regional, and national incidence, prevalence, and years lived with dis-ability for 354 diseases and injuries for 195 countries and territories,1990–2017: a systematic analysis for the global burden of disease study2017. The Lancet, 392(10159):1789–1858, 2018.124[7] D. Souery, P. Oswald, I. Massat, U. Bailer, J. Bollen, K. Demyttenaere,S. Kasper, Y. Lecrubier, S. Montgomery, and A. Serretti. Clinical fac-tors associated with treatment resistance in major depressive disorder:results from a european multicenter study. Journal of Clinical Psychi-atry, 68(7):1062–1070, 2007.[8] C.B. Nemeroff. Prevalence and management of treatment-resistantdepression. Journal of Clinical Psychiatry, 68(8):17, 2007.[9] T.S. Kaster, J. Downar, F. Vila-Rodriguez, K.E. Thorpe, K. Feffer,Y. Noda, P. Giacobbe, Y. Knyahnytska, S.H. Kennedy, and R.W.Lam. Trajectories of response to dorsolateral prefrontal rtms in ma-jor depression: a THREE-D study. American Journal of Psychiatry,176(5):367–375, 2019.[10] D.M. Blumberger, F. Vila-Rodriguez, K.E. Thorpe, K. Feffer,Y. Noda, P. Giacobbe, Y. Knyahnytska, S.H. Kennedy, R.W. Lam,and Z.J. Daskalakis. Effectiveness of theta burst versus high-frequencyrepetitive transcranial magnetic stimulation in patients with depres-sion (THREE-D): a randomised non-inferiority trial. The Lancet,391(10131):1683–1692, 2018.[11] E. Hariton and J.J. Locascio. Randomised controlled trials—the goldstandard for effectiveness research. BJOG: An International Journalof Obstetrics and Gynaecology, 125(13):1716, 2018.[12] P.B. Burns, R.J. Rohrich, and K.C. Chung. The levels of evidenceand their role in evidence-based medicine. Plastic and ReconstructiveSurgery, 128(1):305, 2011.[13] S.-C. Chow, M. Chang, and A. Pong. Statistical consideration of adap-tive methods in clinical development. Journal of BiopharmaceuticalStatistics, 15(4):575–591, 2005.[14] J.J. Park, K. Thorlund, and E.J. Mills. Critical concepts in adaptiveclinical trials. Clinical Epidemiology, pages 343–351, 2018.125[15] S.-C. Chow and M. Chang. Adaptive design methods in clinical tri-als—a review. Orphanet Journal of Rare Diseases, 3(1):11, 2008.[16] M. Rosenblum and D.F. Hanley. Adaptive enrichment designs forstroke clinical trials. Stroke, 48(7):2021–2025, 2017.[17] S.-C. Wang, H.M.J. Hung, and R.T. O’Neill. Adaptive patient enrich-ment designs in therapeutic trials. Biometrical Journal: Journal ofMathematical Methods in Biosciences, 51(2):358–374, 2009.[18] P.F. Thall. Adaptive enrichment designs in clinical trials. AnnualReview of Statistics and Its Application, 8:393–411, 2021.[19] T. Burnett and C. Jennison. Adaptive enrichment trials: What arethe benefits? Statistics in Medicine, 40(3):690–711, 2021.[20] D.A. Berry. Bayesian clinical trials. Nature Reviews Drug Discovery,5(1):27–36, 2006.[21] A. Gelman, J.B. Carlin, H.S. Stern, and D.B. Rubin. Bayesian DataAnalysis. Chapman and Hall/CRC, 1995.[22] Y. Cheng and Y. Shen. Bayesian adaptive designs for clinical trials.Biometrika, 92(3):633–646, 2005.[23] K. Thorlund, S. Golchi, J. Haggstrom, and E. Mills. Highly efficientclinical trials simulator (HECT): Software application for planningand simulating platform adaptive trials. Gates Open Research, 3:1–10, 2019.[24] J.X. Song. A two-stage patient enrichment adaptive design in phaseii oncology trials. Contemporary clinical trials, 37(1):148–154, 2014.[25] X. Chen, J. Zhang, L. Jiang, and F. Yan. Ibis: identify biomarker-based subgroups with a bayesian enrichment design for targeted com-bination therapy. BMC Medical Research Methodology, 23(1):66, 2023.126[26] Y. Park, S. Liu, P.F. Thall, and Y. Yuan. Bayesian group sequen-tial enrichment designs based on adaptive regression of response andsurvival time on baseline biomarkers. Biometrics, 78(1):60–71, 2022.[27] T. Friede, N. Stallard, and N. Parsons. Adaptive seamless clinical trialsusing early outcomes for treatment or subgroup selection: Methods,simulation model and their implementation in r. Biometrical Journal,62(5):1264–1283, 2020.[28] A. Sarraj, A. E. Hassan, M. Abraham, M. Ribo, S. Blackburn,M. Chen, M. S. Hussain, V. M. Pereira, S. Ortega-Gutierrez, C. Sit-ton, et al. A randomized controlled trial to optimize patient’s selectionfor endovascular treatment in acute ischemic stroke (select2): studyprotocol. International Journal of Stroke, 17(6):689–693, 2022.[29] N. Egami and E. Hartman. Covariate selection for generalizing exper-imental results: application to a large-scale development program inuganda. Journal of the Royal Statistical Society: Series A (Statisticsin Society), 184(4):1524–1548, 2021.[30] T.Q. Nguyen, C. Ebnesajjad, S.R. Cole, and E.A. Stuart. Sensitiv-ity analysis for an unobserved moderator in rct-to-target-populationgeneralization of treatment effects. The Annals of Applied Statistics,11:225–247, 2017.[31] B. Colnet, J. Josse, G. Varoquaux, and E. Scornet. Causal effect on atarget population: a sensitivity analysis to handle missing covariates.Journal of Causal Inference, 10(1):372–414, 2022.[32] M. Huang. Sensitivity analysis in the generalization of experimentalresults. arXiv preprint arXiv:2202.03408, 2022.[33] H. Toutenburg. Rubin, DB: Multiple Imputation for Nonresponse inSurveys. Wiley, New York, 1990.[34] J.A.C. Sterne, I.R. White, J.B. Carlin, M. Spratt, P. Royston, M.G.Kenward, A.M. Wood, and J.R. Carpenter. Multiple imputation for127missing data in epidemiological and clinical research: potential andpitfalls. BMJ, 338, 2009.[35] D.B. Rubin. Inference and missing data. Biometrika, 63(3):581–592,1976.[36] I.J. Dahabreh and D.M. Kent. Can the learning health care systembe educated with observational data? JAMA, 312(2):129–130, 2014.[37] R.L. Kravitz, N. Duan, and J. Braslow. Evidence-based medicine,heterogeneity of treatment effects, and the trouble with averages. TheMilbank Quarterly, 82(4):661–687, 2004.[38] J. Concato, N. Shah, and R.I. Horwitz. Randomized, controlled tri-als, observational studies, and the hierarchy of research designs. NewEngland Journal of Medicine, 342(25):1887–1892, 2000.[39] L.W. Green and R.E. Glasgow. Evaluating the relevance, general-ization, and applicability of research: issues in external validationand translation methodology. Evaluation & the Health Professions,29(1):126–153, 2006.[40] T.R. Frieden. Evidence for health decision making—beyond random-ized, controlled trials. New England Journal of Medicine, 377(5):465–475, 2017.[41] A. Silverio, P. Cavallo, R. De Rosa, and G. Galasso. Big health dataand cardiovascular diseases: a challenge for research, an opportunityfor clinical care. Frontiers in Medicine, 6:36, 2019.[42] T. Li, J.P.T. Higgins, and J.J. Deeks. Collecting Data, pages 109–141.2019.[43] B. Colnet, I. Mayer, G. Chen, A. Dieng, R. Li, G. Varoquaux, J.-P.Vert, J. Josse, and S. Yang. Causal inference methods for combiningrandomized trials and observational studies: a review. arXiv preprintarXiv:2011.08047, 2020.128[44] C.L. Brantner, T.-H. Chang, T.Q. Nguyen, H. Hong, S. Di Laura,and E.A. Stuart. Methods for integrating trials and non-experimentaldata to examine treatment effect heterogeneity. arXiv preprintarXiv:2302.13428, 2023.[45] G.W. Imbens. Sensitivity to exogeneity assumptions in program eval-uation. American Economic Review, 93(2):126–132, 2003.[46] P.R. Rosenbaum. Overt Bias in Observational Studies. Springer, 2002.[47] S. Athey, R. Chetty, G. Imbens, and H. Kang. Estimating treatmenteffects using multiple surrogates: The role of the surrogate score andthe surrogate index. arXiv preprint arXiv:1603.09326, 2016.[48] N. Kallus, A.M. Puli, and U. Shalit. Removing hidden confoundingby experimental grounding. In Advances in Neural Information Pro-cessing Systems, volume 31, 2018.[49] T. Hatt, D. Tschernutter, and S. Feuerriegel. Generalizing off-policylearning under sample selection bias. In Uncertainty in Artificial In-telligence: PMLR, pages 769–779, 2022.[50] S. Yang, D. Zeng, and X. Wang. Improved inference for heterogeneoustreatment effects using real-world data subject to hidden confounding.2020.[51] L. Wu and S. Yang. Integrative r learner of heterogeneous treatmenteffects combining experimental and observational studies. In Confer-ence on Causal Learning and Reasoning: PMLR, pages 904–926, 2022.[52] S. Yang, C. Gao, D. Zeng, and X. Wang. Elastic integrative analysis ofrandomised trial and real-world data for treatment heterogeneity esti-mation. Journal of the Royal Statistical Society: Series B (StatisticalMethodology), qkad017, 2023.[53] C. Mehta, H. Scha¨fer, H. Daniel, and S. Irle. Biomarker driven pop-ulation enrichment for adaptive oncology trials with time to eventendpoints. Statistics in Medicine, 33(26):4515–4531, 2014.129[54] L.A. Renfro, M-W. An, and S.J. Mandrekar. Precision oncology: anew era of cancer clinical trials. Cancer Letters, 387:121–126, 2017.[55] N. Simon and R. Simon. Adaptive enrichment designs for clinicaltrials. Biostatistics, 14(4):613–625, 2013.[56] U.S. Food Administration and Drug. Enrichment Strategies for Clin-ical Trials to Support Approval of Human Drugs and Biological Prod-ucts. 2019.[57] Y. Ouyang, J. J. Eng, D. Zhan, and H. Wong. Priors from envisionedposterior judgments: A novel elicitation approach with application tobayesian clinical trials. arXiv preprint, arXiv:2409.05271, 2024.[58] P. Mikkola, O. A. Martin, S. Chandramouli, M. Hartmann,O. Abril Pla, O. Thomas, H. Pesonen, J. Corander, A. Vehtari, andS. Kaski. Prior knowledge elicitation: The past, present, and future.Bayesian Analysis, 19(4):1129–1161, 2024.[59] F. Bockting, S. T. Radev, and P.-C. Bu¨rkner. Expert-elicitationmethod for non-parametric joint priors using normalizing flows. arXivpreprint, arXiv:2411.15826, 2024.[60] N. Simon and R. Simon. Using bayesian modeling in frequentist adap-tive enrichment designs. Biostatistics, 19(1):27–41, 2018.[61] Y. Xu, F. Constantine, Y. Yuan, and Y.L. Pritchett. Asied: a bayesianadaptive subgroup-identification enrichment design. Journal of Bio-pharmaceutical Statistics, 30(4):623–638, 2020.[62] U.S. Food and Drug Administration. Adaptive design clinical trialsfor drugs and biologics: Guidance for industry, 2019. Accessed: Feb.2025.[63] U.S. Food and Drug Administration. Guidance for the use of bayesianstatistics in medical device clinical trials, 2010. Accessed: Feb. 2025.130[64] M.J. Grayling and G.M. Wheeler. A review of available software foradaptive clinical trial design. Clinical Trials, 17(3):323–331, 2020.[65] E.L. Meyer, P. Mesenbrink, T. Mielke, T. Parke, D. Evans, andF. Ko¨nig. Systematic review of available software for multi-arm multi-stage and platform clinical trial design. Trials, 22(1):1–14, 2021.[66] FACTS software page. Available at: https://www.berryconsultants.com/software/facts/.[67] R. Lin, Z. Yang, Y. Yuan, and G. Yin. Sample size re-estimationin adaptive enrichment design. Contemporary Clinical Trials,100:106216, 2021.[68] B.P. Magnusson and B.W. Turnbull. Group sequential enrichmentdesign incorporating subgroup selection. Statistics in Medicine,32(16):2695–2714, 2013.[69] N. Parsons, T. Friede, S. Todd, C. Marquez-Vega, J. Chataway,R. Nicholas, and N. Stallard. An r package for implementing sim-ulations for seamless phase ii/iii clinical trials using early outcomesfor treatment selection. Computational Statistics & Data Analysis,56(5):1150–1160, 2012.[70] T.L. Lai, P.W. Lavori, and O.Y-W. Liao. Adaptive choice of patientsubgroup for comparing two treatments. Contemporary Clinical Trials,39(2):191–200, 2014.[71] L. Wei and S. Durham. The randomized play-the-winner rule inmedical trials. Journal of the American Statistical Association,73(364):840–843, 1978.[72] O. Sverdlov. Modern adaptive randomized clinical trials: statisticaland practical aspects, volume 81. CRC Press, 2015.[73] D.M. Blumberger, F. Vila-Rodriguez, K.E. Thorpe, K. Feffer,Y. Noda, P. Giacobbe, Y. Knyahnytska, S.H. Kennedy, R.W. Lam,131and Z.J. Daskalakis. Effectiveness of theta burst versus high-frequencyrepetitive transcranial magnetic stimulation in patients with depres-sion (THREE-D): a randomised non-inferiority trial. The Lancet,391(10131):1683–1692, 2018.[74] M. Hamilton. The assessment of anxiety states by rating. BritishJournal of Medical Psychology, 32(1):50–55, 1959.[75] H. Wong. Minimum important difference is minimally important insample size calculations. Trials, 24(1):1–4, 2023.[76] J.M. Wason, L. Stecher, and A.P. Mander. Correcting for multiple-testing in multi-arm trials: is it necessary and is it done? Trials,15(1):1–7, 2014.[77] N. Best, M. Ajmi, B. Neuenschwander, G. Saint-Hilary, and S. Wandel.Beyond the classical type i error: Bayesian metrics for bayesian designsusing informative priors. Statistics in Biopharmaceutical Research,pages 1–14, 2024.[78] E.G. Ryan, K. Brock, S. Gates, and D. Slade. Do we need to adjustfor interim analyses in a bayesian adaptive trial design? BMC MedicalResearch Methodology, 20(1):1–9, 2020.[79] S.M. Berry, K.R. Broglio, S. Groshen, and D.A. Berry. Bayesian hier-archical modeling of patient subpopulations: efficient designs of phaseii oncology clinical trials. Clinical Trials, 10(5):720–734, 2013.[80] W. Guo, Y. Ji, and D.V. Catenacci. A subgroup cluster-based bayesianadaptive design for precision medicine. Biometrics, 73(2):367–377,2017.[81] C. Zhang, M.S. Mayo, J.A. Wick, and B.J. Gajewski. Designing andanalyzing clinical trials for personalized medicine via bayesian models.Pharm Stat, 20(3):573–596, 2021.132[82] F. Xia, S.L. George, J. Ning, L. Li, and X. Huang. A signature enrich-ment design with bayesian adaptive randomization. Journal of AppliedStatistics, 48(6):1091–1110, 2021.[83] Y. Park, S. Liu, P.F. Thall, and Y. Yuan. Bayesian group sequen-tial enrichment designs based on adaptive regression of response andsurvival time on baseline biomarkers. Biometrics, 2021.[84] R. Lin, P.F. Thall, and Y. Yuan. Bags: A bayesian adaptive groupsequential trial design with subgroup-specific survival comparisons.Journal of the American Statistical Association, 116(533):322–334,2021.[85] L. S. Freedman. Tables of the number of patients required in clinicaltrials using the logrank test. Statistics in Medicine, 1(2):121–129,1982.[86] J. Concato, N. Shah, and R. I. Horwitz. Randomized, controlled tri-als, observational studies, and the hierarchy of research designs. NewEngland Journal of Medicine, 342(25):1887–1892, 2000.[87] L. M. Friedman, C. Furberg, D. L. DeMets, D. M. Reboussin, andC. B. Granger. Fundamentals of Clinical Trials, volume 4. Springer,2010.[88] S. J. Pocock. Clinical Trials: A Practical Approach. John Wiley &Sons, 2013.[89] K. Thorlund, J. Haggstrom, J. J. Park, and E. J. Mills. Key designconsiderations for adaptive clinical trials: a primer for clinicians. BMJ,360, 2018.[90] P. Shah, D. Slebos, P. Cardoso, E. Cetti, K. Voelker, B. Levine,M. Russell, J. Goldin, M. Brown, and J. Cooper. Bronchoscopiclung-volume reduction with exhale airway stents for emphysema (easetrial): randomised, sham-controlled, multicentre trial. The Lancet,378(9795):997–1005, 2011.133[91] G. Middleton, L. Crack, S. Popat, C. Swanton, S. Hollingsworth,R. Buller, I. Walker, T. Carr, D. Wherton, and L. Billingham. Thenational lung matrix trial: translating the biology of stratification inadvanced non-small-cell lung cancer. Annals of Oncology, 26(12):2464–2469, 2015.[92] J. O. Jansen, P. Pallmann, G. MacLennan, M. K. Campbell, and U-Rt investigators. Bayesian clinical trial designs: Another option fortrauma trials? Journal of Trauma and Acute Care Surgery, 83(4):736–741, 2017.[93] E. G. Ryan, D. L. Couturier, and S. Heritier. Bayesian adaptive clinicaltrial designs for respiratory medicine. Respirology, 27(10):834–843,2022.[94] E. G. Ryan, S. E. Lamb, E. Williamson, and S. Gates. Bayesian adap-tive designs for multi-arm trials: an orthopaedic case study. Trials,21:1–16, 2020.[95] S. M. Berry, B. P. Carlin, J. J. Lee, and P. Muller. Bayesian AdaptiveMethods for Clinical Trials. CRC Press, 2010.[96] G. Tomlinson, A. Al-Khafaji, S. A. Conrad, F. N. Factora, D. M.Foster, C. Galphin, K. J. Gunnerson, S. Khan, R. Kohli-Seth, andP. McCarthy. Bayesian methods: a potential path forward for sepsistrials. Critical Care, 27(1):432, 2023.[97] L. K. Vaughan, J. Divers, M. A. Padilla, D. T. Redden, H. K. Tiwari,D. Pomp, and D. B. Allison. The use of plasmodes as a supplementto simulations: a simple example evaluating individual admixture es-timation methodologies. Computational Statistics & Data Analysis,53(5):1755–1766, 2009.[98] J. M. Franklin, S. Schneeweiss, J. M. Polinski, and J. A. Rassen. Plas-mode simulation for the evaluation of pharmacoepidemiologic meth-ods in complex healthcare databases. Computational Statistics & DataAnalysis, 72:219–226, 2014.134[99] Y. Yuan, X. Huang, and S. Liu. A bayesian response-adaptivecovariate-balanced randomization design with application to aleukemia clinical trial. Statistics in Medicine, 30(11):1218–1229, 2011.[100] D. J. Spiegelhalter, L. S. Freedman, and M. K. Parmar. Bayesian ap-proaches to randomized trials. Journal of the Royal Statistical Society:Series A (Statistics in Society), 157(3):357–387, 1994.[101] E. G. Ryan, K. Brock, S. Gates, and D. Slade. Do we need to adjustfor interim analyses in a bayesian adaptive trial design? BMC MedicalResearch Methodology, 20:1–9, 2020.[102] D. A. Berry. Bayesian statistics and the efficiency and ethics of clinicaltrials. Nature Reviews Drug Discovery, 2004.[103] F. Hu and W. F. Rosenberger. The Theory of Response-AdaptiveRandomization in Clinical Trials. John Wiley & Sons, 2006.[104] P. Thall, P. Fox, and J. Wathen. Statistical controversies in clinicalresearch: Scientific and ethical problems with adaptive randomizationin comparative clinical trials. Annals of Oncology, 26(8):1621–1628,2015.[105] A. Kopp-Schneider, S. Calderazzo, and M. Wiesenfarth. Power gainsby using external information in clinical trials are typically not pos-sible when requiring strict type i error control. Biometrical Journal,62(2):361–374, 2020.[106] J.W. Graham. Missing data analysis: Making it work in the real world.Annual Review of Psychology, 60:549–576, 2009.[107] A.N. Baraldi and C.K. Enders. An introduction to modern missingdata analyses. Journal of School Psychology, 48(1):5–37, 2010.[108] J.R. Carpenter, J.H. Roger, and M.G. Kenward. Analysis of longitudi-nal trials with protocol deviation: A framework for relevant, accessibleassumptions, and inference via multiple imputation. Journal of Bio-pharmaceutical Statistics, 23(6):1352–1371, 2013.135[109] D.B. Rubin. Inference and missing data. Biometrika, 63(3):581–592,1976.[110] M. Takahashi. Statistical inference in missing data by mcmc andnon-mcmc multiple imputation algorithms: Assessing the effects ofbetween-imputation iterations. Data Science Journal, 16:37–37, 2017.[111] T.P. Morris, I.R. White, and M.J. Crowther. Using simulation studiesto evaluate statistical methods. Statistics in Medicine, 38(11):2074–2102, 2019.[112] I.R. White, T.M. Pham, M. Quartagno, and T.P. Morris. How tocheck a simulation study. International Journal of Epidemiology,53(1):dyad134, 2024.[113] M.A. Tanner and W.H. Wong. The calculation of posterior distri-butions by data augmentation. Journal of the American StatisticalAssociation, 82(398):528–540, 1987.[114] S. Athey and G. Imbens. Recursive partitioning for heterogeneouscausal effects. Proceedings of the National Academy of Sciences,113(27):7353–7360, 2016.[115] S.R. Ku¨nzel, J.S. Sekhon, P.J. Bickel, and B. Yu. Metalearners forestimating heterogeneous treatment effects using machine learning.Proceedings of the National Academy of Sciences, 116(10):4156–4165,2019.[116] I. Bica, A.M. Alaa, C. Lambert, and M. Van Der Schaar. From real-world patient data to individualized treatment effects using machinelearning: current and future methods to address underlying challenges.Clinical Pharmacology & Therapeutics, 109(1):87–100, 2021.[117] B.G. Vegetabile. On the distinction between conditional average treat-ment effects (cate) and individual treatment effects (ite) under ignor-ability assumptions. arXiv preprint arXiv:210804939, 2021.136[118] T. Hatt, D. Tschernutter, and S. Feuerreigel. Generalizing off-policylearning under sample selection bias. In Uncertainty in Artificial In-telligence, pages 769–779. PMLR, 2022.[119] A. Coppock, T.J. Leeper, and K.J. Mullinix. Generalizability of het-erogeneous treatment effect estimates across samples. Proceedings ofthe National Academy of Sciences, 115(49):12441–12446, 2018.[120] I. Lipkovich, D. Svensson, B. Ratitch, and A. Dmitrienko. Modernapproaches for evaluating treatment effect heterogeneity from clinicaltrials and observational data. Statistics in Medicine, 43(22):4388–4436, 2024.[121] J. Pearl. Causality. Cambridge University Press, Cambridge, 2009.[122] P.R. Hahn, J.S. Murray, and C.M. Carvalho. Bayesian regression treemodels for causal inference: Regularization, confounding, and hetero-geneous effects (with discussion). Bayesian Analysis, 15(3):965–1056,2020.[123] A. Caron, G. Baio, and I. Manolopoulou. Shrinkage bayesian causalforests for heterogeneous treatment effects estimation. Journal ofComputational and Graphical Statistics, 31(4):1202–1214, 2022.[124] V. Dorie, G. Perrett, J.L. Hill, and B. Goodrich. Stan and bart forcausal inference: Estimating heterogeneous treatment effects usingthe power of stan and the flexibility of machine learning. Entropy,24(12):1782, 2022.[125] E. Sverdrup and Y. Cui. Proximal causal learning of conditional av-erage treatment effects. ICML 2023, 2023.[126] A. Konstantinov, S. Kirpichenko, and L. Utkin. Heterogeneous treat-ment effect with trained kernels of the nadaraya–watson regression.Algorithms, 16(5):226, 2023.[127] R. Singh. Kernel methods for unobserved confounding: Negative con-trols, proxies, and instruments. arXiv preprint arXiv:201210315, 2020.137[128] S. Chen, Y. Wang, Z. Wang, and Z. Yang. A unified framework of pol-icy learning for contextual bandit with confounding bias and missingobservations. arXiv preprint arXiv:230311187, 2023.[129] X. Shi, Z. Pan, and W. Miao. Data integration in causal inference. Wi-ley Interdisciplinary Reviews: Computational Statistics, 15(1):e1581,2023.[130] E. Bareinboim and J. Pearl. Causal inference and the data-fusion prob-lem. Proceedings of the National Academy of Sciences, 113(27):7345–7352, 2016.[131] A vision for use of real-world evidence in EU medicines regu-lation, 2018. [https://www.ema.europa.eu/en/news/vision-use-real-world-evidence-eu-medicines-regulation].[132] Framework for FDA’s real-world evidence program, 2018.[https://www.fda.gov/media/120060/download].[133] S. Wedam, L. Fashoyin-Aje, E. Bloomquist, S. Tang, R. Sridhara,K.B. Goldberg, M.R. Theoret, L. Amiri-Kordestani, R. Pazdur, andJ.A. Beaver. Fda approval summary: palbociclib for male patientswith metastatic breast cancer. Clinical Cancer Research, 26(6):1208–1212, 2020.[134] FDA approves new treatment for certain digestive tract cancers, 2018.[https://www.fda.gov/media/120060/download].[135] Qualification opinion for prognostic covariate adjustment (procova™),2020. [https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/qualification-opinion-prognostic-covariate-adjustment-procovatm-en.pdf].[136] S. Yang and X. Wang. Rwd-integrated randomized clinical trial anal-ysis. Biopharmaceutical report, 29(2):15, 2022.138[137] C.L. Brantner, T.-H. Chang, T.Q. Nguyen, H. Hong, L. Di Ste-fano, and E.A. Stuart. Methods for integrating trials and non-experimental data to examine treatment effect heterogeneity. arXivpreprint arXiv:230213428, 2023.[138] H. Arksey and L. O’Malley. Scoping studies: towards a methodolog-ical framework. International journal of social research methodology,8(1):19–32, 2005.[139] I.J. Dahabreh, S.E. Robertson, E.J. Tchetgen, E.A. Stuart, and M.A.Herna´n. Generalizing causal inferences from individuals in randomizedtrials to all trial-eligible individuals. Biometrics, 75(2):685–694, 2019.[140] D. Cheng and T. Cai. Adaptive combination of randomized and ob-servational data. arXiv preprint arXiv:211115012, 2021.[141] T. Hatt, J. Berrevoets, A. Curth, S. Feuerriegel, and M. van derSchaar. Combining observational and randomized data for estimat-ing heterogeneous treatment effects. arXiv preprint arXiv:220212891,2022.[142] E.T. Rosenman, A.B. Owen, M. Baiocchi, and H.R. Banack. Propen-sity score methods for merging observational and experimentaldatasets. Statistics in Medicine, 41(1):65–86, 2022.[143] E.T. Rosenman, G. Basse, A.B. Owen, and M. Baiocchi. Combiningobservational and experimental datasets using shrinkage estimators.Biometrics, 2020.[144] S. Yang, D. Zeng, and X. Wang. Improved inference for heterogeneoustreatment effects using real-world data subject to hidden confounding.arXiv preprint arXiv:200712922, 2020.[145] Y. Cao and J. Yu. Adjusting for unmeasured confounding in survivalcausal effect using validation data. Computational Statistics & DataAnalysis, 180:107660, 2023.139[146] P. Zhao, J. Josse, and S. Yang. Efficient and robust transfer learning ofoptimal individualized treatment regimes with right-censored survivaldata. arXiv preprint arXiv:230105491, 2023.[147] X. Lin and R.J. Evans. Many data: Combine experimental andobservational data through a power likelihood. arXiv preprintarXiv:230402339, 2023.[148] S.E. Robertson, J.A. Steingrimsson, and I.J. Dahabreh. Regression-based estimation of heterogeneous treatment effects when extendinginferences from a randomized trial to a target population. EuropeanJournal of Epidemiology, 38(2):123–133, 2023.[149] K. Chalkou, T. Hamza, P. Benkert, et al. Combining randomized andnon-randomized data to predict heterogeneous effects of competingtreatments. arXiv preprint arXiv:230214766, 2023.[150] X. Li, Y. Li, Q. Cui, L. Li, and J. Zhou. Robust direct learning forcausal data fusion. In Asian Conference on Machine Learning, pages611–626. PMLR, 2022.[151] L. Wu and S. Yang. Transfer learning of individualized treatment rulesfrom experimental to real-world data. Journal of Computational andGraphical Statistics, 2022.[152] R. Ladhania, A. Haviland, N. Sood, E. Kennedy, and A. Mehrotra.Learning and testing sub-groups with heterogeneous treatment effects:A sequence of two studies. arXiv preprint arXiv:200611628, 2020.[153] C. Ferna´ndez-Lor´ıa and F. Provost. Combining observational and ex-perimental data to improve large-scale decision-making. ICIS 2020Proceedings, 7, 2020.[154] A. Peysakhovich and A. Lada. Combining observational and experi-mental data to find heterogeneous treatment effects. arXiv preprintarXiv:161102385, 2016.140[155] E.H. Kennedy. Towards optimal doubly robust estimation of hetero-geneous causal effects. arXiv preprint arXiv:200414497, 2020.[156] C. Stein. Inadmissibility of the usual estimator for the mean of a mul-tivariate normal distribution. Proceedings of the Third Berkeley Sym-posium on Mathematical Statistics and Probability, Volume 1: Contri-butions to the Theory of Statistics, pages 197–207, 1956.[157] E.J. Green and W.E. Strawderman. A james-stein type estimator forcombining unbiased and possibly biased estimators. Journal of theAmerican Statistical Association, 86(416):1001–1006, 1991.[158] D. Jacob. Cate meets ml: Conditional average treatment effect andmachine learning. Digital Finance, 3(2):99–148, 2021.[159] J. Hill, A. Linero, and J. Murray. Bayesian additive regression trees:A review and look forward. Annual Review of Statistics and Its Ap-plication, 7:251–278, 2020.[160] E.J. Green, W.E. Strawderman, R.L. Amateis, and G.A. Reams. Im-proved estimation for multiple means with heterogeneous variances.Forest Science, 51(1):1–6, 2005.[161] M. Ilse, P. Forre´, M. Welling, and J.M. Mooij. Combining interven-tional and observational data using causal reductions. arXiv preprintarXiv:210304786, 2021.[162] I. Degtiar and S. Rose. A review of generalizability and transporta-bility. Annual Review of Statistics and Its Application, 10:501–524,2023.[163] T. Wendling, K. Jung, A. Callahan, A. Schuler, N.H. Shah, and B. Gal-lego. Comparing methods for estimation of heterogeneous treatmenteffects using observational data from health care databases. Statisticsin Medicine, 37(23):3309–3324, 2018.141[164] A.N. Varga, A.E. Guevara Morel, J. Lokkerbol, J.M. van Dongen,M.W. van Tulder, and J.E. Bosmans. Dealing with confounding inobservational studies: A scoping review of methods evaluated in sim-ulation studies with single-point exposure. Statistics in Medicine,42(4):487–516, 2023.[165] P. Bickel, C. Klaassen, Y. Ritov, and J. Wellner. Efficient and adaptiveinference in semiparametric models. Johns Hopkins University Press,Baltimore, 1993.[166] J.M. Robins. Correcting for non-compliance in randomized trials usingstructural nested mean models. Communications in Statistics-Theoryand Methods, 23(8):2379–2412, 1994.[167] X. Nie and S. Wager. Quasi-oracle estimation of heterogeneous treat-ment effects. Biometrika, 108(2):299–319, 2021.[168] I. Shpitser and T.J. VanderWeele. A complete graphical criterion forthe adjustment formula in mediation analysis. The International Jour-nal of Biostatistics, 7(1):0000102202155746791297, 2011.[169] J.M. Robins, M.A. Hernan, and B. Brumback. Marginal structuralmodels and causal inference in epidemiology. Lww, 11:550–560, 2000.[170] J.M. McGinnis, L. Stuckhardt, R. Saunders, and M. Smith. Best careat lower cost: the path to continuously learning health care in america.2013.[171] Food and Drug Administration (FDA). Adaptive designs for clinicaltrials of drugs and biologics: guidance for industry. Food and DrugAdministration, 2020.[172] E.G. Ryan, K. Brock, S. Gates, and D. Slade. Do we need to adjustfor interim analyses in a bayesian adaptive trial design? BMC MedicalResearch Methodology, 20:1–9, 2020.142Appendix ASupporting information forChapter 2A.1 The input and output descriptions of themajor functionsA.1.1 Example run of function BAET.simFigure A.1 shows an example of the function BAET.sim with all input pa-rameters needed to run a Bayesian adaptive enrichment trial with 3 treat-ments and 2 subpopulations, and Figure A.2 and Figure A.3 show thecorresponding BAET.sim output using the input values.Figure A.1: A snapshot of the main function ‘BAET.sim’ and the requireduser inputs143Figure A.2: A snapshot of the corresponding ‘BAET.sim‘ outputs using theinput values - Part 1.144Figure A.3: A snapshot of the corresponding ‘BAET.sim‘ outputs using theinput values - Part 2.A.1.2 Details of function Multi.BAETMulti.BAET is a wrapper function for BAET.sim, designed to simulate asingle trial multiple times to evaluate the trial design’s properties. It sharesmost of its arguments with BAET.sim, with the addition of two specificparameters: n.sim and n.cores. These allow users to specify the number ofsimulations to run and the number of cores to utilize for parallel computing,respectively.145Figure A.4: A snapshot of the wrapper function ‘Multi.BAET’ and therequired user inputsAn example to run the Multi.BAET function is shown in Figure A.4. Theoutputs of the ‘Multi.BAET’ function are stored in a list. Table A.1 belowdemonstrates the details of each output and its corresponding description146Table A.1: Outputs and descriptions for function Multi.BAETMulti.BAET Output Descriptionsest.mean A ns × nt matrix storing the mean of the poste-rior mean, with row number indicating subpopu-lation and column number indicating treatmentarmest.sd A ns × nt matrix storing the mean of the pos-terior SD, with row number indicating subpopu-lation and column number indicating treatmentarmss.sub.dist A ns × n.sim matrix storing the sample sizeconsumed for each subpopulation in each run,with row number indicating subpopulation andcolumn number indicating the specific runss.sub.mean A vector length ns indicating the average samplesize consumed in each subpopulationss.t.dist A n.sim vector storing the overall sample sizeconsumed in each runss.t.mean A number of average overall sample size con-sumedpower.sub A vector length ns indicating the (empirical)power estimated in each subpopulationcomputation.time Time consumed for computationFigure A.5 shows the ‘Multi.BAET’ output using the input values aboveas an example.147Figure A.5: The corresponding outputs with the input values from the wrap-per function ‘Multi.BAET’148A.2 Data Generative ProcessFor individual participant i, we first generate the subpopulation to whichthe participant belongs, and then generate the treatment assignment. Wefinally combine these two pieces of information into one single vector withM ∗G components which identifies both the subpopulation and the assignedtreatment.Specifically, as defined in Section 2.2.1, where we already denote k =(k1, . . . , kG), as recruitment probability, and ρm|g as the conditional proba-bility of being assigned to treatment m for a participant belonging to g-thsubpopulation, we thus denote ρmg as the joint probability of a participantcoming from g-th subpopulation and being assigned to treatment m, i.e.,ρmg = kg × ρm|g. Then we create the following vector xi with a size ofM ∗G, by setting the (M ∗ (g− 1) +m)-th component to one if participanti belongs to the g-th subpopulation and has been assigned to treatment m,and zero, otherwise:xi ∼Multinomial(1,ρ = (ρ11, ρ21, . . . , ρM1, ρ12, ρ22, . . . , ρMG)).Denote the θ as the vector of the true treatment effects with a size ofM ∗ G, with its (M ∗ (g − 1) +m)-th component indicating the treatmenteffect of treatment m in g-th subpopulation, i.e.,θ = [θ11, θ21, . . . , θM1, θ12, θ22, . . . , θMG].Then for a Gaussian outcome, the individual outcome Yi is generatedfrom:Yi ∼ N (θ⊤xi, σ2),where σ2 is the variance.For binary and count outcomes, the input values for θ become pro-portions and rates, respectively. Specifically, for binary outcome Yi, it is149generated from:Yi ∼ Bernoulli(θ⊤xi),for count outcome, the individual outcome Yi is generated from a Poissondistribution:Yi ∼ Poisson(θ⊤xi).The data will be kept generated until one batch sample size thresholdRbg for g = 1, 2, . . . , G is reached.150A.3 Modeling binary and count outcomesthrough multivariate normal joint priorFor a binary outcome, suppose θmg indicates the probability of the eventfor treatment m (m = 1, 2, . . . ,M) in the subpopulation g (g = 1, 2, . . . , G).The individual level response Yi for patient i (i = 1, 2, . . . , N) in subpopu-lation g receiving treatment m is modeled as:Yi ∼ Bernoulli(θmg),Logit(θmg) = βmgwhere βmg is the log-odds of θmg.Denote the β as the vector of the true log-odds with a size of M ∗ G,with its (M ∗ (g− 1)+m)th component indicating the log-odd of treatmentm in g-th subpopulation βmg, i.e.,β = [β11, β21, . . . , βM1, β12, β22, . . . , βMG].Let vector B denote the prior distribution for β. We then consider amultivariate joint prior structure to account for the correlations between theparameters of the log-odds. This is achieved by incorporating this informa-tion into the variance-covariance matrix of the prior:B ∼MVN (µ0,Σ),where µ0 is the prior mean vector and Σ is the prior variance-covariancematrix. We estimate the posterior from the MCMC samples drawn by Gibbssampling through software JAGS. The log-odd will be converted back to theoriginal scale probability for the next steps.We have similar setups for count outcome and the θmg represents therate. The individual level response Yi is modeled as:Yi ∼ Poisson(θmg),151Log(θmg) = βmgwhere βmg is the log of θmg. We can model the βmg similarly as above forthe binary outcome through a multivariate joint prior structure to accountfor the correlations between the parameters of the log-rate.152Appendix BSupporting information forChapter 3B.1 Details of the proposed Bayesian adaptivedesignB.1.1 Bayesian adaptive design features and inputsModel and Prior SpecificationDenote the θm as the true mean responses for treatment m, the individuallevel responses can thus be modeled as:Yi = θm + ϵiwhere ϵi ∼ N (0, σ2).Let the vector µ denote all the parameters in the expression of θm. Wethen consider a joint prior structure to account for the correlations betweenparameters by incorporating this information into the variance-covariancematrix:µ ∼MVN (µ0,Σ)where µ0 is the prior mean vector and Σ is the prior variance-covariancematrix. Even though the posterior has an analytical form, we compute theposterior through Gibbs sampling by JAGS as later we will still need tocompare θm empirically based on the MCMC samples.Specifically, in our simulation based on THREE-D, we have two treat-153ments, m = 2. We thus have the following prior structure:µ0 = [µ1, µ2]⊤ and Σ =[σ11 σ12σ21 σ22]where µ1, µ2 are the prior means for the treatment effect of HFL and TBS,respectively, σ11 and σ22 are the variances of the outcomes of the two treat-ments, respectively, and σ12 and σ21 are the covariances between the twotreatments.Interim analysis and evaluation criteriaAs the trial progresses, different decisions and adaptations may be madebased on the interim data analysis, and the decision rules are based onthe posterior probabilities. Specifically, the posterior probability of non-inferiority, S, is defined as the probability of the treatment m being betterthan the standard treatment c with a difference larger than the minimumimportant difference (MID), after observing the cumulated data:S = P (θm − θc > MID, for all m ̸= c | data).The S can be estimated empirically by:Sˆ =1NN∑i=11(θm,i − θc,i > MID),where θm,i is the i-th MCMC sample from the posterior distribution of θm, Nis the total number of MCMC iterations, and 1(A) is the indicator functionwhich takes the value 1 when A is true and 0 otherwise.With the S updated after each interim look, we design the trial allowingthe early termination:1. Early stop for non-inferiority:The non-inferiority of treatment m compared to the standard treat-ment c will be claimed if the posterior probability of non-inferiority,154S, is larger than some pre-defined probability threshold U :S > U.2. Early stop for inferiority:The inferiority of treatment m compared to the standard treatmentc will be claimed if the posterior probability of non-inferiority, S, issmaller than some pre-defined probability threshold L:S < L.Also, the whole trial will be terminated if it runs out of samples.B.1.2 The Bayesian adaptive design outputsTreatment effect estimationThe treatment effect of treatment m at the end of the trial, Ef (θm), isestimated empirically by running the simulation for T times and taking themean of the posterior mean from each simulation run:Ef (θm) =1TT∑t=1(E(θm | data)t),where E(θm | data)t is the posterior mean of θm at the t-th simulation run.The power of the trialUnder this specific Bayesian adaptive non-inferiority trial, the power, Power,is defined as the probability that the probability of the treatment m beingbetter than the standard treatment c with a difference larger than the min-imum important difference (MID), i.e.,Power = P (P (θm − θc > MID) ≥ U | data and design).155Sample size calculationsWhen the burn-in sample size R0, batch size R, and maximum sample sizeRmax are specified when planning the trial, the maximum number of interimanalyses, IAmax, is determined as:IAmax =⌊Rmax −R0R⌋+ 1.The sample size at trial termination is an important criterion for theefficiency of the adaptive designs. Intuitively, if the statistical power is high,the trial will often end with a relatively small amount of samples; if thepower is low, the trial will keep running and end up with a larger amount ofsamples, and will reach the maximum sample size without making the finalconclusions. We can visualize the sample size distribution and estimate themean sample size, Eˆ(SS), of a specific design through simulating the trialT times by:Eˆ(SS) =1TT∑t=1Rt,where Rt is the total sample size in the t-th simulation, which is estimatedby:Rˆt =R0 +R(IAt − 1), IAt < IAmaxRmax, IAt = IAmax ,where IAt is the number of interim analyses in the t-th simulation.156Appendix CSupporting information forChapter 4C.1 Derivations of Var(µc)As defined in equation (11):µc = µ∗ + µX − µ∗,Then for the variance:Var(µc) = Var(µ∗ + µX − µ∗)= Var(µ∗) + Var(µX − µ∗),as Cov(µ∗, µX − µ∗) = 0.Notice:µX − µ∗ = (X⊤X)−1X⊤ZS−1Z⊤(P − I)Y ,where:S = Z⊤Z − Z⊤X(X⊤X)−1X⊤Z and P = X(X⊤X)−1X⊤.Then by the law of total variance:Var(µX − µ∗|X) = Var(E(µX − µ∗|X,Z)|X) + E(Var(µX − µ∗|X,Z)|X).For the conditional expectation and conditional variance of the bias-correction term:157E(µX − µ∗|X,Z) = E(− (X⊤X)−1X⊤ZS−1Z⊤(I − P )(Xβ + Zγ + ϵ1|X,Z))= −(X⊤X)−1X⊤Zγ.Var(µX − µ∗|X,Z) = Var(− (X⊤X)−1X⊤ZS−1Z⊤(I − P )(Xβ + Zγ + ϵ1|X,Z))= σ21(X⊤X)−1X⊤ZS−1Z⊤X(X⊤X)−1.To find the marginalized variance of the above expectation over Z, weregress Z on X by the following multivariate regression:Z = XΦ+ ϵ,where:• Φ is a p×m matrix (coefficients),• ϵ is an n×m error matrix.We model the error terms in ϵ with the following block structure:vec(ϵ) ∼MVN (0, In ⊗W ),where:• W is an m×m covariance matrix capturing the relationships betweenthe m response variables (columns of Z),• In is the n × n identity matrix, indicating independence across the nobservations (rows of Z).We aim to compute the variance:Var(−(X⊤X)−1X⊤Zγ | X).158Substitute Z = XΦ+ ϵ into the term:−(X⊤X)−1X⊤Zγ = −(X⊤X)−1X⊤(XΦ+ ϵ)γ.Expanding:−(X⊤X)−1X⊤Zγ = −(X⊤X)−1X⊤(XΦγ)− (X⊤X)−1X⊤(ϵγ).The first term:−(X⊤X)−1X⊤(XΦγ),is deterministic and does not contribute to the variance. The variance comesfrom the second term:Var(−(X⊤X)−1X⊤(ϵγ) | X).Using the properties of variance for linear transformations:Var(−(X⊤X)−1X⊤(ϵγ) | X)= (X⊤X)−1X⊤Var(ϵγ | X)X(X⊤X)−1.Note that:ϵγ =m∑j=1γjϵ·j .Since vec(ϵ) ∼ MVN (0, In ⊗ Σ), the covariance between ϵ·j and ϵ·k isgiven by Wjk. Therefore:Var(ϵγ | X) = (γ⊤Wγ)In.Substitute Var(ϵγ | X) = (γ⊤Wγ)In into the variance expression andsimplify, we obtain:Var(−(X⊤X)−1X⊤(ϵγ) | X)= (γ⊤Wγ)(X⊤X)−1.Finally, we have the approximated theoretical variance of the estimatorµc as shown in equation (13):159ˆVar(µc) = σˆ22(X⊤X)−1+(γˆ⊤Wˆ γˆ)(X⊤X)−1 + σˆ12(X⊤X)−1X⊤ZS−1Z⊤X(X⊤X)−1160C.2 Derivations of Var(µw)Based on equation (11) and equation (19), the variance of µw is:Var(µw) = Var(w1µ∗ + µX − w1µ∗),= w21Var(µ∗) + Var(µX) + w21Var(µ∗)− 2w1Cov(µX , µ∗).Notice that2Cov(µX , µ∗) = Var(µX) + Var(µ∗)−Var(µX − µ∗),andVar(µX − µ∗) ≈ (γˆ⊤Wˆ γˆ)(X⊤X)−1 + σˆ12(X⊤X)−1X⊤ZS−1Z⊤X(X⊤X)−1.We can finally obtain equation (20). Take the first derivative of Var(µw)with respect to w1 and set it to 0:(Var(µw))′ii= 0We can obtain equation (21). Check the second derivative:(Var(µw))′′ii= 2((Var(µ∗))ii+(Var(µ∗))ii)≥ 0,we find that(Var(µw))′is monotonic increasing with respect to w1, andtherefore there exists a global minimum. When w1,min /∈ [0, 1],w1,min =0, w1,min < 0,1, w1,min > 1.161C.3 Additional simulation resultsTable C.1: The performance of bias-corrected estimator under simulationsetting S1–S3 where a = b = 0.2Scenario Bias MSE TheorSE EmpSE CoverageS1 90% old data 0.001 0.007 0.086 0.085 0.954S2 50% old data 0.000 0.009 0.094 0.094 0.951S3 10% old data -0.001 0.043 0.205 0.207 0.947Table C.2: Simulation performances of the 3 methods when there is 90% ofold data, a = 0.2, and b = 2.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC -0.002 0.034 0.183 0.183 0.947 1HonestPrecisionposterior (HP)-0.002 0.033 0.183 0.183 0.951 11621Na¨ıveposterior (NP)-0.003 0.034 0.183 0.183 0.949 11812∗ The computation speed is calculated by treating the speed of MCMC as one unit.Table C.3: Simulation performances of the 3 methods when there is 50% ofold data, a = 0.2, and b = 2.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC 0.002 0.007 0.086 0.086 0.950 1HonestPrecisionposterior (HP)0.002 0.007 0.086 0.086 0.950 6921Na¨ıveposterior (NP)0.002 0.007 0.086 0.086 0.949 6936∗ The computation speed is calculated by treating the speed of MCMC as one unit.162Table C.4: Simulation performances of the 3 methods when there is 10% ofold data, a = 0.2, and b = 2.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC -0.001 0.004 0.066 0.066 0.955 1HonestPrecisionposterior (HP)-0.001 0.004 0.066 0.066 0.954 3851Na¨ıveposterior (NP)-0.001 0.004 0.066 0.066 0.954 3815∗ The computation speed is calculated by treating the speed of MCMC as one unit.Table C.5: Simulation performances of the 3 methods when there is 90% ofold data, a = 0.2, and b = 20.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC 0.002 0.042 0.204 0.205 0.944 1HonestPrecisionposterior (HP)0.003 0.042 0.202 0.204 0.945 11545Na¨ıveposterior (NP)0.002 0.042 0.202 0.204 0.945 11442∗ The computation speed is calculated by treating the speed of MCMC as one unit.Table C.6: Simulation performances of the 3 methods when there is 50% ofold data, a = 0.2, and b = 20.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC 0.001 0.008 0.090 0.089 0.952 1HonestPrecisionposterior (HP)0.001 0.008 0.090 0.089 0.951 6895Na¨ıveposterior (NP)0.001 0.008 0.090 0.089 0.952 6881∗ The computation speed is calculated by treating the speed of MCMC as one unit.163Table C.7: Simulation performances of the 3 methods when there is 10% ofold data, a = 0.2, and b = 20.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC 0.000 0.004 0.067 0.067 0.948 1HonestPrecisionposterior (HP)0.000 0.004 0.067 0.067 0.948 3801Na¨ıveposterior (NP)0.000 0.004 0.067 0.067 0.948 3786∗ The computation speed is calculated by treating the speed of MCMC as one unit.Table C.8: Simulation performances of the 3 methods when there is 90% ofold data, a = 1.2, and b = 0.2.Methods Bias MSE TheorSE EmpSE CoverageComputa-tion speed∗MCMC 0.007 0.016 0.127 0.127 0.949 1HonestPrecisionposterior (HP)0.001 0.017 0.131 0.131 0.951 11861Na¨ıveposterior (NP)0.003 0.018 0.115 0.115 0.934 11712∗ The computation speed is calculated by treating the speed of MCMC as one unit.164Appendix DSupporting information forChapter 6D.1 More results of percentage differencesbetween the three methods and the trueCATEWhen X2 is fixed at 1 and b is 1, the overall pattern is quite similar to whatwas observed when b is 0.5, as shown in Figure D.1. However, when b is 2.5,Kallus’s approach performs best when X1 is around -1.5, Yang et al.’s elasticapproach is most effective when X1 is around -0.75, and the R-int approachhas the best performance as X1 approaches 1.5, as shown in Figure D.2.165Figure D.1: Percentage differences between the three methods and the trueCATE under simulation setting S1a with OBS size m = 500 and b = 1. Inthis scenario, X2 is held constant at 1.166Figure D.2: Percentage differences between the three methods and the trueCATE under simulation setting S1a with OBS size m = 500 and b = 2.5. Inthis scenario, X2 is held constant at 1.D.2 Simulation results for 3 R-learner methodsTo better understand how the “R-int” approach works, we also checked theresults from 3 other R-learner calculations: R-learner based on solely RCT(“R-rct”), R-learner based on solely OBS (“R-obs”), and R-learner basedon naive combination of RCT and OBS assuming there is no unmeasuredconfounding in OBS (“R-na¨ıve”). These evaluations were conducted for thesimulation scenarios described in Chapter 6. The results are summarized inthe following tables from Table D.1 to D.4.167Table D.1: The averaged RMSE for the 3 R-learner methods based on (1a)procedure to generate RCT samples while the OBS sample size m is 500with the corresponding RCT sample size n is 200. The results are based on5000 simulation runs.b R-rct R-obs R-naive0 0.69 (0.01) 0.43 (0.01) 0.32 (0.01)0.5 0.68 (0.01) 0.64 (0.01) 0.49 (0.01)1 0.67 (0.01) 0.94 (0.01) 0.75 (0.01)1.5 0.69 (0.01) 1.16 (0.01) 0.94 (0.01)2.5 0.68 (0.01) 1.38 (0.01) 1.14 (0.01)5 0.69 (0.01) 1.54 (0.01) 1.27 (0.01)Table D.2: The averaged RMSE for the 3 R-learner methods based on (1a)procedure to generate RCT samples while the OBS sample size m is 2000with the corresponding RCT sample size n is 200. The results are based on5000 simulation runs.b R-rct R-obs R-naive0 0.68 (0.01) 0.30 (0.01) 0.25 (0.01)0.5 0.68 (0.01) 0.56 (0.01) 0.48 (0.01)1 0.69 (0.01) 0.90 (0.01) 0.78 (0.01)1.5 0.68 (0.01) 1.14 (0.01) 1.01 (0.01)2.5 0.69 (0.01) 1.37 (0.01) 1.21 (0.01)5 0.68 (0.01) 1.53 (0.01) 1.36 (0.01)Table D.3: The averaged RMSE for the 3 R-learner methods based on (1b)procedure to generate RCT samples, with n = 200 and m = 500. Theresults are based on 5000 simulation runs.b R-rct R-obs R-naive0 1.69 (0.02) 0.42 (0.01) 1.13 (0.01)0.5 1.69 (0.01) 0.64 (0.01) 1.10 (0.01)1 1.68 (0.01) 0.94 (0.01) 1.12 (0.01)1.5 1.70 (0.02) 1.16 (0.01) 1.16 (0.01)2.5 1.66 (0.01) 1.38 (0.01) 1.21 (0.01)5 1.67 (0.01) 1.54 (0.01) 1.27 (0.01)168Table D.4: The averaged RMSE for the 6 methods based on the secondprocedure to generate RCT samples while the OBS sample size m is 500,1000, and 2000 with the corresponding RCT sample size n fixed at 200. Theresults are based on 5000 simulation runs.m R-rct R-obs R-naive500 0.27 (0.01) 0.57 (0.01) 0.50 (0.01)1000 0.27 (0.01) 0.57 (0.01) 0.54 (0.01)2000 0.27 (0.01) 0.57 (0.01) 0.55 (0.01)169
Thesis/Dissertation
eng
Population and Public Health
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Wong, Hubert
Vila-Rodriguez, Fidel
Developing statistical methodologies to support embedding clinical trials within learning health systems
Text