Optimal Treatment Planning underConsideration of PatientHeterogeneity and PreparationLead-TimebyMohammad Reza SkandariB.Sc. in Industrial Engineering, Sharif University, 2007M.Sc. in Socio-Economics Engineering, University of Tehran, 2009M.Sc. in Industrial & Systems Engineering, The University of Florida, 2011A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe Faculty of Graduate and Postdoctoral Studies(Business Administration)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)August 2016© Mohammad Reza Skandari 2016AbstractThis thesis comprises three chapters with applications of stochastic optimizationmodels to vascular access planning for patients with chronic kidney disease (CKD).Hemodialysis (HD) is the most common treatment for patients with end-stage renaldisease, the last stage of CKD. There are two primary types of vascular accessesused for HD, arteriovenous fistula (AVF), and central venous catheter (CVC). AnAVF, which is created via a surgical procedure, is often considered the gold standardfor delivering HD due to better patient survival and higher quality of life. However,there exists a preparation lead-time for establishing a functional AVF since it takesseveral months to know whether the surgery was successful, and a majority of AVFsurgeries end in failure.In this thesis, we address the question of whether and when to perform AVFsurgery on patients with CKD with the aim of finding individualized policies thatoptimize patient outcomes. In Chapter 2, we focus on vascular access planningfor HD dependent patients. Using a continuous-time dynamic programming modeland under data-driven assumptions, we establish structural properties of optimalpolicies that maximize a patient’s probability of survival and quality-adjusted lifeexpectancy. We provide further insights for policy makers through our numericalexperiments.In Chapter 3, we develop a Monte-Carlo simulation model to address the timingof AVF preparation for progressive CKD patients who have not yet initiated HD. Weconsider two types of strategies based on approaches suggested in recently publishedguidelines. We evaluate these strategies over a range of values for each strategy,compare them with respect to different performance metrics (e.g., percentage ofpatients with an unnecessary AVF creation), and provide policy recommendations.Our simulation results suggest that the timing of AVF referral should be guided bythe individual rate of CKD progression.Motivated by our findings in Chapter 3, we develop a dynamic programmingmodel in Chapter 4 that incorporates patient heterogeneity in disease progressionwhen making clinical decisions. We then apply this modeling framework to thecase of the AVF preparation timing problem introduced in Chapter 3 and providerecommendations that consider patient heterogeneity in CKD progression.iiPrefaceA version of Chapter 2 has been published at Manufacturing & Service OperationsManagement, 17(4): 608 - 619 (2015). A version of Chapter 3 has been publishedat the American Journal of Kidney Diseases, 63(1): 95-10 (2014).These two papers are co-authored with Prof. Steven Shechter and Dr. NadiaZalunardo. They were involved in the stages of problem formulation and analysis,provided feedback during the course of both research projects, and contributedto manuscript edits. I was responsible for writing the majority of these papers,developing and implementing all the models, and preparing all the numerical results.iiiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Optimal Vascular Access Planning on Hemodialysis . . . . . . . . . 21.2 Optimal Vascular Access Planning Prior to Hemodialysis . . . . . . 21.3 Patient Type Bayes-Adaptive Treatment Plans . . . . . . . . . . . . 32 Optimal Vascular Access Planning on Hemodialysis . . . . . . . . 52.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2.1 Optimal Timing of Medical Interventions . . . . . . . . . . . 72.2.2 Vascular Access Choice . . . . . . . . . . . . . . . . . . . . . 72.2.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 Modeling Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3.1 Access-Based Patient Survival . . . . . . . . . . . . . . . . . 102.3.2 AVF Creation Process . . . . . . . . . . . . . . . . . . . . . . 122.3.3 Objective Functions . . . . . . . . . . . . . . . . . . . . . . . 132.3.4 Dynamic Programming Formulation . . . . . . . . . . . . . . 142.4 Analytical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4.1 Total Lifetime . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4.2 QALE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.4.3 Critical Disutility . . . . . . . . . . . . . . . . . . . . . . . . 162.4.4 Kidney Transplant . . . . . . . . . . . . . . . . . . . . . . . . 17ivTable of Contents2.5 Numerical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.5.1 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . 212.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Optimal Vascular Access Planning Prior to Hemodialysis . . . . 253.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.2 Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.3.1 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.3.2 Modeling eGFR Progression . . . . . . . . . . . . . . . . . . 273.3.3 AVF Creation and Long-term Patency . . . . . . . . . . . . 273.3.4 Patient Survival . . . . . . . . . . . . . . . . . . . . . . . . . 313.3.5 AVF Referral Decision Making . . . . . . . . . . . . . . . . . 313.3.6 Actual versus Estimated HD Start Date . . . . . . . . . . . 313.3.7 Model Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . 323.3.8 Model Validation . . . . . . . . . . . . . . . . . . . . . . . . 323.3.9 Sensitivity Analyses . . . . . . . . . . . . . . . . . . . . . . . 323.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.4.1 Incident Vascular Access Type and Percent Having an Unnec-essary AVF Creation . . . . . . . . . . . . . . . . . . . . . . 333.4.2 Life Expectancy . . . . . . . . . . . . . . . . . . . . . . . . . 333.4.3 Effects of Age . . . . . . . . . . . . . . . . . . . . . . . . . . 333.4.4 Sensitivity Analyses . . . . . . . . . . . . . . . . . . . . . . . 343.4.5 Probabilistic Sensitivity Analysis . . . . . . . . . . . . . . . 373.5 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . 404 Patient Type Bayes-Adaptive Treatment Plans . . . . . . . . . . . 434.1 Related Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.1.1 Methodological Papers . . . . . . . . . . . . . . . . . . . . . 444.1.2 Application Papers . . . . . . . . . . . . . . . . . . . . . . . 454.1.3 Contributions & Chapter Structure . . . . . . . . . . . . . . 464.2 Monotonicity Results . . . . . . . . . . . . . . . . . . . . . . . . . . 474.2.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.2.2 Monotone Value Functions . . . . . . . . . . . . . . . . . . . 474.2.3 Monotone Optimal Policies . . . . . . . . . . . . . . . . . . . 484.3 Bayes-adaptive Treatment Plans . . . . . . . . . . . . . . . . . . . . 494.3.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 494.3.2 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.3.3 MDP Formulation . . . . . . . . . . . . . . . . . . . . . . . . 504.3.4 Monotone Value Functions . . . . . . . . . . . . . . . . . . . 514.3.5 Monotone Policies in Optimal Stopping Problems . . . . . . 534.4 Optimal Timing of AVF Preparation . . . . . . . . . . . . . . . . . 544.4.1 Timing of AVF Preparation . . . . . . . . . . . . . . . . . . 55vTable of Contents4.4.2 MDP Formulation . . . . . . . . . . . . . . . . . . . . . . . . 564.4.3 Numerical Results . . . . . . . . . . . . . . . . . . . . . . . . 594.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615 Conclusions, Extensions and Further Applications . . . . . . . . . 62Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66AppendicesA Chapter 2 Mathematical Proofs . . . . . . . . . . . . . . . . . . . . . 76A.1 General Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76A.2 Analytical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77A.2.1 Proof of Theorem 2.1: . . . . . . . . . . . . . . . . . . . . . . 77A.2.2 Proofs of Theorems 2.2, 2.3-2.5, Corollaries 2.1-2.2, and Propo-sition 2.1: . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79A.2.3 Proofs of Theorems 2.6, 2.7: . . . . . . . . . . . . . . . . . . 87B Chapter 4 Mathematical Proofs . . . . . . . . . . . . . . . . . . . . . 90B.1 General Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90B.2 Analytical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91B.2.1 Proofs of Results in Section 4.2: . . . . . . . . . . . . . . . . 91B.2.2 Proof of Results in Section 4.3 . . . . . . . . . . . . . . . . . 92B.2.3 Proof of Results in Section 4.4: . . . . . . . . . . . . . . . . 95viList of Tables2.1 Baseline parameters used for calculating the critical disutility. . . . . 192.2 Sensitivity analysis for the critical disutility. . . . . . . . . . . . . . . 233.1 Baseline model parameters . . . . . . . . . . . . . . . . . . . . . . . . 303.2 Simulation results for preparation window policies . . . . . . . . . . 343.3 Simulation results for threshold policies . . . . . . . . . . . . . . . . 353.4 One-way sensitivity analysis parameters. . . . . . . . . . . . . . . . 383.5 Two-way sensitivity analysis results on . . . . . . . . . . . . . . . . 393.6 Parameters for probabilistic sensitivity analysis . . . . . . . . . . . . 393.7 PSA results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.1 Heterogeneity of eGFR progression for chronic kidney disease patients 56viiList of Figures2.1 Modeling framework for vascular access dynamics. . . . . . . . . . . 92.2 Survival probability and failure rate for a 67 year old HD patient. . . 102.3 QALE plot for a patient with transplant option. . . . . . . . . . . . 182.4 Base case hazard rate functions for a 67 year old patient’s lifetime. . 202.5 Critical disutility and HD duration for 67 and 82 year old patients. . 212.6 % Remaining QALE increase from the non-optimal policy to . . . . . 223.1 An overview of the Monte Carlo simulation model. . . . . . . . . . . 293.2 Validation plots. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.3 Policy comparisons: tradeoff curve . . . . . . . . . . . . . . . . . . . 363.4 Policy comparisons – expected lifetime on hemodialysis . . . . . . . 374.1 Earliness/lateness cost of AVF ready time. . . . . . . . . . . . . . . . 574.2 Probability density function of AVF preparation time. . . . . . . . . 594.3 Optimal policy for AVF preparation timing . . . . . . . . . . . . . . 60A.1 Possible cases for FL(u)(a). . . . . . . . . . . . . . . . . . . . . . . . 78A.2 Induction step and the hypothetical random variable L′. . . . . . . . 79A.3 Linking v(NF, n, t, y), v(NF, n, t+ y, 0), and vpi0(NF, n, t+ y). . . . 80viiiAcknowledgmentsMy sincere thanks go to my supervisor, Prof. Steven Shechter at the Sauder Schoolof Business for his invaluable help and support during my PhD studies. I am thankfulto the two other members of my committee, Prof. Mahesh Nagarajan at the SauderSchool of Business, and Dr. Nadia Zalunardo (MD) at the Department of Medicinefor their insightful comments and suggestions.I would also like to thank my wife Negar and my parents for their support andencouragement throughout these years which made all this possible.ixTo my wife, Negar.xChapter 1IntroductionThere has been a growing interest in the application of mathematical models de-veloped in the field of Operations Research (OR) to health-care problems. Thesemodels can help policy makers and practitioners to deliver high quality care, at alow cost, in a timely manner. Treatment planning, an active area in the field ofhealth-care analytics, deals with complex decisions a clinician faces on a daily basis;decisions such as when to start a medication regiment, if/when to refer a patient forsurgery, and how often to monitor patients with a chronic disease.Approximately 23 million American adults have chronic kidney disease (CKD) [1]and 550,000 have end-stage renal disease (ESRD) [2]. Most ESRD patients aretreated with hemodialysis (HD) [3]. In order to perform HD, patients need tohave a vascular access. The preferred vascular access for HD is an arteriovenousfistula (AVF) [3] due to greater longevity and lower complication rates; however,it may take several months and more than one surgical procedure to establish ausable AVF [4, 5]. If the AVF is created too late, it may not mature in time, anda central venous catheter (CVC) may be used; however, CVCs are associated withan increased risk of morbidity and mortality [6–9]. On the other hand, creating anAVF too early is undesirable due to a small increase in risk of complications andwasting the limited lifetime of an AVF before HD is needed [10].Existing guidelines for whether/when to refer a patient for the AVF creationsurgery are inconsistent and based on expert opinion [10]. In my thesis, I investigatethe vascular access planning problem for patients with CKD. One of the key featuresof this problem is that unlike other treatments where they can be administeredwhenever desired, using an AVF as the vascular access for HD requires a stochasticpreparation lead-time (the time from the first AVF surgery until an functional AVFbecomes available). Heterogeneity of patients with respect to the rates at whichCKD progresses [11] is another feature of this problem. We develop three data-drivenanalytical models that incorporate these features when designing patient-specifictreatment plans. In the remainder of this chapter, I briefly describe and motivateeach chapter, discuss the objectives, and outline main results of our models.11.1. Optimal Vascular Access Planning on Hemodialysis1.1 Optimal Vascular Access Planning onHemodialysisIn Chapter 2, we investigate vascular access planning for CKD patients who havestarted HD. Using AVF for HD is associated with better survival and quality oflife in comparison with HD using a CVC [7, 12]. Nevertheless, the process of AVFcreation has some disutility associated with it, which can be attributed to the surgeryand post-surgery inconveniences, complications or costs. Therefore, it is not clearunder what conditions an HD dependent patients should undergo the AVF creationsurgery.The purpose of this chapter is to address the following questions: 1. Whethernew HD patients should undergo a surgery for AVF creation or not, and 2. Whetheran AVF surgery should be performed on existing HD patients if a previous AVF fails.We develop a dynamic programming model to find individualized optimal policiesthat maximize a patient’s probability of survival and remaining quality adjustedlife expectancy considering factors such as the patient’s age at the time of decisionand hemodialysis onset, probability of AVF surgery success, hemodialysis relatedutilities, and the AVF creation disutility. We show structural properties of opti-mal policies under certain modeling assumption. As an extension, we consider thepossibility of kidney transplant and how it affects optimal vascular access planningdecisions. We provide further insights for policy makers through our numericalexperiments.1.2 Optimal Vascular Access Planning Prior toHemodialysisIn Chapter 3, we investigate vascular access planning for CKD patients who have notyet started HD. Due to the AVF preparation lead-time (the time from the first AVFsurgery until a functional AVF becomes available), guidelines recommend startingthe AVF preparation process well in advance of HD need. If the AVF is created toolate, it may not be ready in time, and a CVC may be used until an AVF becomesavailable. A late AVF is unfavorable since the risk of morbidity and mortalityincreases when dialyzing with a CVC [6–9]. On the other hand, creating an AVFtoo early is undesirable due to wasting the limited lifetime of an AVF before HD isneeded. To avoid the consequences of having a functional AVF earlier or later thanHD start time, it is ideal for the patient to have an AVF that becomes functionalright at the time of HD start. Nevertheless, due to intrinsic uncertainties in theAVF preparation lead-time as well as the time of HD initiation, the ideal case ishardly achievable.Estimated glomerular filtration rate (eGFR) is often used as the primary measureof kidney health. Nephrologists monitor eGFR progression periodically to decide21.3. Patient Type Bayes-Adaptive Treatment Planswhen to initiate HD as well as when to start AVF preparation. In this chapter, wedevelop a detailed data-driven Monte Carlo simulation model to determine the opti-mal timing of the AVF preparation. We evaluated 2 strategies based on approachessuggested in recently published guidelines [13–16]:1. a “preparation window” strategy, where AVF preparation starts as soon as apatient’s HD is anticipated to begin within a specific time window (e.g., thenext 12 months),2. an “eGFR threshold” strategy, where AVF preparation starts as soon as a pa-tient’s eGFR falls below a specific threshold (e.g., eGFR< 15 mL/min/1.73m2).We evaluate these strategies over a range of values for each strategy (preparationwindows ranging between 3-18 months and eGFR thresholds ranging between 10-30 mL/min/1.73m2) with respect to different performance metrics (e.g., a patient’slife expectancy after HD initiation and percentage of patients with an unnecessaryAVF creation). We also discuss how different strategies might perform when appliedacross a cohort of patients that vary in initial age, level of kidney function, and rateof CKD progression.1.3 Patient Type Bayes-Adaptive Treatment PlansHeterogeneity of patients with respect to disease progression and response to medicalinterventions is an important characteristic of clinical decision making problems. InChapter 4, we formulate and analyze the problem of designing patient type Bayes-adaptive treatment plans defined as follows. We consider designing treatment planswhen treatment-dependent patient outcomes vary across the population in a waythat 1) we can categorize patients into distinct types, 2) we cannot perfectly identifya patient’s type a priori, and 3) the patient type can be observed partially bymonitoring the patient health over time. We assume a Bayesian setting in which westart with some prior belief about the patient type and update our belief by observingthe patient health over time using Bayes’ rule, hence the name “patient type Bayes-adaptive treatment plans”. We formulate the problem as a partially observableMarkov decision process (POMDP) with a two-dimensional state-space, where thestate consists of the patient health and progression type. We first provide structuralproperties of the value-function, as well as the optimal policy for the special case ofoptimal stopping problems for a multi-dimensional state-space. Then, we provideconditions under which these results can be applied to our POMDP model.Using the framework developed for designing patient type Bayes-adaptive treat-ment plans, we revisit the AVF preparation timing question posed in Chapter 3 byconsidering two types of patients, patients with slow and fast eGFR progression. Weshow that under data-driven assumptions, the optimal AVF preparation timing pol-icy is monotone in a patient’s current eGFR as well as our belief that the patient is a31.3. Patient Type Bayes-Adaptive Treatment Plansslow progressor. Through numerical experiments we provide recommendations thatconsider patient heterogeneity in chronic kidney disease progression when decidingif/when to begin the AVF preparation process. We also discuss model outputs andcompare the resulting policies with existing guidelines and the policy implications.4Chapter 2Optimal Vascular AccessPlanning on Hemodialysis 12.1 IntroductionEnd-stage renal disease (ESRD), the final stage of chronic kidney disease (CKD),occurs when the kidneys can no longer perform their essential task of removing wasteproducts from the blood. Patients with ESRD require one of two interventions tostay alive: dialysis or kidney transplantation. Dialysis refers to the removal of wasteand excess water from the body by circulating blood through a filter surrounded byclean fluid. While kidney transplantation yields better patient outcomes [17], thedemand for organs far outstrips the available supply, and nearly 100,000 patientsawait a kidney transplant in the US [18]. Therefore, dialysis is the only realistictreatment option for the majority of patients with ESRD.Hemodialysis (HD) is the most common form of dialysis, accounting for 92% ofthe incident dialysis cases in 2011 [19]. HD involves the circulation of blood from apatient through a dialysis machine. The blood stream is typically accessed in one oftwo ways: by creation of an arteriovenous fistula (AVF) or by insertion of a centralvenous catheter (CVC). An AVF is created by a surgical procedure in which anartery is connected to a vein in the lower or upper arm. In contrast, placing a CVCis a minor procedure in which synthetic tubing is inserted directly into a large vein,usually in the neck. The AVF is often considered the gold standard for vascularaccess [14] because it is associated with lower infection and mortality rates [9] andhigher quality of life [7,12]. The preference for using AVFs for HD is underscored bythe Fistula First Breakthrough Initiative (FFBI), whose mission is “to improve thesurvival and quality of life of hemodialysis patients by optimizing vascular accessselection - which for most patients will be an AV fistula . . . ” [20]. Current guidelinesreflect this by suggesting that patients on HD should be referred for an AVF surgerywhen possible [14,16].Although the benefits of an AVF over a CVC may seem clear, there are somemajor differences between them that deserve careful consideration before recom-mending one access versus the other. First, a CVC can be used immediately afterplacement for HD, whereas an AVF requires a lead time of approximately 3 months1A version of this chapter has been published at Manufacturing & Service Operations Manage-ment, 17(4): 608 - 619 (2015).52.1. Introductionfrom the time of surgical creation until it has matured for possible use in HD [21].This is the time it takes for the vein used in the AVF to become thick and largeenough to support the insertion of needles necessary for each HD session. However,a significant proportion of created AVFs (around 50%) do not mature to a pointthey can be used for HD [22, 23]. In these cases, patients and their doctors maydecide to undergo a subsequent AVF surgery, provided there are still suitable vesselslocated elsewhere on the arms to allow for this (typically two locations on each armmay be considered). Furthermore, even if AVF creation is successful, a mature andfunctional AVF has a limited lifetime, with a 15% annual failure probability [24,25].Finally, while an AVF has quality of life and morbidity advantages relative to a CVConce it is in use for HD, it still has several disadvantages associated with it priorto that time. Since the procedure is more invasive than a CVC insertion, it bringsabout the usual concerns with any surgery (e.g., patient anxiety, infection, post-operative recovery). In some cases, an AVF creation might compromise the bloodsupply to the hand, which can lead to permanent tissue and neurological damage.Furthermore, AVFs impose physical limitations (e.g., heavy lifting with the AVFarm is not advised), and some patients find AVFs disfiguring. In summary, an AVFis superior to a CVC conditional on being available for immediate use in HD. How-ever, that is not the decision faced by patients and their doctors. Instead, theymust decide whether or not to begin the AVF creation process, with the uncertainoutcomes, disutilities, and durations just described.The renal community has recently begun debating the complexities of vascularaccess choice [10], raising concerns about whether “fistula first” should continue tobe the treatment paradigm for all patients. [26] and [27] discuss opposing viewsregarding whether or not AVF is the best vascular access for HD patients, and[28] comments on this debate. The decision is especially germane for the elderlypopulation; [29] suggests considering factors such as an elderly patient’s remaininglife expectancy and personal preferences when making a recommendation of vascularaccess. This relates to the growing momentum in the medical community to takea personalized and shared approach (between clinicians and patients) to medicaldecisions, rather than the one-size-fits-all approach of most clinical guidelines [30].The need for individualized renal care has also been emphasized in [31].The goal of this chapter is to bring a data-driven, analytical approach to inves-tigate if and when HD patients should undergo an AVF surgery. In the spirit ofpatient-centered care, we focus on the patient’s perspective and consider objectivesrelated to patient lifetime and quality-adjusted lifetime. Our study is also in linewith the recent emphasis in the US on the use of comparative effectiveness research(CER) for guiding evidence-based decision making in medicine [32]. The importanceof CER for guiding renal disease treatments in particular is discussed in [33]. Ona similar note, [29] noted the importance of future quantitative studies evaluatingtiming and type of vascular access to improve mortality and quality of life in el-derly patients. Our work provides decision makers with both high level analytical62.2. Literature Reviewinsights on AVF vs. CVC decisions as well as quantitative studies to guide decisionsspecifically for different patient types (including the elderly).2.2 Literature ReviewIn this section, we review existing literature related to our research in two categories:1. Operations Research/Management Science (OR/MS) papers on the optimal tim-ing of medical interventions, and 2. clinical papers describing decision-analyticmodels of vascular access choice for renal disease patients.2.2.1 Optimal Timing of Medical InterventionsDecisions regarding the optimal time to apply a medical treatment or screen patientsfor some disease have received growing attention in the OR/MS community in thepast decade. For instance, [34] developed a Markov decision model to investigatethe optimal timing of a living-donor liver transplant to maximize a patient’s qualityadjusted life expectancy (QALE). [35] addressed the question of when to initiateHIV treatment so as to maximize the expected lifetime or quality-adjusted life-time of a patient. [36] and [37] applied partially observable Markov decision process(POMDP) models related to breast cancer treatment. [38] used a simulation-basedapproximate dynamic programming algorithm to derive near optimal strategies forinitiation and management of dialysis therapy. [39] investigated the problem of op-timal prostate biopsy referral decisions and proved the existence of a control-limittype policy that maximizes a patient’s QALE. [40] studied the effect of budgetaryrestrictions on breast cancer diagnostic decisions by solving a mixed-integer programthat maximizes a patient’s total QALE under resource constraints.2.2.2 Vascular Access ChoiceA number of decision analytic models related to AVF decision making have ap-peared in the recent clinical literature. [41] developed a Markov model to studycost-effectiveness of different vascular access alternatives among incident HD pa-tients. They found that the decision of whether to use AVFs or arteriovenous grafts(AVGs), another type of vascular access used in HD, for patients with incident HDdepends highly on the AVF maturation failure probability, and they suggested takingthis into account for individualized access planning. [42] compared two AVF creationtiming policies for a 70-year-old patient with stage 4 CKD using a Markov modeland reported life expectancy and quality adjusted life expectancy as the outcomes.They recommended further research on patient preference and cost implicationswhen making AVF creation recommendations. Using a data-driven Monte-Carlosimulation model, [43] investigated policies of AVF surgery timing for CKD pa-tients. They assessed two classes of AVF referral policies over a range of values interms of patient expected lifetime, proportion of AVF incident HD patients, and72.2. Literature Reviewproportion of unused AVFs. A recent study by [44], using the framework of [41],found that a patient’s characteristics such as diabetes status and gender also affectthe cost-effectiveness of a vascular access choice.2.2.3 ContributionsThe purpose of this chapter is to address the following questions: 1. Whether newHD patients should undergo a surgery for AVF creation or not, and 2. Whether anAVF surgery should be performed on existing HD patients if a previous AVF fails.We aim to find individualized optimal policies that maximize a patient’s probabilityof survival and remaining quality adjusted life expectancy, and we consider howAVF timing policies depend on patient age.Unlike existing papers on vascular access choice for HD patients whose recom-mendations are simulation-based [41, 42, 44], we have tackled the problem analyti-cally. For instance, we prove the form of optimal policy for both lifetime and QALEmetrics. Also, our work provides the optimal decision for vascular access choice forthe whole duration of a patient’s dependency on HD, whereas existing literatureonly focuses on the vascular access decision at the time of HD initiation.Existing recommendations for vascular access choice for HD-dependent patientsdo not appear evidence-based, and are not patient-specific. We construct an ana-lytical, data-driven model that incorporates several key factors when making AVFsurgery decisions. In particular, patient age, AVF success probabilities, hazard ratefunctions for patient survival on an AVF vs. CVC, and patient quality of life mea-sures are important drivers of our model-based recommendations.One of the key model components in determining the optimal policy, the AVFcreation disutility, may be difficult to estimate and varies from patient to patient.To circumvent this issue, we introduce a dual view of the optimal policy by us-ing the notion of a critical disutility. We prove that at each decision point, thenephrologist needs to know only if a patient’s AVF creation disutility is below orabove a critical factor, rather than its exact value, to make the optimal decision.This involves engaging patients in the decision making process, by assessing theirindividual tolerances for undergoing surgery.Several unique features of our research contribute to the OR/MS literature onmedical decision making. We model a patient’s lifetime as a continuous randomvariable, which facilitates our consideration of a patient’s treatment-based non-stationary mortality rate. One key difference between our framework and otherclinical decision making papers in the literature is that we consider treatment op-tions which require a stochastic lead-time before they are effective. Whereas theprevious models can assume a mammography, transplantation, or HIV treatmentcan be administered whenever it is desired, an AVF cannot be created instanta-neously. Moreover, there is uncertainty regarding if and when a successful AVF willbe attained. This brings an interesting dynamic to the decision, because the benefitof the AVF may not be as substantial at the time it is ready, and moreover, the82.3. Modeling Frameworkpatient may die beforehand.2.3 Modeling FrameworkWe consider an ESRD patient already on HD with at least one unused AVF oppor-tunity. Note that our model will answer two types of AVF creation timing questions:1. should patients who just begin HD on a CVC undergo an AVF surgery (assumingno AVF is already in process), and 2. should patients who have an AVF fail duringthe course of HD undergo an AVF surgery? We assume that the patient choosesbetween two vascular access types: CVC, and AVF. We discuss the role of AVGsin Section 2.6. In Figure 2.1, the decision making framework is illustrated. As thedecision flowchart suggests, we make the following assumption:Assumption 2.1 (Decision points). A patient can undergo an AVF surgery at anytime, provided there are remaining AVF opportunities and an AVF is not underpreparation or being used.Any remaining AVF Chance?AVF Surgery?Use AVF Use CVCDeathAVF FailsAVF based survival CVC based survivalYesAVF Maturation YesNoAVF Timing PolicyWhetherWhenNoStart HDPatient has AVF?YesNoAVF SurgeryFigure 2.1: Modeling framework for vascular access dynamics (including decisionsand events) for an HD-dependent patient.The dynamics and principles of the model can be summarized as follows. Apatient receives HD via an AVF as long as she has an established one. When there isno functional AVF (either when one fails or at the beginning of HD when the patientstarts HD without a functional AVF) the patient dialyzes via a CVC as a bridgeaccess. During this time, the policy determines whether and when to perform AVFsurgery on the patient. If the policy recommends an AVF surgery, the patient goes92.3. Modeling Frameworkthrough the AVF creation process and waits until possibly attaining a functionalAVF. If all AVF opportunities have been used up, or the policy recommends nofurther AVF creation, the patient remains on HD with a CVC until death.We discuss clinical factors impacting the decision of whether and when to useAVF opportunities in the following sections.2.3.1 Access-Based Patient SurvivalPatient survival on HD depends on the vascular access being used [9,45]. Figure 2.2(left), obtained from [9], shows that patients receiving HD via an AVF experiencestochastically better survival than those who receive it via a CVC. Nevertheless, thesurvival benefit of AVF over CVC, measured by the failure rate difference, diminishesas a patient continues using HD (see Figure 2.2 (right)). In addition, a patient’sfailure rate on either access types increases as the HD duration increases. 30%40%50%60%70%80%90%100%0 1 2 3 4 5% Surviving HD Duration (years) AVF CVC 0.050.100.150.200.250 1 2 3 4Estimated Failure Rate HD Duration (years) AVF CVCFigure 2.2: Access-based survival probability and failure rate for a 67 year old HDpatient. Survival probability (left) is obtained from [9], and failure rate (right) isestimated from survival probabilities.We use these data-driven observations to justify further assumptions below.First, we describe some notation: t: time since the patient started HD FX(t): survival probability function of a random variable X until time t(FX(t) = P[X > t]) fX(t): probability density function of a random variable X at time t rX(t): hazard rate function of a random variable X at time t Xt: residual lifetime of a random variable X at time t (a random variable de-noting the remaining lifetime of X from time t onward conditional on survivaluntil time t)102.3. Modeling Framework µ(t) ∈ {a, c}: patient’s HD access type at time t (a if it is an AVF, and c, ifit is a CVC). C: random variable denoting patient’s lifetime when remaining on a CVCfrom HD initiation time until death. A: random variable denoting patient’s lifetime when remaining on an AVFfrom HD initiation time until death. L: random variable denoting patient’s lifetimeNote that the distributions of C and A are dependent on a patient’s age at the timeHD commences, but we do not denote this dependency for ease of notation.Our next assumption describes how survival depends on HD duration and vascularaccess type.Assumption 2.2 (Survival distribution). A patient’s remaining survival only de-pends on the length of time that the patient has been on HD and the ongoing modeof HD access (an AVF or a CVC).Mathematically, Assumption 2.2 impliesP(Lt ≥ x∣∣µ(t′) for all t′ ≤ t, µ(s) = a for all t ≤ s ≤ x+ t) = FAt(x), (2.1)P(Lt ≥ x∣∣µ(t′) for all t′ ≤ t, µ(s) = c for all t ≤ s ≤ x+ t) = FCt(x). (2.2)We can explain Equation 2.1 (and similarly Equation 2.2) as follows. If a patientwould remain on an AVF from t until t+x, her probability of surviving until t+x isthe same as a patient who has been on an AVF from HD start and has survived untilt. Note that this assumption has been applied in related clinical research papers aswell (see [41,44] for instance).The following are the definitions for common types of stochastic orders for randomvariables.Definition 2.1 (Usual stochastic order). We say X ≤st Y , if and only if FX(t) ≤FY(t) : ∀t.Definition 2.2 (Hazard rate order). We say X ≤hr Y , if and only if rY(t) ≤ rX(t) :∀t.The following three assumptions formalize the data-driven observations of Figure2.2 (right).Assumption 2.3 (Relative performance). The hazard rate of C is higher than orequal to the hazard rate of A, at all ages. Mathematically, we have rC(t) ≥ rA(t), ∀t.Note that Assumption 2.3 corresponds to the CVC hazard rate curve lying abovethe AVF hazard rate curve in Figure 2.2 (right), and is equivalent to C ≤hr A bydefinition.112.3. Modeling FrameworkAssumption 2.4 (Diminishing difference). The difference between hazard rates ofC and A decreases in time, i.e., rC(t)− rA(t) is decreasing in t.Note that Assumption 2.4 corresponds to the diminishing gap between the CVChazard rate curve and the AVF hazard rate curve of Figure 2.2 (right). As we showin the Appendix A, Lemma A.3, we have that rC(t)− rA(t) is decreasing in t if andonly if FC(t)FA(t)is log-convex in t.Finally, the following assumption states that an HD patient’s mortality rate, oneither access type, increases with patient age (or rather, we should more preciselysay with “duration on HD”).Assumption 2.5 (Diminishing performance). Random variables A and C have theincreasing failure rate (IFR) property, i.e., rA(t) and rC(t) are increasing in t.Assumption 2.5 is demonstrated by the fact that both curves in Figure 2.2 (right)are increasing.We believe that assumptions posed on a patient’s survival (Assumptions 2.3-2.5)are intuitive. For instance, that a patient’s failure rates increase by age, or that thebenefit of one intervention over another decreases with time can be justified by theaging process and increasing presence of co-morbidities as a patient ages.2.3.2 AVF Creation ProcessAfter a patient and her clinician decide to use an AVF for HD, she visits a vascularsurgeon for AVF placement. After the surgery is performed, the AVF maturation,a process by which a fistula becomes suitable to use for HD, begins (e.g., developsadequate flow, wall thickness, and diameter). It takes approximately 3 months ofAVF maturation to learn whether the AVF is usable or not for HD. However, amajor issue for AVF placement is that around 50% of AVFs fail to mature [41,46]. Furthermore, even if an AVF creation is successful, it has an annual failureprobability of 15% [24, 25]. These factors are critical to the decision of whether ornot a patient should undergo an AVF surgery.We use the following notation for random variables describing the AVF creationprocess: Mi: random variable denoting the maturation time of the ith AVF Bi: random variable denoting whether the creation of the ith AVF is successful(Bi = 1 if successful, 0 otherwise) Zi: random variable denoting the total lifetime of the ith AVF given that itmatures NFt: number of failed AVFs creations up to time t122.3. Modeling FrameworkNote that NF0 is not necessarily zero since the patient may have AVF creationsprior to HD initiation. We make the following assumption about the AVF creationprocess.Assumption 2.6 (AVF maturation and lifetime). All respective random variablesdescribing the AVF creation process are stationary. Furthermore, Mi, and Zi areidentically distributed (across subsequent creations) and independent of the historyof previous AVF creations.The stationarity of AVF creation variables is justified by the relatively short lifeexpectancy of HD patients (average of 6.2 years [19]). To the best of our knowledge,there is no evidence in the literature on the dependence of AVF maturation timeand lifetime on the history of previous creations. However, it is natural to thinkthat patients who fail to achieve a mature AVF on one attempt are more likely tohave a failed creation in future attempts, and that the failure probability increaseswith the number of past AVF failures.Assumption 2.7 (AVF creation success probability). The probability that an AVFmatures is a decreasing function of NF , the number of previous maturation failures,i.e., Pt(Bi = 1|NFt) is decreasing in NFt for any i.Henceforth, by “AVF surgery success”, we mean achieving a functional AVFafter the maturation period.2.3.3 Objective FunctionsTotal LifetimeA natural metric for comparing policies is the total lifetime of a patient. Thus, weconsider maximizing a patient’s total lifetime (in the usual stochastic order) as oneof the objective functions.Quality Adjusted Life Expectancy (QALE)Using AVF for HD not only brings better survival, but also has a slightly higherquality of life for the patient, in comparison with HD using a CVC [7,12]. Neverthe-less, the process of AVF creation has some disutility associated with it, which can beattributed to the surgery and post-surgery inconveniences, complications or costs.We define a patient’s quality adjusted life expectancy as the quality adjusted life-time on each vascular access minus the AVF surgery disutility for each AVF surgeryperformed (whether successful or not).The following parameters are used in defining a patient’s QALE: qa, qc: utility of being an HD patient who receives HD via an AVF or a CVC,respectively.132.3. Modeling Framework d: AVF creation disutilityBased on the estimates in the literature [7, 12], we make the following assumptionabout the access-based quality of life coefficients.Assumption 2.8 (Relative quality of life). Patients experience a better quality oflife dialyzing via an AVF than via a CVC, i.e., we assume qa ≥ qc.2.3.4 Dynamic Programming FormulationTo explain the dynamics of the model to optimize a patient’s QALE and proveour analytical results, we formalize the decision making process with a dynamicprogramming model. The model components are as follows: States: The set of vectors (NF, n, t) consisting of NF , the number of previousAVF maturation failures, n, the number of AVF chances left, and t, the timesince the patient started HD, corresponds to a living state, and the absorbingstate ∆ corresponds to the death state. Sufficiency of (NF, n, t) to representa living state is justified by our assumptions on patient survival (Assumption2.2) and AVF variables (Assumption 2.6). The choice of these variables torepresent a patient’s state will become clear when we discuss state transitions. Actions: At each state (NF, n ≥ 1, t), one of two actions can be taken: eitherto perform an AVF surgery at time t+y or to perform no more AVF surgerieson the patient. Note that the no more AVF action is the case of AVF surgeryat y =∞. Nevertheless, we keep it in the action space for clarity. When n = 0,the only option is to remain on CVC for the remainder of the patient’s lifetime(the no more surgeries action). The choice of the next AVF surgery time torepresent actions in the modeling framework is justified by Assumption 2.1about decision times. Transitions: Based on Assumption 2.1, we only need to consider transitionsbetween “decision states” (i.e., the subset of living states for which the pa-tient does not have a functional or maturing AVF but has AVF opportunitiesremaining), the first transition to state (NF, 0, t), and the transition to state∆.From decision state (NF, n ≥ 1, t) and planning for surgery at t+y, the patientmay transition to one of three possible states. If the AVF matures and thepatient survives until t1 = t + y + M + Z, she transitions to (NF, n − 1, t1).If the AVF creation fails but the patient survives until t2 = t + y + M , shetransitions to (NF +1, n−1, t2). Otherwise, the patient does not survive untilthe next decision state and transitions to ∆. Immediate reward: The immediate reward consists of a patient’s QALEfrom time t until the next living state or death time. If the decision is to remain142.4. Analytical Resultson CVC until death (either because the policy in use recommends this, or thepatient uses up her AVF chances), the patient receives an immediate rewardequal to her CVC utility weighted remaining lifetime (qcECt). Otherwise,at state (NF, n, t) and when the surgery is planned at t + y, the patient’simmediate reward includes her expected weighted lifetime from time t until t′(the next decision time) or death time (sometime between t and t′), and mayinclude an AVF creation disutility (if she survives until t+ y). The value of t′depends on whether AVF matures or not as it was discussed in the previoussection.We discuss the value function and other components of our dynamic program-ming model as needed in the proofs in Appendix A.2.4 Analytical ResultsIn this section, we present analytical results. All of the proofs for the analyticalresults are given in Appendix A.2.4.1 Total LifetimeOur main result concerning total lifetime is that in order to maximize an HD pa-tient’s survival probability until any time t′ (and as a result to maximize expectedlifetime), she should undergo an AVF surgery as soon as an opportunity becomesavailable. We prove this in a stochastic ordering sense: an identical patient whoundergoes an AVF surgery earlier than another patient lives stochastically longerthan that patient.Theorem 2.1. Under Assumptions 2.1-2.6, delaying AVF surgery stochasticallydecreases a patient’s lifetime.Note that the stochastic ordering result means that the immediate surgery policymaximizes the chance a patient may survive until a kidney transplant, either througha deceased or living kidney donor (see Theorem 2.6).We have the following general result regarding the difference in mean residuallifetimes of variables A and C.Theorem 2.2 (Mean residual lifetime difference). Let A and C be any arbitraryrandom variables satisfying Assumptions 2.3-2.5. We have E[At]−E[Ct] is decreasingin t.We can explain Theorem 2.2 intuitively as follows. Assumptions 2.3 and 2.4imply that the absolute difference of hazard rates of variables A and C is decreasingin time. Also, using the definition of hazard rate function, we have rXt(s) = rX(t+s)for any random variable X and t, s ≥ 0. Therefore, the difference of hazard rates ofrandom variables At′ and Ct′ at any arbitrary time is less than that of At and Ctfor any t′ ≥ t.152.4. Analytical Results2.4.2 QALEIn this section, we prove the optimality of a class of policies for the QALE metricthat we refer to as HD duration threshold polices. Let τ denote a policy that atstate (NF, n ≥ 1, t) recommends an AVF surgery immediately, if t < τ(NF ), andrecommends a CVC, otherwise. Then, we have:Theorem 2.3 (Optimality of Threshold Policies). Under Assumptions 2.1-2.8, thereexists a threshold policy τ∗ that maximizes the QALE of the patient.Corollary 2.1. The optimal HD duration threshold, τ∗, is decreasing in NF .Note that the optimal policy is independent of the number of remaining AVFchances. In the next proposition, we prove that the optimal threshold can be foundusing a binary search.Proposition 2.1 (Binary Search). An optimal threshold policy can be found usinga binary search for τ∗ over [0, tmax], where tmax is a reasonable upper bound for τ∗.We can set tmax equal to the time at which the patient reaches the age of 100years because patients never undergo AVF surgeries after that age.2.4.3 Critical DisutilityThe result of Theorem 2.3 assumes one already has an estimate of the patient’sdisutility for an AVF creation. However, this may be difficult to estimate preciselyin practice. Also, the optimal HD duration threshold needs to be calculated fordifferent values of NF . To circumvent these challenges, we introduce a dual viewof the HD duration threshold policy. We show that at any time, the decision ofwhether to do an AVF surgery or not is determined by comparing the patient’sAVF creation disutility with a critical value. Thus, in order to make a decision, weonly need to know whether the AVF creation disutility is above the critical value ornot, rather than require a precise estimate of the AVF disutility itself.Theorem 2.4 (Critical Disutility). Under Assumptions 2.1-2.8, for any HD du-ration t, there exists a non-negative critical AVF creation disutility, denoted bydcr(NF, t), such that the optimal decision at time t is to perform an AVF surgeryimmediately if the patient’s AVF creation disutility is less than the critical disutility(i.e., if d < dcr(NF, t)), and is to use a CVC for the rest of patient’s life, otherwise.The critical disutility at t is defined as the residual QALE difference betweenimmediate AVF surgery at t before subtracting the AVF creation disutility andstaying on a CVC until death for a patient with only one AVF chance. In Theorem2.5, we show that the critical disutility is proportional to the success probabilityof the current AVF creation. Therefore, one can calculate the critical disutilityfor different values of NF by calculating it for some baseline AVF creation successprobability and then multiplying it by some factor (the ratio of the current AVFsuccess probability given NF previous failures to the baseline value).162.4. Analytical ResultsTheorem 2.5. Under Assumptions 2.1-2.8, the critical disutility is proportional tothe AVF creation success probability.Based on the following corollary, we can use the critical disutility function to findthe critical HD duration for patients with different values of AVF creation disutilityd > 0.Corollary 2.2 (Relationship between Critical Disutility and Critical Duration).Suppose Assumptions 2.1-2.8. Then, τ∗(NF ) = inf(t : dcr(NF, t) ≤ d).Note that Theorem 2.4 provides an alternative way of comparing the optimalpolicy for individual patients as follows: if the critical disutility for one patientis always smaller than another, then the first patient has a smaller HD durationthreshold, given that both patients have the same AVF creation disutility.2.4.4 Kidney TransplantIn this section, we investigate an extension to the basic model by considering kidneytransplant as a possible renal replacement therapy (RRT) for the patient. Sincekidney transplant provides the best long-term health outcomes for the patient [17],we assume that a patient’s residual QALE on transplant is higher than on HD, andthe patient switches to kidney transplant as their RRT as soon as she is offered afavorable donated kidney. In other words, we assume that the decision of whetherto accept a kidney donation is exogenous to our model.Let Ψ be the (stochastic) time until a favorable kidney donation becomes avail-able. We assume that once the patient receives the donated kidney, her futuresurvival is independent of her HD history. Then, we can easily show that Theorem2.1 holds under the extended model as follows.Theorem 2.6. Under Assumptions 2.1-2.6, delaying AVF surgery stochasticallydecreases a patient’s lifetime, when the patient receives a donated kidney at timeΨ.For the QALE metric on the other hand, the result of the basic model (optimalityof threshold policies) does not necessarily extend even under a deterministic timeuntil transplant. We show in the following example that the optimal policy canbe neither immediate surgery, nor to stay on CVC forever (i.e., until transplant ordeath).Consider a patient whose access-dependent lifetime on HD follows exponentialdistributions with means 3 and 1.5 months, for AVF and CVC access types, respec-tively. Assume that the patient has a living donor who can donate a kidney after6 months (the wait time can be due to medical tests the donor and patient shouldundergo, the operating room and surgeon availability, etc.). If the patient survivesuntil transplant time, she receives 16 additional QALE months. Also assume that172.4. Analytical Resultsthe maturation time is negligible, AVFs all mature and do not expire in the first 6months, and qA = qC = 1.In Figure 2.3, total QALE as a function of AVF surgery time for a patient withAVF disutility of 3 months is depicted. In this case, the optimal decision is to waitand perform surgery at t = 1.5 months. This demonstrates that the optimal policyis not of the form “perform AVF now or never.”We can explain the behavior observed in this example as follows. On the onehand, using the AVF for HD can benefit the patient by giving her a better qualityof life as well as increasing her chances of survival until the transplant time. Onthe other hand, because of the AVF creation disutility, the AVF should not beused too early either since the patient may die well in advance of the transplanttime. Therefore, the patient should use up some survival time without the AVF, toincrease the chance that when the AVF is created, it bridges the patient’s survivaluntil the time of transplant (and therefore the AVF creation is not wasted). 1.71.81.922.10 1 2 3 4 5 6Total QALE (months)AVF Surgery Time (months)Figure 2.3: QALE plot for a patient with transplant option. Solid line depicts apatient’s total QALE at t = 0 as a function of AVF surgery time. The dotted lineshows total QALE when the patient stays on CVC until death or transplant.Although threshold policies may be suboptimal in general, we prove their opti-mality under additional assumptions in the following theorem. For this result, weassume the patient has a living donor, and thus, a deterministic time until trans-plant seems reasonable. Also, we assume that time until transplant is short enoughthat AVFs, if mature, do not expire before transplant time. Finally, we allow forthe possibility of transplant cancellation, for instance, if the donor changes his mindor if the donated kidney is to be found incompatible as a result of the tests.Theorem 2.7. Suppose that the time until transplant, if it is not canceled, isdeterministic, i.e., Ψ = ψ for some known ψ, FAt(ψ − t)−FCt(ψ − t) is decreasing182.5. Numerical Resultsin t, and an AVF’s lifetime, if it matures, is greater than the time until transplantwith probability one. Then under Assumptions 2.1-2.8, there exists a thresholdpolicy τ∗ that maximizes the QALE of the patient.Note that FAt(ψ − t) (and similarly FCt(ψ − t)) can be interpreted as thesurvival probability of a patient on an AVF (CVC) until the transplant time, givenher survival until t. The assumption that FAt(ψ − t) − FCt(ψ − t) is decreasingin t is supported by the empirical data given in [9]. Also, Theorem 2.7 cannot beapplied to the aforementioned example, because FAt(ψ − t) − FCt(ψ − t) equalse−13(6−t) − e− 11.5 (6−t), which is not a decreasing function.2.5 Numerical ResultsTo demonstrate the results of Theorems 2.3 and 2.4, we performed a numericalstudy. The baseline values for different model parameters and sources used aregiven in Table 2.1.Table 2.1: Baseline parameters used for calculating the critical disutility.Variable Value ReferenceOn-HD survival (67 year old) — [9]On-HD survival (82 year old) — [9,45]AVF primary failure probability (67 year old) 50% [22,23]AVF primary failure probability (82 year old) 75% [22,23]Yearly failure probability for a functional AVF 15% [24,25]Maturation time (months) Uniform(2,4) [21,22]Utility of dialysis with AVF 0.81 [7, 12]Utility of dialysis with CVC 0.77 [7, 12]For patients’ HD survival, we used [9], which provides only the first five years ofsurvival outcomes for a cohort of 67 year old patients. To obtain complete survivalcurves, we extrapolate the hazard rate functions so that Assumptions 2.3-2.5 aresatisfied. Specifically, we assume that the AVF and CVC hazard rates increaselinearly after the last observed hazard rate with slopes αA and αC , respectively.We need to assume αA ≥ αC ≥ 0, so that Assumptions 2.4 and 2.5 are satisfied.To have Assumption 2.3 met, we modify the hazard rates for CVC such that afterthe point the hazard rate curves meet (if they ever meet, which is always the casewhen αA > αC), we have that rC(t) = rA(t), with the slope of the line equal toαA (see Figure 2.4 for an illustration). We calculated the average rate of increasefor the AVF and CVC hazard rate functions (that is the slope connecting first andlast observed hazard rates). Denoting these slopes with r¯A and r¯C respectively, weassumed αA = r¯A and αC = r¯C (below, we perform one-way sensitivity analyses byconsidering scenarios in which αA = (1± 25%)r¯A and αC = (1± 25%)r¯C).192.5. Numerical Results 0.0750.1250.1750.2250.2750.3250 1 2 3 4 5 6 7 8 9Hazard Rate On-HD Duration (years) AVF - Actual CVC - ActualAVF - Base Case Projection CVC- Base Case ProjectionC A Figure 2.4: Base case hazard rate functions for a 67 year old patient’s lifetime onHD.Based on the hazard rate functions, a 67 year old patient’s entire survival curvewas calculated. We used the result of Theorem 2.4 to calculate the critical disutilityas a function of HD-duration using Monte-Carlo simulation (see the proof of The-orem 2.4 in the online supplement). Figure 2.5 (left) shows the critical disutilityunder the baseline assumption for survival extrapolation. For example, a 67 year oldpatient who has been on HD for 2 (3) years should undergo AVF surgery providedher AVF disutility is less than 85 (65) QALE days.Note that we assume the same probability of AVF success, regardless of NF , asthe clinical literature does not yet provide this detail when discussing maturationfailure rates. Nevertheless, one can easily calculate the critical disutilities as afunction of NF by multiplying the function by a proper factor (see Theorem 2.5 formore details).Recall that the motivation for the critical disutility approach was for cases inwhich it might be difficult to estimate precisely a patient’s disutility for the AVFsurgery. However, based on Corollary 2.2, Figure 2.5 (left) can also be invertedto answer questions regarding a patient for whom a precise estimate of the AVFdisutility is obtained. For example, the figure also indicates that if a 67 year oldpatient has a disutility of 85 (65) QALE days, then she should undergo an AVFsurgery as long has she has been on HD less than 2 (3) years.To visualize the impact of age at HD initiation on the critical disutility, we haveplotted the critical disutility curves for patients who start HD at ages of 67 and 82years in Figure 2.5 (right). As the plot shows, the critical disutility of the olderpatient is always smaller. For instance for the time of HD initiation, a 67 year oldpatient should undergo AVF surgery as long as her AVF creation disutility is below130 QALE days, while for an 82 year old patient AVF surgery is advisable only202.5. Numerical Results 0204060801001201400 1 2 3 4AVF Creation Disutility (QALE days) On-HD Duration (years) refer for AVF remain on CVC 𝑑𝑐𝑟 0 = 130 0204060801001201400 1 2 3 4AVF Creation Disutility (QALE days) On-HD Duration (years) remain on CVC for HD Start Age =67, 82 refer for AVF for HD Start Age =67, 82 refer for AVF for HD Start Age =67, remain on CVC for HD Start Age =82 Figure 2.5: Critical disutility and HD duration for 67 and 82 year old patients.Onthe left, the critical HD duration for 67 year old patients with AVF creation disutilityof 65 and 85 QALE days is illustrated. It also shows the critical disutility for a 67year old who just begins HD is 130 QALE days. On the right, the critical disutilityfor 67 and 82 year old patients is illustrated.when her AVF creation disutility is below 70 QALE days (Figure 2.5 (right)).In Figure 2.6, we plot the % QALE increase from the non-optimal policy to theoptimal policy as a function of the AVF creation disutility for an 82 year old patientwith one AVF chance, i.e., for n = 1. We have compared the two policies of “no AVFsurgery” and “surgery at HD initiation”, as they represent two opposing opinionsin the literature [26, 27], and therefore the figure indicates what can be gained if adecision maker adheres to a suboptimal policy on one side of the threshold or theother. For d < dcr(0), the optimal policy is to perform AVF surgery on the patientat the time of HD initiation, whereas for d ≥ dcr(0), the optimal policy is to remainon a CVC.2.5.1 Sensitivity AnalysisWe also performed a sensitivity analysis to see how robust the results are to thechanges in the input parameters. The parameters and values tested for one-wayand two-way sensitivity analyses and the corresponding critical disutilities at thetime of HD initiation are given in Table 2.2. For instance, the critical disutilitiesfor patients with 60% and 20% chances of success in having a matured AVF are223 and 76 QALE days, respectively. Since the first patient has a higher chanceof surgery success, she benefits from the surgery more than the other patient, andas a result, she should be undergo AVF surgery at the time of HD initiation aslong as her surgery disutility is less than 223 QALE days, while the other patient212.6. Conclusion 0%2%4%6%8%10%12%0 30 60 90 120% QALE increase AVF creation disutility (QALE days) 𝑑𝑐𝑟(0) Optimal policy: Refer for AVF at HD initiation Optimal policy: Remain on a CVC Figure 2.6: % Remaining QALE increase from the non-optimal policy to the optimalpolicy as a function of the AVF creation disutility. We have compared the twopolicies of “no AVF surgery” and “surgery at HD initiation”for an 82 year oldpatient with n = 1 (other parameters are given in Table 2.1), with the former beingoptimal for d ≥ dcr(0) and the latter for d ≤ dcr(0).benefits from AVF surgery only when the surgery disutility is less than 76 QALEdays. As the results in Table 2.2 suggest, the critical disutility is most sensitive tothe AVF surgery success probability. Based on Theorem 2.5, the critical disutilityis proportional to this parameter, and therefore, it can be easily adjusted by anephrologist based on her perception of a patient’s AVF surgery success probabilityor existing statistics in the local practice.2.6 ConclusionIn this chapter, we considered the problem of vascular access choice between a CVCand an AVF for HD patients, with a goal of maximizing a patient’s total lifetime andQALE. We analytically proved that delaying AVF surgery stochastically decreasesa patient’s lifetime. As a result, the policy of “use the next AVF (opportunity)as soon as a patient starts HD or when the one being used fails” maximizes apatient’s survival probability. We also proved that the optimal policy to maximizea patient’s QALE is of a threshold type: there is an HD duration threshold beforewhich immediate surgery is the optimal choice, while after that time, CVC is theoptimal vascular access choice for the remainder of the patient’s lifetime. Thisthreshold depends on the number of past AVF maturation failures.The AVF creation disutility plays an essential role in determining the criticalHD duration of the QALE optimal policy. Since patients may feel differently about222.6. ConclusionTable 2.2: Sensitivity analysis for the critical disutility (QALE days) of a 67 year oldHD incident patient computed using Monte-Carlo simulation. The default valuesfor each parameter are given in Table 2.1.Parameter Value Critical disutilityN/A Default 151AVF Surgery Success Proba-bility0.2 760.6 223Functional AVF Annual Fail-ure Rate0.1 1720.2 134Maturation Time (months)Uniform [3,5] 150Uniform [4,6] 149Uniform [1,6] 150QALE Coeff [CVC, AVF][0.73,0.81] 164[0.75,0.81] 158[0.81,0.81] 139Patient’s survival projectionparameters [αA, αC ][¯rA,1.25 ∗ r¯C ] 151[¯rA,0.75 ∗ r¯C ] 150[1.25 ∗ r¯A ,¯rC ] 147[0.75 ∗ r¯A ,¯rC ] 155the disutility of AVF surgery, and also because it is not an easy parameter to elicitfrom a patient, our model provides an alternative way to make the optimal AVFtiming decision. We showed that the decision of whether to perform an AVF surgeryor not can be determined solely by comparing the patient’s AVF creation disutilitywith a boundary value reflecting the prospective additional quality lifetime for thepatient, which we refer to as the critical disutility. Thus, a nephrologist can informthe patient of the benefits and inconveniences of undergoing the AVF surgery, andthen, they can collectively decide whether to do the surgery or not. Even if a roughestimate of the patient’s disutility for AVF surgery indicates that it is clearly belowor above the critical disutility, then it will be clear that the patient should or shouldnot, respectively, undergo an AVF surgery. Estimates of a patient’s disutilitiescan be obtained using standard elicitation methods in the medical decision makingcommunity, such as the standard gamble, time trade-off, and visual analog scale [47].This also facilitates getting patients involved in the decision making process, one ofthe key recommendations of the Institute of Medicine’s report on patient-centeredcare, which has been emphasized in the medical community in the past decade [48].We also found that the possibility of receiving a kidney transplant adds newcomplexities to the model and optimal policy structure. Although the optimal policyunder the total lifetime remains the same, the result on QALE metric (optimality232.6. Conclusionof threshold policies) does not necessarily extend, even when the time of transplantis known with certainty. Nevertheless, we provided a theorem which proves thatunder additional assumptions (which are supported by data), threshold policiesremain optimal.Our framework and analytical results may also be relevant to operational ques-tions outside of health care, particularly in the area of machine maintenance andequipment reliability. For example, consider a machine with a vital component. Ifthe component breaks down, it may be replaced with a cheap, available spare. Addi-tionally, one may order a more expensive, higher-quality component, which involvesa lead time for delivery. This is analogous to deciding whether and when to refer apatient for an AVF versus letting them continue to receive HD through a CVC. AnAVF provides higher quality HD outcomes compared to a CVC, but an AVF cannotbe created quickly, and it is more expensive in the sense of the surgical disutility itimposes on patients.24Chapter 3Optimal Vascular AccessPlanning Prior to Hemodialysis23.1 IntroductionApproximately 23 million American adults have chronic kidney disease (CKD) [1]and 550,000 have end-stage kidney disease [2]. Most of these patients are treatedwith hemodialysis (HD) [3]. The preferred vascular access for HD is an arteriovenousfistula (AVF) [3] due to greater longevity and lower complication rates; however,it may take several months and more than one procedure to establish a usableAVF [4, 5]. If the AVF is created too late, it may not mature in time, and acentral venous catheter (CVC) may be used; however, CVCs are associated withan increased risk of morbidity and mortality [6–9]. On the other hand, creating anAVF too early is undesirable due to a small increase in risk of complications andwasting the limited lifetime of an AVF before HD is needed [10]. In 2008 over 80%of incident HD patients in the United States used a CVC as their initial vascularaccess [49]. Although there are multiple reasons for this, suboptimal timing of AVFreferral has contributed to low incident AVF rates [50].Existing guidelines for AVF referral are inconsistent and based on expert opin-ion [10]. Over the past decade, the Kidney Disease Outcomes Quality Initiative(KDOQI) recommendations have varied from referral for AVF creation when HD isanticipated within 12 months (2000) [13], within 6 months (2006) [14], or when esti-mated glomerular filtration rate (eGFR) falls below 30 mL/min/1.73m2 (2002) [15].In 2006, the Canadian Society of Nephrology (CSN) guidelines suggested referral atan eGFR of 15-20 mL/min/1.73m2 in patients with progressive CKD [16].Establishing clearer guidelines may improve incident AVF rates in HD patientsand thereby positively impact on patient outcomes. In this chapter, we developa data-driven, decision-analytic model to provide an objective approach to timingAVF referral in CKD.2A version of this chapter has been published at the American Journal of Kidney Diseases,63(1): 95-10 (2014).253.2. Related Literature3.2 Related LiteratureThe existing literature related to the optimal vascular access planning for incidenthemodialysis patient is discussed in Chapter 3. To my knowledge the only decisionmodel for vascular access planning prior to hemodialysis is the study by [42]. Hire-math et al. [42] compared two AVF creation timing policies for a 70-year-old patientwith stage 4 CKD using a Markov model and reported life expectancy, quality ad-justed life expectancy and costs as the outcomes. Our modeling perspective differsfrom that of Hiremath et al. [42], who took a comparative effectiveness approach.Rather than considering only two possible strategies, we considered a wide rangeof possible referral policies and how they might perform when applied across a co-hort of patients that vary in initial age, level of kidney function, and rate of CKDprogression.3.3 Methods3.3.1 Study DesignWe developed a Monte Carlo computer simulation model in C++ to determine theoptimal timing of AVF referral in patients with CKD. We evaluated 2 AVF referralstrategies based on approaches suggested in recently published guidelines [13–16]:1. a “preparation window” strategy, where referral occurs as soon as HD is an-ticipated to begin within a specific time window (e.g., the next 12 months),2. an “eGFR threshold” strategy, where patients are referred as soon as theireGFR falls below a specific threshold (e.g., eGFR < 15 mL/min/1.73m2).We examined both strategies over a wide range of values for their respective param-eters (see Figures 3.3, 3.4).Figure 3.1 provides an overview of the model. In each simulation replicationof a given referral strategy, patients from a sample cohort enter the model, andtheir eGFR measurements are simulated at periodic intervals. After each eGFRmeasurement, the Nephrologist decides whether to refer the patient for AVF creationor not. For simulated patients who survive until HD commences, we simulate on-dialysis survival according to whether HD is delivered via AVF or CVC. After asimulated patient dies, the next patient in the cohort enters the model, and after allthe patients in the cohort have gone through the model, one simulation replicationis complete. For each AVF referral strategy, we run the same patient cohort through100,000 independent replications.Table 3.1 indicates the base-case parameters of our model, which were derivedfrom the literature and from primary data analysis (further described below). Expertopinion was used in cases where literature-based estimates were unavailable.263.3. Methods3.3.2 Modeling eGFR ProgressionWhile standard time series models (e.g., autoregressive of order one, or AR-1) canconsider correlation from one observation to the next, they assume equally spacedmeasurements [51]. This is unsuitable for our purposes, as the timing of patienteGFR measurements is highly irregular (the average standard deviation of inter-testtimes is approximately one month across patients). Therefore, we applied statisticalmethods proposed by Erdogan et al. [52], which extend standard AR-1 models toirregularly spaced time series. Consider the following OLS linear regression modelfor eGFR progression:eGFR(ti) = β0 + β1ti + tiwhere ti is the time of the ith eGFR measurement. In the OLS model, it is assumedthat the residual terms ti are mutually independent across measurements. Themodel proposed in [52] instead assumes a systematic correlation structure betweenconsecutive residuals as follows:ti = ti−1θ(ti−ti−1) + ωtiThe term θ(ti−ti−1) (with θ between 0 and 1) represents the correlation betweenconsecutive residuals spaced ti − ti−1 time units apart, and the ωti are independentwhite noise terms. This model captures commonly observed properties of residualsin longitudinal data analysis [53]. They are positively correlated, with the degree ofcorrelation decreasing with longer separation between measurements.We used Matlab to fit these regression models to each of the 860 patients in ourcohort of patients who were enrolled in a multi-disciplinary kidney clinic at Vancou-ver General Hospital (VGH) between Jan 1, 1994 and Nov 9, 2010. As a validationstep, we compared the goodness of fit for these two models using the coefficient ofdetermination R2. For the proposed model the coefficient of determination is onaverage 0.51, while for the OLS model it is 0.44.Our simulation model simulates eGFR values for a given patient as follows: firstat the ith eGFR measurement time, ti, the mean value of the patient’s eGFR iscalculated β0 + β1ti. Then, a residual term is added, which is calculated by multi-plying the previous residual by the correlation factor that depends on the elapsedtime since the last measurement (ti−1θ(ti−ti−1)). Finally, a normally distributedwhite noise term is added to this (ωti).3.3.3 AVF Creation and Long-term PatencyOnce an AVF referral is made, the patient first waits for a surgical creation dateand then waits for the AVF to (possibly) mature. The distribution of surgical waittimes was based on 209 HD patients who had AVFs created at Vancouver GeneralHospital between 2005 and 2009. The median surgery wait time was 28 days, with273.3. Methodsa maximum of 65 days. Using the distribution fitting tool of the Arena simulationsoftware [54], we found that a Uniform (0,65) (days) distribution best representedthe variability observed for this duration.In the base case analysis, the probability of an AVF failing to mature was 0.4if a patient had no prior CVC [4, 22, 23, 55–57] and 0.6 if HD had already startedwith a CVC [4, 41, 46, 55, 58]. We assumed the time it takes to determine whetheran AVF is functional for HD or not (with interventions if necessary) is uniformlydistributed between 2 and 4 months [21].We assumed functional AVFs have annual failure probabilities of 0.15 or 0.075,depending on whether the AVF is being used for HD or not, respectively [24, 25].If the AVF fails to mature, or a mature AVF fails, the patient is again referred forAVF creation. We assume a maximum of three attempts at AVF creation, with nomore than two attempts occurring in the predialysis period.283.3. MethodsEnter next patient from cohortSimulate Pre-HD SurvivalSimulate next eGFR readingDialysisRefer for AVF creation?waitmaturationAVF successful?AVF failureSimulate AVF lifetimeNoYesHas AVF?Wait for death or AVF failureAlready referred for AVF?AVF chance left?Wait for death or a successful AVFUse AVFUse CVCPatient DiesNoNo YesYesYesNoNoYesAVF referralAVF surgeryLast patient in the cohort?Last replication?NoYesYesGuideline PerformanceRecord outcomes(total lifetime, ...)Simulate post-HD Survival with AVFSimulate post-HD Survival with CVCGo back to patient 1NoFinishStartFistula initiation guidelinesFigure 3.1: An overview of the Monte Carlo simulation model. One replicationconsists of all patients from the cohort going through the model one at a time, fromthe time they are referred to a kidney clinic, until they die. Dashed lines indicateevents that can occur at any time. For example, dialysis may occur at any timebetween eGFR measurements, while waiting for AVF surgery, or while waiting foran AVF to mature.293.3. MethodsTable 3.1: Baseline model parameters.Model parameter Value ReferencePatient-specific eGFR progressionparameters (β0, β1, σ, θ)Varies by patient Primary data analysis*,[59, 60]Average rate of eGFR decline(mL/min/1.73m2 per year)5.29 Primary data analysis,[59, 60]eGFR level at which dialysis starts Normal(10, 2.5) distri-butionPrimary data analysisSurvival for CKD predialysis Age and gender-dependent[61]Survival for ESRD on dialysis Age, gender, and accesstype-dependent Base-line mortality relativerisk, CVC vs AVF: 1.53[61,62]Time from AVF referral to surgicalcreation (in days)Uniform (0, 65) Primary data analysis,[21]Maximum number of AVF attemptsin predialysis period2 Expert opinion†Maximum number of AVF creationattempts in total3 Expert opinionTime from AVF creation to achievean AVF usable for HD (with inter-ventions if necessary), or AVF aban-donment due to failure (in months)Uniform (2,4) [5, 21]Probability patient is willing to havemore than one AVF attempt1 Baseline assumptionProbability of AVF failing to mature(without history of CVC use)0.4 [22,23,55–57,63]Probability of AVF failing to mature(with history of CVC use)0.6 [4, 41,55,58]Probability of a functional AVF fail-ing, per year (if used for HD)0.15 [24,25]Probability of a functional AVF fail-ing, per year (if not used for HD)0.075 Expert opinion*indicates parameters obtained from analysis of patients treated at the multidisciplinarykidney clinic at Vancouver General Hospital.† indicates values provided by Dr. Nadia Zalunardo, Clinical Associate Professor at theUniversity of British Columbia, Division of Nephrology303.3. Methods3.3.4 Patient SurvivalPatient survival was simulated according to whether the patient is CKD not yeton HD, on HD with an AVF, or on HD with a CVC. We used survival data fromthe USRDS [64] to model predialysis survival, and data from the USRDS [62,64] tomodel vascular access-specific survival for HD patients. To simulate survival timesbeyond the time horizon of the survival curves in these studies, we used completestatistical life tables [61] and estimated relative risk ratios [62] to extrapolate thesurvival curves.3.3.5 AVF Referral Decision MakingWe assume a patient’s eGFR is measured every 3 months for Stage 3 CKD, every 2months for Stage 4 CKD, and every month for Stage 5 CKD. After each simulatedeGFR measurement, a decision is made to refer the patient for AVF creation or towait and reevaluate after the next eGFR measurement.The timing of AVF referral is specified by the strategy being tested. For theeGFR threshold strategy, AVF referral occurs once the simulated eGFR falls belowthe threshold value being tested. For the preparation window strategy, AVF referraloccurs once the anticipated HD start date is within the time window being tested.In reality, a Nephrologist’s recommendation to start HD is based on eGFR combinedwith other important factors such as uremic symptoms. However, these symptomsgenerally appear closer to the HD start date, and are not awaited before AVFreferral. We therefore assume the Nephrologist estimates the HD start date byfitting a regression line through the patient’s history of eGFR measurements, anddetermining when this line would fall to 10 mL/min/1.73m2 (the mean eGFR atHD start in our CKD cohort). We assume AVF referral occurs at the start of HD ifit did not occur before that.3.3.6 Actual versus Estimated HD Start DateThe difference between the Nephrologist’s estimated HD start time and the actualHD start time affects the degree to which an AVF will be ready before or afterHD commences. To account for the considerable inter-patient variability in theactual eGFR at HD initiation in clinical practice, we used the Arena simulationsoftware [54] to fit a probability distribution to the eGFR values at the start ofHD for 204 HD patients in our cohort. The best fit was a Normal distribution,with a mean of 10 and a standard deviation of 2.5 mL/min/1.73m2. We used thisdistribution to simulate the actual eGFR at which HD would commence for eachpatient.313.3. Methods3.3.7 Model OutcomesThe outcomes of interest were: expected remaining lifetime (measured from dialysisinitiation until death), percentage of HD patients who begin dialysis with a CVC,and percentage of patients who have an unnecessary AVF creation (patients whohave an AVF created and die before requiring HD, and patients who had at least onefunctional AVF fail before HD start). We evaluated these outcomes in the overallcohort and stratified by age (at the time of referral to the kidney clinic) in thefollowing groups: 50-60, 60-70, 70-80, and 80-90 years old.3.3.8 Model ValidationWe compared survival curves of simulated patients who enter the clinic in Stage 3and Stage 4 CKD with the Kaplan Meier survival curves of our kidney clinic cohortwho entered in the same stages. The simulated survival curves were within the 95%confidence intervals of the actual survival curves (See Figure 3.2).0 2 4 6 8 1000.20.40.60.81Time (years)Survival probability0 2 4 6 8 1000.20.40.60.81Time (years)Survival probabilityFigure 3.2: Comparison between the Kaplan-Meier survival curves of the actualcohort (the solid step function, with the 95% confidence intervals shown by thedashed step function), with the survival curves of the simulated cohort (smoothsolid lines). The plot on the left shows survival, from the time patients first enterthe clinic in Stage 3 CKD until their death. The plot on the right is similar, forpatients who enter the clinic with Stage 4 CKD.3.3.9 Sensitivity AnalysesWe performed a variety of one-and two- way sensitivity analyses as well as a prob-abilistic sensitivity analyses (PSA) [65]. We compared policies based on the total323.4. ResultsHD lifetime outcome. The parameters and values tested for one-and two-way sen-sitivity analyses are given in Tables 3.4 and 3.5. In each replication of the PSA,we simultaneously sampled all the parameters of the model according to a proba-bility distribution (parameters given in Table 3.6) and then compared the lifetimeobtained by applying the baseline optimal policy to the optimal policy under theset of sampled parameters.3.4 Results3.4.1 Incident Vascular Access Type and Percent Having anUnnecessary AVF CreationFigure 3.3 and Tables 3.2 and 3.3 demonstrate the tradeoff that is observed for bothtypes of strategies: as AVF referral occurred earlier (larger preparation window orhigher eGFR threshold), the percentage starting HD with a CVC decreased butthe percentage with an unnecessary AVF increased. Overall, referral 15 monthsbefore anticipated HD initiation resulted in 34% starting HD with a CVC and 14%having an unnecessary AVF creation. Referral windows between 12 and 18 monthsperformed similarly.Relative differences between referral strategies were more pronounced for thresh-old policies, with respect to the unnecessary AVF creation outcome in particular.For example, a referral threshold eGFR of 20 mL/min/1.73m2 compared to 15mL/min/1.73m2 resulted in a doubling of the percent with an unnecessary AVFcreation from 10 to 20%.3.4.2 Life ExpectancyFigure 3.4 and Tables 3.2 and 3.3 indicate life expectancy differences for a rangeof strategies tested in the base case analysis. The optimal preparation window was15 months before anticipated HD, which yielded an expected lifetime increase of 14days over a preparation window of 6 months. However, any preparation windowbetween 9 and 18 months performed nearly optimally. The optimal eGFR thresholdstrategy was 20 mL/min/1.73m2; however, thresholds of 15 mL/min/1.73m2 orgreater performed similarly.Figure 3.4 displays the result of a strategy where AVF referral is delayed untilHD starts (with a CVC). This yields a shortened expected total lifetime of 73 dayscompared to referral 15 months before anticipated HD start.3.4.3 Effects of AgeAge-stratified results for selected AVF referral strategies are also shown in tables 3.2and 3.3. For any given strategy, aging had a greater relative effect on the percentageof patients with an unnecessary AVF creation (which increased with age mainly due333.4. ResultsTable 3.2: Various output measures from the simulation, for both the overall cohortas well as by 10-year age ranges for preparation window policies. Age is determinedat the time of kidney clinic enrollment. For the total lifetime, the average reductionfrom the best strategy (for preparation window and threshold strategies separately)is reported. A zero indicates that policy was optimal for that cohort. All differencesreported are statistically significant at level = 0.05, using a t-test of equality ofmeans between two policies.Cohort Output MeasurePreparation window (months)3 6 9 12 15 18Overall CohortLifetime reduction (days) 37 13.8 3.8 0.3 0 1% Starting HD with CVC 74% 52% 41% 36% 34% 34%% with unnecessary AVF 3% 6% 9% 11% 14% 16%50-60 year oldsLifetime reduction (days) 41.9 16.5 3.4 0 0.3 1.4% Starting HD with CVC 75% 53% 42% 37% 35% 35%% with unnecessary AVF 1% 3% 5% 7% 9% 11%60-70 year oldsLifetime reduction (days) 36.2 13.5 3.4 0 0.2 0.9% Starting HD with CVC 72% 50% 39% 35% 33% 33%% with unnecessary AVF 3% 5% 8% 10% 13% 15%70-80 year oldsLifetime reduction (days) 30.5 11.7 3.2 0.4 0 0.2% Starting HD with CVC 73% 50% 39% 35% 33% 33%% with unnecessary AVF 4% 7% 11% 14% 16% 19%80-90 year oldsLifetime reduction (days) 27.3 10.4 3.1 0.6 0 0% Starting HD with CVC 74% 52% 42% 38% 37% 36%% with unnecessary AVF 5% 9% 13% 16% 19% 22%to the competing risk of death before HD start) than on the percentage of HDpatients starting with a CVC, which changed little. For example, with the strategyof AVF referral 15 months before anticipated HD start, the percentage of patientswith an unnecessary AVF creation increased from 9% to 19% as age increased from50-60 to 80-90 years old, whereas the percent starting HD with a CVC remainedsimilar at 33 37%.For the two oldest cohorts (70-80 and 80-90 years old), preparation widow strate-gies of 15 18 months resulted in 16 22% with an unnecessary AVF creation; anyeGFR threshold strategy of 20 mL/min/1.73 m2 or higher resulted in unnecessaryAVF percentage consistently above 20%.3.4.4 Sensitivity AnalysesOptimal policies from the base case analysis were robust across one-way sensitivityanalyses. A 15 month preparation window and eGFR threshold of 20 mL/min/1.73m2were optimal or within 0.05% of optimal in each case. In the sensitivity analysis343.4. ResultsTable 3.3: Various output measures from the simulation, for both the overall cohortas well as by 10-year age ranges for threshold policies. Age is determined at thetime of kidney clinic enrollment. For the total lifetime, the average reduction fromthe best strategy (for preparation window and threshold strategies separately) isreported. A zero indicates that policy was optimal for that cohort. All differencesreported are statistically significant at level = 0.05, using a t-test of equality ofmeans between two policies.Cohort Output MeasureeGFR mL/min/1.73m210 15 20 25 30Overall CohortLifetime reduction (days) 37.4 8.1 0 1 2.6% Starting HD with CVC 78% 51% 38% 36% 36%% with unnecessary AVF 4% 10% 20% 30% 38%50-60 year oldsLifetime reduction (days) 34 5.5 0 4.6 7.9% Starting HD with CVC 74% 49% 40% 39% 39%% with unnecessary AVF 4% 9% 16% 23% 28%60-70 year oldsLifetime reduction (days) 37.8 8.1 0 2 3.6% Starting HD with CVC 75% 47% 36% 35% 35%% with unnecessary AVF 4% 10% 18% 28% 37%70-80 year oldsLifetime reduction (days) 36.6 9.7 0 0.2 1% Starting HD with CVC 78% 49% 35% 33% 33%% with unnecessary AVF 5% 12% 22% 33% 41%80-90 year oldsLifetime reduction (days) 36.7 10.9 1.8 0 0% Starting HD with CVC 82% 54% 41% 37% 37%% with unnecessary AVF 5% 12% 24% 38% 48%where HD begins at a mean eGFR of 7 mL/min/1.73m2, the performance of prepara-tion windows between 9 and 15 months was essentially identical (within 1 day of oneanother) as was the performance of eGFR threshold policies of 15 mL/min/1.73m2or greater.Results were robust in two-way sensitivity analysis for AVF maturation failureprobabilities (Table 3.5). A preparation window of 12 months performed optimallyin many cases where AVF failure probabilities were lower than in the baseline case,although the absolute lifetime differences between the 12 and 15 month preparationwindow policies was small (less than 2 days). When the AVF maturation failureprobability was equivalent before and after CVC use, later referral strategies werefavored (preparation window 9 months, eGFR threshold 15 mL/min/1.73m2).We assessed the performance of policies for the range of average cohort CKDprogression rates from 2.78 (average progression in our cohort) to 7 mL/min/1.73m2per year (fast progressors). To achieve a similar incident CVC percentage as the353.4. Results 69121510152031825 3020%30%40%50%60%70%80%90%0% 10% 20% 30% 40% 50%% starting HD with CVC % With an Unnecessary AVF CreationFigure 3.3: Tradeoff curve between % of hemodialysis patients who start witha CVC and % of patients who have an AVF created unnecessarily. Each pointrepresents the value of these two measures for a given AVF referral policy. Thepreparation window (months) and eGFR threshold (mL/min/1.73m2) strategies areshown by circles and squares, respectively.9 month preparation window policy in the baseline case (about 40%), referral15 18 months before anticipated HD start would be required for those progress-ing at 7 mL/min/1.73m2 per year. For threshold policies, referral at eGFR 25mL/min/1.73m2 for fast progressors yielded a similar incident CVC percentage(about 40%) as referral at 20 mL/min/1.73m2 in the baseline case. In our (moreslowly progressing) cohort, similar results were achieved with referral at eGFR 15mL/min/1.73m2.The relative risk (RR) of mortality on HD with CVC versus AVF is a key de-terminant of the absolute magnitude of the lifetime differences between policies.The downside of a late AVF referral is magnified when the relative risk of mortalityis larger. For instance, the lifetime reduction of using a preparation window of 6months instead of 15 months increases from 1 day for RR 1.05 versus 35 days forRR 2.75 (the range of RRs reported by Ravani et al. [62]).363.4. Results3 6 9 12 15 181670168016901700171017201730174017501760At HD1015202530Preparation Window (months)Expected HD lifetime (days)eGFR Threshold (mL/min/1.73m2) eGFR Threshold Preparation WindowFigure 3.4: Policy comparisons with respect to expected lifetime on hemodialysis.The preparation window and eGFR threshold strategies are shown by circles andsquares, respectively. The figure also shows the result of a policy that waits untildialysis begins to refer a patient for AVF (the diamond).3.4.5 Probabilistic Sensitivity AnalysisTo check the robustness of the optimal preparation window and threshold policiesobtained from the baseline model, we performed a probabilistic sensitivity analy-sis (PSA) [65, 66]. In each replication of the PSA, we simultaneously sampled allmodel parameters according to a probability distribution given in Table 3.6, andthen we compared the lifetime obtained by applying the baseline optimal policy, i.e.the preparation window of 15 months and eGFR threshold of 20 mL/min/1.73m2to the optimal policy under the set of sampled parameters. The PSA results (re-ported in Table 3.7) show that the baseline optimal policies are quite robust; thelifetime reduction of the optimal baseline policy from the optimal policy across allPSA samples were on average 4.1 and 2.6 days for preparation window and eGFRthreshold strategies, respectively. Similar to the baseline results, any preparationwindows between 9 and 15 months performed similarly in this respect, and theywere optimal in 70% of all PSA samples. The eGFR threshold policies 15 to 25also had a similar performance and they contributed to the 87% of optimal policies373.4. ResultsTable 3.4: One-way sensitivity analysis for a set of plausible values for model pa-rameters.Parameter(s) Value(s)Time from AVF referral to surgicalcreation (in days)Uniform (0, 30)Uniform (0, 120)eGFR decline rate (mL/min/1.73m2per year)2.787Time from AVF creation to achievean AVF usable for HD or AVF aban-donment due to failure (in months)Uniform (1,3)Uniform (3,5)Uniform (4,6)Uniform (1,6)Probability of AVF failing to mature(Without, With) history of CVC use(0.2, 0.3)(0.3, 0.45)(0.4, 0.4)(0.4, 0.5)(0.5, 0.75)Maximum number of AVF creationattempts in total4Probability patient is willing to havemore than one AVF attempt0.70.5Time between eGFR measurements(in months)stage 3: Uniform (2,4)stage 4: Uniform (1,3)stage 5: Uniform (0,2)Probability of a functional AVF fail-ing, per year (If not used, If used)for HD(0.15, 0.15)(0.115, 0.15)(0.05, 0.1)(0.1, 0.2)Hemodialysis mortality relative riskwith CVC vs. AVF1.051.411.672eGFR level at which dialysis starts Normal (7, 2.5)383.4. ResultsTable 3.5: Two-way sensitivity analysis results on probability of AVF failing tomature with and without history of CVC use. The results in the parenthesis showthe best preparation window and threshold strategies, respectively.Probability of AVF failing to maturewith history of CVC use0.3 0.4 0.45 0.5 .6Probability of AVF fail-ing to mature withouthistory of CVC use0.2 (12, 20) (12, 20) (15, 20) (12, 20) (15, 25)0.3 (9, 15) (12, 20) (12, 20) (12, 20) (15, 20)0.4 (9, 15) (12, 20) (12, 20) (15, 20)0.5 (9, 15) (12, 20)Table 3.6: Parameters and distributions used for probabilistic sensitivity analysis.Parameter(s) DistributionsTime from AVF referral to surgicalcreation (in days)Uniform (0, max), max ∼ Uni-form(30,120)Maximum number of AVF creationattempts in totalSample from (1, 2, 3, 4) with prob-ability (0.2,0. 3,0.4, 0.1)Time from AVF creation to achievean AVF usable for HD or AVF aban-donment due to failure (in months)Equal probability selection fromUniform (1,3), Uniform (3,5), Uni-form (4,6), Uniform (1,6)Probability of AVF failing to mature(Without, With) history of CVC useEqual probability selection from thetwo way sensitivity table (see Table3.5).Probability of a functional AVF fail-ing, per year (If not used, If used)for HDEqual probability selection from{(0.05,0.1),(0.05,.015),(0.05,0.2),(0.075,0.1),(0.075,0.15),(0.075,0.2),(0.1,0.1),(0.1,0.15),(0.1,0.2),(0.115,0.15),(0.115,0.2),(0.15,0.15),(0.15,0.2)}Time between eGFR measurements(in months)stage 3: Uniform (2,4), stage 4: Uni-form (1,3), stage 5: Uniform (0,2),393.5. Discussion and Conclusionacross all PSA samples.3.5 Discussion and ConclusionWe used a simulation model to assess the performance of a range of AVF referralstrategies in individuals with CKD. Except in cases where AVF referral occurredvery late, the differences in expected HD lifetime between policies were modest. Theeffects of different strategies on incident vascular access type and the likelihood ofcreating an unnecessary AVF were clinically meaningful and useful as a guide tooptimizing the timing of AVF referral. The results for the overall cohort suggestAVF referral about 12 months before HD is anticipated is appropriate; this supportsKDOQI guidelines published in 2000 [13, 14]. An eGFR threshold for referral of15-20 mL/min/1.73m2, as suggested by the CSN guidelines, was also appropriateoverall [16]. However, the choice of strategy should also be guided by an assessmentof the individual’s rate of CKD progression to avoid excessively early or late referralsin slow or rapid progressors, respectively.Threshold strategies have the advantage of easier implementation since they donot require forecasting the anticipated HD start date; however, they fail to considera patient’s rate of CKD progression. In contrast, preparation window strategiesconsider the rate of CKD progression rather than just the most recent measure-ment; however, accurately estimating the time to HD start is a major challenge forclinicians in part because the decision to start HD is based on multiple factors inaddition to the eGFR.The lowest incident CVC percentage we observed was about 35%. The combina-tion of AVFs failing to mature and a limited number of AVF opportunities limits howlow this number can be; however, it can probably be further reduced by consideringAV grafts as an option. AV grafts have the advantage of near certain short-termpatency and no prolonged maturation time compared to AVFs. AV grafts can beplaced nearly immediately before HD is required and in certain patients may be thepreferred approach if AVF maturation is felt to be very unlikely, as suggested byRosas et al. [67] We did not consider AV grafts in our model since there is almost nouncertainty regarding early patency. Finally, incident CVC rates are also likely tobe lower where AVF maturation failure probabilities are substantially less than the0.4 we used in our baseline model, which was based primarily on North Americanreports.Dialysis planning in very elderly individuals (where the competing risk of deathis high) is a challenge receiving increasing attention [68–70]. With a 15 month prepa-ration window, 19% of AVFs created for 80-90 year olds are unnecessary comparedto 9% for 50-60 year olds (age is determined at the start of kidney clinic follow-up).The increased risk of creating an unnecessary AVF in the elderly is a potentially sig-nificant source of morbidity and health care resource utilization with no benefit. Atailored approach to AVF referral based on age is therefore indicated. In our model,403.5. Discussion and Conclusionreferral for AVF creation 6 months before the anticipated HD start for 80-90 yearolds, and 9 months for 70-80 year olds yielded a similar percentage with unnecessaryAVF creations as a 12 to 15 month window for 50-60 year olds.Our model focused on patient outcomes and did not consider system costs. Whilecost-effectiveness analysis (CEA) is an important analysis for health policy evalu-ation, we chose to perform a comparative effectiveness analysis (CER) instead. Inrecent years, there has been significant interest in CER, which focus on how poli-cies compare with respect to health outcomes, rather than costs. A recent articleunderscored the importance and need for CER in evaluating treatments for kidneydisease [33].The strengths of our simulation model include its mimicking of the dynamicforecasting and AVF decision making process faced by Nephrologists. It explicitlyfactors in forecast inaccuracies when evaluating the various preparation window-based referral policies. Our model also considers a wide variety of patient types interms of their initial eGFR and rate of disease progression. We performed a varietyof sensitivity analyses, and the robustness of our results is reassuring and potentiallysupports the generalizability of our findings to CKD populations elsewhere.A number of assumptions pose limitations to our model. We did not account foruncommon complications of AVF creation in our model (e.g. high output cardiacfailure and limb ischemia), we did not include AV grafts, and we did not includea transition to kidney transplant. Further, there were few studies in the literaturefrom which to obtain maturation probabilities for second AVFs and AVFs createdafter CVC use. However, we performed sensitivity analyses to determine the impactof this limitation. Finally, it is regrettable that the literature on the impact ofvascular access and dialysis related interventions on quality of life is quite limited.Our modeling framework can easily incorporate improved quality of life data (andthereby also report on quality-adjusted life expectancy) whenever good estimatesbecome available.We modeled eGFR decline using linear regression, an approach which is con-sistent with other studies [60, 71, 72]. However, recent reports indicate that somepatients do not experience a linear decline in eGFR [11, 73]. In a study by O’Hareet al. [11], 12% of patients experienced a nonlinear, rapid rate of eGFR decline inthe two years before the start of dialysis. Our model did not consider patients whoexperience a sudden acceleration in eGFR decline, leading to a much earlier require-ment for HD. Our simulated incident AVF percentage applies to a large proportionof patients stably progressing to ESRD (and who are followed in a multidisciplinarykidney clinic). Since a significant number of patients do not fall in this category,the incident AVF results we report are optimistic if applied indiscriminately to allCKD patients. Therefore, they should not be used as a specific target for incidentAVF percentages in all CKD patients.In conclusion, our results suggest that the optimal policy for AVF referral is whenthe estimated time to HD initiation is within about 12 months, or when eGFR falls413.5. Discussion and Conclusionbelow 15-20 mL/min/1.73m2. However, the choice of strategy should also be guidedby an assessment of the individual’s rate of CKD progression to avoid excessivelyearly or late referrals. Since elderly CKD patients have a greater risk of having anunnecessary AVF creation due to the competing risk of death, later referral seemsappropriate in this group.Table 3.7: Results for 10,000 PSA samples. For each AVF referral strategy(preparation window and eGFR threshold) the average lifetime reduction from theoptimal policy (across each PSA setting) and the percentage of times when eachpolicy was found optimal is shown.PolicyPreparation window (months)3 6 9 12 15 18Average lifetime reduction (day)† 25.3 9.6 3.7 3.1 4.1 5.5Optimality percentage 3% 13% 26% 25% 19% 14%PolicyeGFR threshold (mL/min/1.73m2)At HD 10 15 20 25 30Average lifetime reduction (day)‡ 45.6 25.1 5.9 2.6 5.4 7.5Optimality percentage 1% 4% 29% 42% 17% 8%† from the optimal preparation windows policy (15 months)‡ from the optimal eGFR threshold policy (20 mL/min/1.73m2)42Chapter 4Patient Type Bayes-AdaptiveTreatment PlansThere has been a growing interest in the application of operations research methodsto treatment planning for different diseases. Due to the nature of chronic disease,patients are frequently seen by their specialist doctors, and their health status ismeasured periodically. The purpose of frequent follow-ups is three-fold: 1. to un-derstand the patient’s current health status, 2. to know how fast the disease isprogressing, 3. to (possibly) revise the treatment plan (treatment type or intensity)based on the information learned. The periodic follow-up, uncertainty in health pro-gression, and sequential decision making make Markov Decision Processes (MDPs)an important tool in clinical decision making of chronic diseases.Heterogeneity of patients with respect to disease progression and response tomedical interventions is an important characteristic of clinical decision making prob-lems. There is strong evidence in the clinical literature that patient characteristicssuch as age, gender, race, ethnicity, and culture play an important role in deter-mining patients’ responses to treatment and intervention outcomes including theirsurvivals. Therefore, patient-specific treatment plans are essential in achieving bet-ter patient outcomes at a lower cost.Patient heterogeneity is observed in several areas of clinical problems, for in-stance, adherence to screening procedures (e.g., adherence to colorectal cancer screen-ing [74, 75] and mammography screening [76]), adherence to medication (e.g., ad-herence to HIV treatment [35]), response to interventions (e.g., response to multi-ple sclerosis medications [77] and chemotherapy for prostate cancer patients [78]),dependence on medical devices (e.g., weekly usage of implantable cardioverter de-fibrillator devices [79]), and disease progression rate (e.g., chronic kidney diseaseprogression [11]).Although patient characteristics may inform the decision maker about a certainparameter of the clinical problem (e.g., whether the patient responds well to a certainmedication) and decrease uncertainty around that parameter, they provide partialinformation, and variability among patients of the same sub-population still exists.Due to the long treatment horizon for chronic diseases, the decision maker has thechance to incorporate the information obtained during the course of the disease tolearn about the patient disease progression profile and adjust the patient’s treatmentbased on the learned information.434.1. Related LiteratureIn this chapter, we develop a model that incorporates patient heterogeneity indisease progression when making clinical decisions and study structural propertiesof the model under certain modeling assumptions. Then, we apply this modelingframework to the case of AVF preparation timing problem introduced in Chapter 3and provide recommendations that consider patient heterogeneity in chronic kidneydisease progression when deciding if/when to begin the AVF preparation process.4.1 Related LiteratureIn this section, we review existing literature related to our research in two categories:1. methodological papers, 2. application papers.4.1.1 Methodological PapersWe often face uncertainty in parameters that define a decision model. In MDPs,parameter uncertainty can be present in different model components including re-wards and transition probability matrices. In clinical decision making models wherepatient heterogeneity is present, model parameters usually depend on the patienttype, e.g., the utility that a patient receives from a treatment or the efficacy oftreatments on slowing the progression of diseases may vary across the population.The uncertainty in parameters that form an MDP problem can be addressed intwo major ways, by solving the problem as a 1. Bayesian MDP, or a 2. RobustMDP. In the Bayesian setting, it is assumed that uncertain model parameters haveprior distributions. Using Bayes’ rule, a posterior distribution can be formed afterinformation is gained through the course of the sequential decision making process.In the robust setting, model parameters are chosen by nature from an uncertaintyset. When nature is modeled as an adversary, the problem can be formulated as arobust optimization problem [80].Satia and Lave [81] considered a robust setting, where at each decision epoch,the transition matrix row for each action is chosen from an uncertainty set by thenature. They considered max-min (a robust optimization framework) and max-maxcriteria in expected total reward maximization problems and presented -algorithmsthat solve the problem in a finite number of iterations. Goh et al. [82] considered arobust optimization framework in which the uncertainty set has a row-wise structureand provide bounds on the performance of such uncertain MDPs. They providean iterative algorithm for solving the problem under the row-wise structure andshow that a slight relaxation of the structure makes the problem computationallyintractable (NP-hard). They also applied their model to assess the cost-effectivenessof fecal immunochemical testing, a new screening method for colorectal cancer.Martin wrote a seminal book on Bayesian MDPs [83] and formulated a problemin which each row of the transition probability matrix for each action has some priordistribution. Using Bayes’ rule, a posterior distribution is obtained after observing444.1. Related Literaturestate transitions. Satia and Lave [81] considered conjugate beta distribution priorsand presented a decision tree solution algorithm that solved the problem for a givenprior distribution. Bayes-adaptive MDPs is an active research area in the computerscience community with a focus on solution algorithms, e.g., see [84–86].Bayesian MDPs can be cast as a partially observable Markov decision process(POMDP) [87–90]. Due to the high complexity of POMDP problems (see [91]for a complexity analysis), proving structural properties of the value function orthe optimal policy of POMDPs can facilitate obtaining efficient solution algorithmsand also provide managerial insights to problems. Lovejoy [92] provided sufficientconditions for monotonicity of POMDP value function in belief vectors. He alsoprovided conditions under which the set of beliefs where an action is optimal formsa convex set [93].4.1.2 Application PapersSeveral research papers have incorporated patient heterogeneity in disease progres-sion in their decision model. Lavieri et al. [78] studied the decision of when toswitch from chemotherapy to radiation therapy for prostate cancer patients basedon predictions of the time when the prostate specific antigen (PSA) level of a patientreaches its lowest point. They identified clusters of patients with respect to PSAprogression parameters and formed a prior distribution on the cluster each patientbelongs to, which was then updated after observing PSA levels over time. Helm etal. [94] considered the question of when to monitor a glaucoma patient and developeda model to predict the likelihood of glaucoma progression, where using a Kalmanfilter, a patient’s disease progression parameters are learned sequentially throughmedical tests combined with population information. Negoescu et al. [95] addressedtreatment planning for patients with chronic diseases, where a patient was either aresponder or non-responder to some medication. They considered dosage between0% and 100% as possible actions in each belief state. By continuously monitoringthe health of the patient as well as observing critical health events, the likelihoodof being a responder was then updated and the treatment plan revised accordingly.POMDPs are also applicable where due to observation errors, the state of a sys-tem is partially observed. For instance, a patient’s health (e.g., whether the patienthas cancer or not) may not be perfectly identifiable due to diagnostic errors. Herewe briefly survey such application of POMDPs in clinical decision making prob-lems. Zhang et al. [39] addressed the prostate biopsy referral decision in a POMDPframework. They used PSA levels to update the belief on whether a patient hasprostate cancer, based on which, prostate biopsy referral decision was made. Ayeret al. [96] used a POMDP model to provide personalized mammography screeningpolicy based on a patient’s screening history, where the belief on whether the patienthas breast cancer was updated based on self-detection and mammography screening.Ayer et al. [76] addressed the role of patient adherence to mammography recommen-dations, and heterogeneity thereof, on optimal breast cancer screening policies in a454.1. Related LiteraturePOMDP framework. Unlike the above papers, we apply the POMDP frameworkto incorporate the patient heterogeneity in disease progression in clinical decisionmaking problems and develop a model that learns the patient type partially throughobserving health transitions.4.1.3 Contributions & Chapter StructureIn this work, we formulate and analyze the problem of designing ongoing treatmentplans for a population whose patients’ response to treatments or disease progressionin the absence of treatment vary from patient to patient in a way that 1) we can rec-ognize distinct types of patients, and 2) each patient’s type can be learned partiallyby monitoring her health over time. We formulate the problem as a two-dimensionalstate-space POMDP, where the state consists of the patient health and type. In ourmodel, we assume that the patient health is observed perfectly, whereas the patienttype is revealed only partially through observing health transitions.In Section 4.2.2, we provide sufficient conditions under which the value functionof an MDP with state-space Rn is monotone in state. This result generalizes theknown result in the literature for one dimensional state spaces (e.g., see Proposition4.7.3 in [97]). In Section 4.2.3, we provide conditions for having monotone optimalpolicies for optimal stopping timing problems with state-space Rn. We then applythese result to Bayes adaptive treatment plan design problems defined and analyzedin Section 4.3. Finally, in Section 4.4, we apply the results of Section 4.3 to theAVF preparation timing problem introduced in Chapter 3.We contribute to the OR/MS literature by providing results on the structureof multi-dimensional state-space MDPs. We also develop a framework for incor-porating the heterogeneity of patient disease progression in an MDP and providestructural properties of the associated POMDP. This framework enables cliniciansto dynamically adjust a patient’s ongoing treatment plan based on the patient healthand the belief about the patient’s disease progression type. We also contribute tothe clinical literature on vascular access planning for patients with chronic kidneydisease by finding optimal AVF preparation timing policies that consider a patient’srate of disease progression in addition to the kidney health state.Our framework bears similarities and differences to [95]. The authors assumedtwo types of patients, responders and non-responders, and used rewards gained un-der a certain medication as well as critical life events to partially learn the patienttype. Our model differs from [95] since in our work, treatment decisions may de-pend on a patient’s current health state in addition to our belief about the patienttype. Similar to [92], we provide structural properties for POMDP problems, witha distinction that in our setting, the two-dimensional state-space consists of cor-related observable (the health state) and partially observable (the patient type)components.464.2. Monotonicity Results4.2 Monotonicity ResultsIn this section, we provide monotonicity results for MDPs with state-space Rn. Wefirst define notation that will be useful in the discussion that follows.4.2.1 Notation T : planning horizon. Decisions are made for time periods t = 1, . . . , T . A: finite set of actions available in periods t = 1, . . . , T xt: random vector in Rn denoting the state of the system in period t x˜at (x): random vector denoting the state of the system in period t + 1 whenthe system is at state x in period t and and action a is taken rat (x) : Rn → R: immediate reward received in period t ≤ T , when action a istaken and the system is at state x R(x) : Rn → R: terminal reward received at t = T + 1 when the system is atstate x4.2.2 Monotone Value FunctionsWe consider discounted expected total reward maximization MDPs characterized byaction space A, rewards rat (x) and R(x), and state transitions indicated by x˜at (x).Let vt(x) : Rn → R be the period t value function. Then, by the principle ofoptimality vt(x) satisfies:vt(x) ={R(x), t = T + 1,maxa∈A{rat (x) + βEvt+1(x˜a(x))}, o.w. ,where β is the discount factor.We use the usual stochastic order of random vectors defined below (see [98]). Inthe following definition, we use upper sets in Rn defined as follows. A set U ∈ Rnis called upper if y ∈ U whenever y ≥ x and x ∈ U . Note that the we say x ≤ x′,whenever xi ≤ x′i for all i, i.e., we compare vectors component-wise.Definition 4.1 (Usual stochastic order). Let X and Y be random vectors. We sayX is smaller than Y in the usual stochastic order, denoted by X ≤st Y , wheneverwe have P[X ∈ U ] ≤ P[Y ∈ U ] for all upper sets U .Shaked and Shanthikumar [98] provide the following interpretation of the usualstochastic order of random vectors: X is said to be smaller than Y in the usual474.2. Monotonicity Resultsstochastic order when X is less likely than Y to take on large values. Alterna-tively, [98] shows that X ≤st Y whenever we have Ef(X) ≤ Ef(Y ) for all boundedincreasing functions f .Puterman [97] (Proposition 4.7.3), provides sufficient condition for the mono-tonicity of an MDP value function for one dimensional state spaces. We extend theresult to n-dimensional spaces in Proposition 4.1. All of the proofs for the analyticalresults are given in Appendix B.Proposition 4.1. (Monotonicity of Value Function)vt(x) increases with x for all t, if we have:(a) rat (x) increases with x for all a ∈ A for all t,(b) R(x) increases with x,(c) x˜at (x) ≤st x˜at (x′) for all x ≤ x′ and a ∈ A.Assumptions (a,b) state that the immediate reward of all actions and the termi-nal reward increase with x, respectively, and assumption (c) states that the systemis more likely to transitions to higher states in period t + 1, when the system is athigher states in period t. The underlying transition probability structure, x˜a(x),depends on the context. We discuss the transition probability structure for Bayesadaptive treatment design problems in Section 4.3.4.2.3 Monotone Optimal PoliciesIn finite horizon optimal stopping problems, for each t ≤ T we have a choice betweentwo actions, continue and stop. When stop is chosen, a state-dependent lump-sumreward, Rt(x) is received. If on the other hand continue is chosen, a state-dependentimmediate reward rt(x) is received, the system evolves, and we face a similar decisionin the next period (the choice between stop and continue). At t = T + 1, a state-dependent terminal reward RT+1(x) is received.Since the system evolution matters only under the continue action, each optimalstopping MDP can be characterized by the 3-tuple (rt(x), Rt(x), x˜(x)), where x˜(x)is a random vector denoting the state of the system in period t+ 1 when the systemis at state x in period t, and action continue is taken.Define the one-step benefit function δt(x) as the difference in the expected rewardbetween waiting in period t and stopping in period t + 1, and stopping in period twhen the state is x, i.e., let δt(x) := rt(x) + βERt+1(x˜t(x))−Rt(x). Oh (2012) [99]showed that the optimal policy is monotone in x when δt(x) is monotone (increasingor decreasing) in x and x˜t(x) ≤st x˜t(x′) for all x ≤ x′. Here we state Proposition2.5. in [99].Proposition 4.2.If for all t we have:(a) δt(x) increases with x,(b) x˜t(x) increases with x in the usual stochastic order.484.3. Bayes-adaptive Treatment Plansthen, it is optimal to stop at state x, whenever stopping is optimal at state x′, forany x ≤ x′.Consider two optimal stopping time problems indexed by i = 1, 2, and charac-terized by the 3-tuple (rit(x), Rit(x), x˜it(x)). Also, let δit(x) be the one-step benefitfunction for problem i. We show that if the one-step benefit function is alwayssmaller for problem 1 and the state dynamics is stochastically the same for bothproblems, then it is optimal to stop in problem 1 whenever stopping is optimal inproblem 2. We use this result to provide comparative statics on the optimal policyof an optimal stopping time problem defined in Section 4.4.Proposition 4.3.If for all t we have:(a) δ1t (x) ≤ δ2t (x) for any x,(b) x˜2t (x) has the same distribution as x˜1t (x) for any x.Then, stopping is optimal in state x in problem 1 whenever it is optimal to stop instate x in problem 2.Similar to δt(x), define σt(x) as the difference between the immediate rewardof waiting and the lump-sum reward of stopping in period t, i.e., σt(x) := rt(x) −Rt(x). In the following proposition, we show that if x˜(x) increases with xi, the ithcomponent of x, in the usual stochastic order, the value function is increasing in x,and σt(x) is increasing in xi, then the optimal policy is monotone in xi. Let x−idenote all components of vector x except for the ith component.Proposition 4.4.If for all t we have:(a) x˜t(x) increases with xi in the usual stochastic order,(b) vt+1(x) increases with x,(c) σt(x) increases with xi.Then, it is optimal to stop at x, whenever stopping is optimal at state x′, wherexi ≤ x′i, and x′−i = x−i.Note that Proposition 4.1 provides sufficient conditions for assumption (b) tohold.4.3 Bayes-adaptive Treatment Plans4.3.1 Problem StatementIn this section, we formulate and analyze the problem of designing patient typeBayes-adaptive treatment plans defined as follows. We consider designing treatmentplans when treatment-dependent patient outcomes vary across the population in away that 1) we can categorize patients into distinct types, 2) we cannot perfectly494.3. Bayes-adaptive Treatment Plansidentify a patient’s type a priori, and 3) the patient type can be observed partiallyby monitoring the patient health over time. One example of such case is wherethere are two types of patients in the population with respect to the response to acertain medication, good and bad responders, and the type of the patient cannotbe identified a priori; nevertheless, whether the patient is a good responder to themedication or not can be learned (partially) by monitoring the patient’s responseto the medication over time. Another example is where there are two differenttypes of “natural history” (i.e., disease progression in the absence of treatment),and the decision on if/when to administer a certain treatment plan depends on thetrue underlying disease progression process. We assume a Bayesian setting in whichwe start with some prior belief about the patient type and update our belief byobserving the patient health over time using Bayes’ rule, hence the name “patienttype Bayes-adaptive treatment plans”.In the following section, we formulate the problem as a MDP with a two-dimensional state-space, where the state consists of the patient health and the beliefabout the patient type (“better” or “worse” disease progression type).4.3.2 NotationWe first define notation that will be useful in the discussion that follows. For easeof notation, in this section we only consider stationary MDPs. The result can beeasily extended to non-stationary MDPs. C = {w, b}: set of two patient types. Types can represent differing stochas-tic progressions of disease (fast or slow) or response to medical interventions(responder or non-responder). Below, we will assume an ordering of the typesso that type b, the better type, represents slower disease progression or betterresponse to medical intervention, e.g., in terms of on-going rewards, comparedto type w, the worse type. γam: random-variable denoting the per-period decrement of the patient healthwhen patient type is m, and action a is taken. We let fam denote the pdf (pmf)of γam. ram(e): immediate reward received in period t ≤ T , when action a is taken,and patient health and type are e and m, respectively. Rm(e): terminal reward received at t = T + 1, when patient health state andtype are e and m, respectively.4.3.3 MDP Formulation States: the state of the system at time t is comprised of the patient type, bor w, and health state. We assume that the health state is observed perfectlywhile patient type is observed only partially through health transitions, and504.3. Bayes-adaptive Treatment Planswe assume that the health state is one-dimensional (i.e., a scalar). We denotethe system state at t by xt = [et, pt], where et represents health state, and ptrepresents our belief (i.e., the probability) that the patient is of type b. Wewill assume an ordering of the states so that higher states represent betterhealth conditions. Transition dynamics: We assume that the health state of patient type mevolves according toet+1,m = et,m − γam.Let x˜a(xt) denote the state in period t + 1 when in period t, action a ischosen, and the state is xt. Then, we have et+1 = et − γapt , where γapt isdefined as a random variable with pdf (pmf) given by fpt = ptfab + (1− pt)faw.After observing et+1, we update our patient type belief using Bayes’ rule bypt+1 := Ba(et+1 − et, pt), where Ba(d, p) is defined by:Ba(d, p) := pfab (d)pfab (d) + (1− p)faw(d)=pfab (d)fp(d). (4.1)Therefore, x˜ satisfiesx˜a(e, p) =[e− γap ,Ba(γap , p)]. Rewards: At t ≤ T , an immediate reward ram(e) is received when action a istaken, and at t = T +1, a terminal reward Rm(e) is received, when the patienthealth and type are m and e, respectively. Therefore, for state x = [e, p] wehave:ra(e, p) = prab (e) + (1− p)raw(e),R(e, p) = pRb(e) + (1− p)Rw(e). Optimality condition: let vt(s, p) be the period t value function. Then, thevalue function satisfies:vt(e, p) ={maxa∈A{ra(e, p) + βEvt+1(x˜a(e, p))}t ≤ TR(e, p) t = T + 1.4.3.4 Monotone Value FunctionsWe use the monotone likelihood ratio (MLR) defined below (see [98]):514.3. Bayes-adaptive Treatment PlansDefinition 4.2 (MLR Order). Let X and Y be random variables with pdf’s (pmf’s)f and g, respectively. Then, we say X is smaller than Y in the monotone likelihoodratio (MLR) order, denoted by X ≤r Y , whenever g(z)/f(z) increases in z (hereb/0 is taken to be equal to ∞ whenever b > 0).Intuitively, X is said to be smaller than Y in the MLR order, when the likelihoodratio of taking large values to small values is higher for Y than X. Note that theMLR order implies the usual stochastic order of random variables [90].In the following, we provide conditions under which the Bayesian update ismonotone. Since we assume disease progression is slower for patient type b, a smalldecline of health can be a signal for slower disease progression, i.e., patient typeb. The lemma compares the posterior belief for different values of health-declineand prior beliefs. More specifically, it states that when health decline is smaller forpatient type b in the MLR order, the posterior belief about the patient being of typeb is higher when our prior belief is higher, or when we observe smaller decline in thethe patient health.Lemma 4.1 (Monotonicity of Bayesian Operator).If γab ≤r γaw for action a, then Ba(d, p) increases with p and decreases with d.Intuitively, γab ≤r γaw means that the likelihood ratio of observing higher healthdeclines to lower health declines is higher for patient type w (the worse progressortype).The following key result compares the random vector x˜a(x) for different valuesof x. It states that when the health decline is smaller for patient type b in the MLRorder, transitions to higher health states and patient type beliefs are more likelywhen current health state and patient type belief are higher.Lemma 4.2.If for action a we have γab ≤r γaw, then for any x ≤ x′ we have:x˜a(x) ≤st x˜a(x′).In the following proposition, we show that the total expected reward is higherif the patient is healthier (i.e., health state is higher) or when our belief about thepatient being of the better type (type b) is higher. We use Proposition 4.1 andLemma 4.2 to prove the result. For ease of notation, we order patient types in setC such that w < b.Proposition 4.5. (Monotonicity of Value Function)vt(e, p) increases with e and p for any t, if we have:(a) For any a ∈ A, ram(e) increases with e and is higher for m = b,(b) Rm(e) increases with e and is higher for m = b,(c) γab ≤r γaw for any a ∈ A.524.3. Bayes-adaptive Treatment PlansAssumptions (a,b) state that the immediate reward for any action and the ter-minal reward are higher when the patient health is higher or her type is better.Assumption (c) states that the health decline is smaller for patient type b (com-pared to patient type w) in the MLR order.4.3.5 Monotone Policies in Optimal Stopping ProblemsIn clinical optimal stopping problems, the stop action can represent any medicalintervention such as performing surgery (e.g., organ transplant) or starting a medi-cation regiment (e.g., HIV treatment). Below, we will apply our modeling frameworkto the problem of timing AVF preparation for patient with progressive chronic kid-ney disease (defined in Chapter 3), where in each period, we face the decision ofwhether the patient should start the AVF preparation process, or we should waitand reconsider the decision in the subsequent period.We simplify our notation for optimal stopping problems as follow. For each t, letrm(e), Rm(e) denote the immediate reward of continuing and the terminal reward ofstopping when patient health state and type are e and m, respectively. Also, sincethe system evolution matters only under the continue action, we let γm representthe health decline for patient type m under the continue action.In what follows, define σm(e) := rm(e) − Rm(e). We can explain σm(e) as thedifference between the immediate reward of waiting for one period and the lump-sum reward of the intervention, when patient health state and type are e and m,respectively. In the following proposition, we show that if σm(e) increases with mand e, i.e., when the incremental benefit of intervention is higher for sicker patients(patients with lower health) and for patients with faster disease progression (pa-tients type w), and conditions of Proposition 4.5 hold, then intervention is optimalwhenever it is optimal for healthier patients or when our belief that the patient is aslow progressor is higher.Proposition 4.6.If we have:(a) For any a ∈ A, ram(e) increases with e and is higher for m = b,(b) Rm(e) increases with e and is higher for m = b,(c) γb ≤r γw,(d) σm(e) increases with e and is higher for m = b.then, it is optimal to stop at (e, p), whenever stopping is optimal at state (e′, p′) forany e ≤ e′ and p ≤ p′.Note that assumptions (a-c) are the same as assumptions in Proposition 4.5 andused to obtain the monotonicity of the value function. We use Propositions 4.4 and4.5 to prove Proposition 4.6 (in Appendix B).Next, define δm(e) := rm(e) + βERm(e− γm)−Rm(e). We can explain δm(e) asthe incremental benefit of waiting for one period and intervening in the subsequentperiod over intervening in the current period, when patient health state and type are534.4. Optimal Timing of AVF Preparatione and m, respectively. In the following proposition, we show that if δm(e) increaseswith m and e, i.e., when the incremental benefit of waiting for one period andintervening in the subsequent period over intervening in the current period is higherfor sicker patients (patients with lower health) and for patients with faster diseaseprogression (patients type w), and the health decline is smaller for patient type bin the MLR order, then intervention is optimal whenever it is optimal for healthierpatients or when our belief that the patient is a slow progressor is higher. We useProposition 4.2 and Lemma 4.2 to prove Proposition 4.7.Proposition 4.7.If we have:(a) δm(e) increases with e and is higher for m = b,(b) γah ≤r γal for any a ∈ A.Then, it is optimal to stop at (e, p), whenever stopping is optimal at state (e′, p′)for any e ≤ e′ and p ≤ p′.It is intuitive that δm(e) increases with e since sicker patients, in comparisonwith healthier patients, gain more from medical interventions. For instance, thebenefit of taking pain relief drugs is more pronounced for patients with higher painlevels. On the other hand, whether or not δm(e) increases with m is context-specificand not something we expect to intuitively hold in all contexts and not somethingwe necessarily expect to hold. The following provides alternative conditions basedon model primitives, to check if we can expect δm(e) to increase with m. We showthat when patient types differ only in their disease progression rate (not in rewardsthey receive at different states), the terminal reward increases in the patient health,and the health decline is smaller for patient type b in the MLR order, then δm(e)increases with m.Proposition 4.8.If we have:(a) Rw(e) = Rb(e) and rw(e) = rb(e) for all e,(b) Rb(e) increases with e,(c) γb ≤st γw.Then, we have δw(e) ≤ δb(e).4.4 Optimal Timing of AVF PreparationIn this section, we revisit the AVF preparation timing question of Chapter 3. Wemodel the AVF preparation timing problem as an optimal stopping MDP problem.We incorporate the heterogeneity of patient disease progression in our model usingthe framework of Section 4.3.544.4. Optimal Timing of AVF Preparation4.4.1 Timing of AVF PreparationThe preferred vascular access for HD is an AVF [3] due to greater longevity andlower complication rates; however, it may take several months and more than oneprocedure to establish a usable AVF [4,5]. If the AVF is created too late, it may notmature in time, and a central venous catheter (CVC) may be used; however, CVCsare associated with an increased risk of morbidity and mortality [6–9]. On the otherhand, creating an AVF too early is undesirable due to a small increase in risk ofcomplications and wasting the limited lifetime of an AVF before HD is needed [10].To avoid the consequences of having a functional AVF earlier or later than HD starttime, it is ideal for the patient to have an AVF that becomes functional right at thetime of HD start. Nevertheless, due to intrinsic uncertainties in AVF preparationlead-time (the time from the first AVF surgery until an functional AVF becomesavailable) as well as the time of HD start, the ideal case is hardly achievable.Figure 4.1 depicts the costs associated with deviating from the ideal case. Morespecifically, it shows the differential in life-expectancy between the ideal case andthe case when AVF becomes ready earlier or later than HD start time. When AVFbecomes ready after HD starts (positive values on the x-axis), the patient loses anaverage of 1.6 of expected life-month for each 6 months of lateness, whereas anaverage of .88 of expected life-month is lost for each 6 months AVF is ready earlierthan HD start time (negative values on the x-axis). The loss of life-expectancy forearly AVFs are associated with the waste of AVF’s limited lifetime before HD starts.The loss of life-expectancy for late AVFs is due to lower patient survival on a CVC(until an AVF becomes functional).Estimated glomerular filtration rate (eGFR) is often used as the primary measureof kidney health. Nephrologists monitor eGFR progression periodically to decidewhen to initiate HD as well as when to start AVF preparation. HD is often initiatedwhen a patient’s eGFR falls below 10 mL/min/1.73m2 [43]. Due to AVF prepara-tion lead-time, CKD patients should start the AVF preparation in advance of HDstart time, i.e., when eGFR is well above the HD start threshold. The CanadianSociety of Nephrology (CSN) guidelines suggest starting the AVF preparation at aneGFR of 15-20 mL/min/1.73m2 [16]. We develop a data-driven, dynamic program-ming approach to provide recommendations regarding if/when to begin the AVFpreparation.The eGFR value at which the AVF preparation starts as well as the rate atwhich eGFR deteriorates over time determine (stochastically) how much earlier orlater than HD start time an AVF becomes available, and in turn, affect a patient’slife-expectancy (Figure 4.1). The rates at which eGFR progresses over time variesconsiderably across the population. For example, O’hare et al. [11] identified fourdistinct types of patients with respect to eGFR progression rates (Table 4.1). There-fore, it is important to take into account the progression heterogeneity when makingAVF preparation timing decisions. In Chapter 3, we used a Monte-Carlo simulationmodel to find the best time to start the AVF preparation. In this section, we model554.4. Optimal Timing of AVF Preparationthe problem as an optimal stopping MDP, where in each period, we make a decisionwhether to start the AVF preparation (the stop action) or wait another period andreconsider the decision in the subsequent period. We use the framework of Section4.3 to factor in the patient heterogeneity in disease progression.Table 4.1: Heterogeneity of eGFR progression for chronic kidney disease patients.Table includes mean eGFR decline for different types of eGFR progression as wellas their prevalence in the population [11].EGFR progression type mean eGFR decline† PrevalencePersistently low eGFR levels 7.7 63%Progressive eGFR loss 16.3 25%Accelerated eGFR loss 32.3 9%Catastrophic eGFR loss 50.7 3%†: mL/min/1.73 m2 per year.4.4.2 MDP Formulation Patient Types: we associate type b with patients that have stochasticallyslower decline of eGFR (compared to type w). States: the state of the system in period t is comprised of pt, our belief aboutpatient type, and et, the kidney health state measured by the eGFR value. Weassume that HD starts when the eGFR value falls below ed, a certain eGFRthreshold. The time at which patient transitions to a state below the HDthreshold is used as a proxy for HD start time. Transition dynamics: each month eGFR declines according to a randomvariable γm, for patient type m, i.e., we have:et+1,m = et,m − γm.After observing a decline d in the eGFR, we update our belief about patienttype according to Eq. 4.1.Note that the constant expected eGFR decline in our eGFR progression modelis consistent with other studies that model eGFR decline using linear regression[43,60,71,72]. Decision epochs and actions: each month, an eGFR reading is taken fromthe patient, and a decision whether to start AVF preparation or wait until thenext period is made, provided that HD and AVF preparation processes have564.4. Optimal Timing of AVF Preparation y = -0.1468xR² = 0.986y = 0.2699xR² = 0.9891012345678910-30 -24 -18 -12 -6 0 6 12 18 24 30Expected Life Month LossAVF Ready – Dialysis Start (months)Early Preparation Late PreparationFigure 4.1: Earliness/lateness cost of AVF ready time. Plot shows a patient’s ex-pected life month loss to imperfect AVF ready time. On the x-axis, we have thedifference between the time AVF is ready and HD start time. On the y-axis, wecalculated the differential in life-expectancy between the ideal case (when AVF isready at the time of HD start) and the case when AVF is ready earlier or later thanHD start time for different values of AVF Ready - Dialysis Start. Life-expectancyfor different scenarios are calculated using Monte-Carlo simulation with parametersgiven in Table 3.1.not started yet. When HD starts (i.e., when the eGFR falls below the HDthreshold), we start the AVF preparation if it has not already started. Costs: when we start the AVF preparation process at state (e, p), a lump-sumcost d(e, p) is incurred, which is associated with AVF earliness/lateness. Lettd and ta denote time instants at which dialysis starts, and a functional AVFbecomes available, respectively. Then, we assume that the cost under thisscenario equals to c(ta − td) for a real-valued earliness/lateness cost functionc. We fit a piece-wise linear function to data-points in Figure 4.1 to createfunction c. Let dm(e) be the expected earliness/lateness cost when the AVFpreparation starts at eGFR e and the patient type is m. Also, let Tme denotethe time until HD starts when eGFR is at e, and the patient type is m.Note that Tme is endogenous to the problem parameters. Assume that at the574.4. Optimal Timing of AVF Preparationbeginning of month t, eGFR is at e. Then, Tme is defined by the following:Tme := min{τ :t+τ∑i=tγim ≥ e− ed}where γim denotes the decline of eGFR in month i.Let L denote the AVF preparation lead-time. We assume that the AVF prepa-ration is independent of patient type. Then, we have:dm(e) = EL,Tme [c(L− Tme )].Therefore, we have d(e, p) := pdb(e)+(1−p)dw(e). For e ≤ ed, we have Tme = 0by definition. Therefore, dm(e) = Ec(L), and thus we have p(e, p) = Ec(L).We assume no immediate cost is incurred when we take the wait action, andactions do not affect the time of HD need (i.e., AVF preparation does notaffect progression of chronic kidney disease). Optimality condition: let v(e, p) be the value function. Then, the valuefunction satisfies:v(e, p) ={Ec(L) e ≤ edmin[d(e, p),Ev(e− γp,B(γp, p))]o.w.In the following proposition, we show that under certain conditions, the optimalpolicy for starting the AVF preparation is of threshold type, i.e., it is optimal tostart the AVF preparation, whenever starting is optimal for higher eGFR values(better kidney health) or for higher patient type beliefs.Proposition 4.9. Optimal Timing of AVF PreparationIf we have:(a) γb ≤r γw,(b) Function c is convex.then, it is optimal to start the AVF preparation at state (e, p), whenever starting isoptimal at state (e′, p′), for any e ≤ e′ and p ≤ p′.Assumption (a) states that the likelihood ratio of observing a higher eGFRdecline to a lower eGFR decline is higher for patient type w. Note that the costfunction in Figure 4.1 is approximately linear on both sides of zero. Therefore, wehave c(x) = h[−x]+ + b[x]+, where [x]+ = max[0, x]. Note that c is convex for allh, b ≥ 0, thus satisfying assumption (b) of Proposition 4.9 .In the following proposition, we compare the optimal AVF preparation policyunder different AVF preparation lead-time. We show that when the conditions ofProposition 4.9 are met, it is optimal to start the AVF preparation at any state,584.4. Optimal Timing of AVF Preparationwhenever starting the AVF preparation is optimal under longer preparation lead-times (in the usual stochastic order).Proposition 4.10. Comparative StaticsConsider two problem instances indexed by 1, 2, each satisfying assumptions (a)and (b) of Proposition 4.9. Assume that problems only differ in AVF preparationlead-time with L1 ≤st L2. Then, it is optimal to start the AVF preparation at anystate (s, p) in problem 1, whenever starting is optimal in problem 2. 0.000.020.040.060.080.100.120.140.160.181 2 3 4 5 6 7 8 9 10 11 12 13 14 15Probabilty AVF Preparation Time (months) Probability density function of AVF preparation time Figure 4.2: Empirical probability mass function of the AVF preparation time, gen-erated using Monte-Carlo simulation. We considered a series of AVF surgeries per-formed one after the other until one surgery is successful, with at most 4 of AVFcreations. The values for AVF surgery success probability and maturation timesand the sources used for each parameter are given in Table 3.1.4.4.3 Numerical ResultsTo demonstrate the results of Proposition 4.9, we performed a numerical study. Theparameters used for the study are as follows. We used the earliness/lateness costfunction depicted in Figure 4.1 and chose c(x) = h[−x]++b[x]+. We set h = 26.5 andb = 48 and assume c(x) and x are measured in days and months, respectively. Weused the patient types defined in [11] (see Table 4.1) and let types b and w represent‘persistently low eGFR’ and ‘progressive eGFR loss’. These two types represented88% of the CKD population in the study by [11]. Note that the two other patienttypes (‘accelerated eGFR loss’ and ‘catastrophic eGFR loss’ patients) can be easily594.4. Optimal Timing of AVF Preparation 00.20.40.60.8122 21 20 19 18 17 16 15 14Belief that Patient is Slow ProgressoreGFR (mL/min/1.73 m2 )Waiting is optimalCommencing the AVF preparation is optimalFigure 4.3: Optimal policy for AVF preparation timing. The optimal policy fordifferent values of eGFR as well as the belief that the patient is of type b, i.e., theslow progressor, is shown. The states at which the optimal policy is ‘start the AVFpreparation’ and ‘wait’ are depreciated in black and yellow, respectively. As figuresuggest, the optimal policy is monotone in both state dimensions.distinguished from the types we consider here since they have noticeably highereGFR decline rates. We modeled eGFR monthly decrement for patient types b andw as normally distributed random variables, i.e., we assumed γm ∼ N (µm, σ2m).We used the average monthly eGFR decline given for patient types b and w in [11]and set [µw, µb] = [1.4, .64]. To calculate σm, we performed primary data analysison the eGFR trajectories of 1048 patients treated at the multidisciplinary kidneyclinic at Vancouver General Hospital. We found that [σw, σb] = [1.5, 1.4]. Weassumed that HD starts when eGFR falls below 10 mL/min/1.73m2, i.e., ed = 10[43]. We calculated the probability mass function of AVF preparation lead-timeusing parameters given in Table 3.1 (see Figure 4.4.2 for more details).To solve the problem numerically, we discretize the state-space with a uniformgrid, in which eGFR values are grouped in buckets of 1 mL/min/1.73m2, and beliefsare grouped in buckets of 0.01. We have N (µb, σ2b ) ≤r N (µw, σ2w) whenever σb = σwand µb ≤ µw (see [100]). Although we have µb ≤ µw, the condition that σb = σwdoes not hold; nevertheless, we can show that assumption (a) of Proposition 4.9 em-pirically holds for our discretize state-space. Finally, assumption (b) of Proposition4.9 holds since for all values of parameters h, b ≥ 0, function c(x) = h[−x]+ + b[x]+is convex. Therefore by Proposition 4.9, the optimal policy for starting the AVFpreparation is of threshold type, i.e., it is optimal to start the AVF preparation,whenever starting is optimal for higher eGFR values and higher beliefs (that the604.5. Conclusionpatient is a slow progressor).Figure 4.3 depicts the optimal policy for AVF preparation timing for differenteGFR values and patient type beliefs. As we expect, the optimal policy is monotonein both state dimensions. When the patient type is a slow progressor (fast progres-sor) with certainty, the optimal eGFR threshold beyond which starting the AVFpreparation is optimal is 15 mL/min/1.73m2 (20 mL/min/1.73m2). This is consis-tent with the Canadian Society of Nephrology (CSN) guidelines which suggests start-ing the AVF preparation at an eGFR of between 15 and 20 mL/min/1.73m2 [16].Our results sharpen the guidelines by matching the lower bound of 15 with pa-tients classified as slow progressors (mean eGFR decline of .64 mL/min/1.73m2 permonth) and the upper bound of 20 with the fast progressors (mean eGFR declineof 1.4 mL/min/1.73m2 per month).4.5 ConclusionIn this chapter, we analyzed the problem of designing ongoing treatment plans for aheterogeneous population with respect to disease progression and response to med-ical interventions. We created a model that learns the patient type by monitoringthe patient health over time and updates a patient’s treatment plan according tothe gathered information. We formulated the problem as a two-dimensional state-space POMDP and provided structural properties of the value-function, as well asthe optimal policy for the special case of optimal stopping timing problems. Thisframework can be extended to other contexts where an MPD is applicable andtransition parameters can be learned by observing state transitions.We also applied the framework to the AVF preparation timing question posed inChapter 3 by considering two types of patients, patients with slow and fast eGFRprogression. We showed that under data-driven assumptions, the optimal AVFpreparation timing policy is monotone in a patient’s current eGFR as well as ourbelief that the patient is a slow progressor.Although we considered two patient types, our results can be extended to caseswith multiple patient types. We also believe that the framework can be appliedto other chronic diseases where heterogeneity in disease progression is present. Weconsidered a special structure for the state dynamics of MDPs where random vari-ables representing the difference between the states in periods t and t + 1 for eachaction do not depend on the state in period t. It would be interesting to extend theresult to a more general setting for state transitions. Finally, we only investigatedthe monotonicity of optimal policies for optimal stopping timing problems. As adirection for future research, one might consider extending the results to the casewith a more general action space.61Chapter 5Conclusions, Extensions andFurther ApplicationsThe research in this dissertation focused on the application of stochastic optimiza-tion models to vascular access planning for patients with chronic kidney disease.In this section, we provide a review of the problem, analytical models developedto address the research questions, and the main results. Furthermore, we discusspossible extensions and avenues for further research.Hemodialysis is the most common form of renal replacement therapy. There aretwo primary types of vascular accesses used for HD, arteriovenous fistula (AVF), andcentral venous catheter (CVC). An AVF, which is created via a surgical procedure, isoften considered the gold standard for delivering HD due to better patient survivaland higher quality of life. However, it may take several months and more thanone procedure to establish a functional AVF, whereas a CVC can be inserted via asimple procedure and used immediately after placement. In this thesis, we addressthe question of whether and when to perform AVF surgery on patients with CKDwith the aim of finding individualized policies that optimize patient outcomes. Thisquestion is relevant in two stages of the disease, before HD commences and after.In Chapter 2, we focused on vascular access planing for patients already on HD.Using AVF for HD not only brings better survival, but also has a slightly higherquality of life for the patient, in comparison with HD using a CVC. Nevertheless,the process of AVF creation has some disutility associated with it, which can beattributed to the surgery and post-surgery inconveniences, complications or costs.Therefore, it is not clear under what conditions an HD dependent patient shouldundergo the AVF creation surgery. We developed a continuous-time dynamic pro-gramming model to find optimal policies that maximize a patient’s life expectancyand Quality-adjusted life expectancy (QALE).We analytically proved that delaying AVF surgery stochastically decreases apatient’s lifetime. As a result, the policy of “use the next AVF (opportunity) assoon as a patient starts HD or when the one being used fails” maximizes a patient’ssurvival probability. We also proved that the optimal policy to maximize a patient’sQALE is of a threshold type: there is an HD duration threshold before whichimmediate surgery is the optimal choice, while after that time, CVC is the optimalvascular access choice for the remainder of the patient’s lifetime. This thresholddepends on the number of past AVF maturation failures.62Chapter 5. Conclusions, Extensions and Further ApplicationsThe AVF creation disutility plays an essential role in determining the optimalpolicy when maximizing QALE. Since patients may feel differently about the disu-tility of AVF surgery, and also because it is not an easy parameter to elicit froma patient, our model provides an alternative way to make the optimal AVF timingdecision. We showed that the decision of whether to perform an AVF surgery or notcan be determined solely by comparing the patient’s AVF creation disutility with aboundary value reflecting the prospective additional QALE for the patient, whichwe refer to as the critical disutility. Thus, a nephrologist can inform the patient ofthe benefits and inconveniences of undergoing the AVF surgery, and then, they cancollectively decide whether to do the surgery or not. Even if a rough estimate ofthe patient’s disutility for AVF surgery indicates that it is clearly below or abovethe critical disutility, then it will be clear that the patient should or should not,respectively, undergo an AVF surgery.We also found that the possibility of receiving a kidney transplant adds newcomplexities to the model and optimal policy structure. Although the optimal policyunder the total lifetime remains the same, the result on QALE metric (optimality ofthreshold policies) does not necessarily extend, even when the time of transplant isknown with certainty. Nevertheless, we provided a theorem which proves that underadditional assumptions (which are supported by data), threshold policies remainoptimal. It would be interesting to investigate how the possibility of cadavericdonations and random wait times affect the vascular access planning for ESRDpatients under the QALE metric. We did not consider costs in our model andfocused on patient outcomes. It would be interesting to perform a cost-effectivenessanalysis and investigate whether suggested policies are cost-effective or not.Our framework and analytical results may also be relevant to operational ques-tions outside of health care, particularly in the area of machine maintenance andequipment reliability. For example, consider a machine with a vital component. Ifthe component breaks down, it may be replaced with a cheap, available spare. Addi-tionally, one may order a more expensive, higher-quality component, which involvesa lead time for delivery. This is analogous to deciding whether and when to refer apatient for an AVF versus letting them continue to receive HD through a CVC. AnAVF provides higher quality HD outcomes compared to a CVC, but an AVF cannotbe created quickly, and it is more expensive in the sense of the surgical disutility itimposes on patients.In Chapter 3, we developed a Monte-Carlo simulation model to address thetiming of AVF preparation for progressive CKD patients who have not yet initiatedHD. We considered two types of strategies based on approaches suggested in recentlypublished guidelines: refer when hemodialysis is anticipated to begin within a certaintime frame or refer when eGFR drops below a certain threshold. We evaluated thesestrategies over a range of values for each strategy, compared them with respect todifferent performance metrics (e.g., a patient’s life expectancy after HD initiationand percentage of patients with an unnecessary AVF creation), and provided policy63Chapter 5. Conclusions, Extensions and Further Applicationsrecommendations.Our simulation results shows that in general, AVF referral within about 12months of the estimated time to dialysis performed best among time frame strate-gies, and referral at eGFR between 15 and 20 mL/min/1.73m2 performed best amongthreshold strategies. Elderly patients with CKD could be referred later to reducethe risk of creating an AVF that is never used. Similar to Chapter 2, the focus inthis chapter was on patients outcomes rather than costs. A future cost-effectivenessanalysis can elaborate whether an early AVF referral is cost-effective, especially forthe elderly who benefit the least from hemodialysis, are more frail and have multi-ple co-morbidities. We did not consider arteriovenous grafts (AVGs) as a vascularaccess choice in our model. It would be interesting to investigate how consideringAVGs would affect the optimal policy structure, patient outcomes, and costs.One of our results in Chapter 3 was that the timing of referral should be guidedby the individual rate of CKD progression. In Chapter 4, motivated by this finding,we analyzed the problem of designing ongoing treatment plans for a heterogeneouspopulation with respect to disease progression and response to medical interventions.We developed a dynamic programming model that incorporates patient heterogene-ity in disease progression when making clinical decisions. The designed model learnsthe patient type by monitoring the patient health over time and updates a patient’streatment plan according to the gathered information. We formulated the prob-lem as a two-dimensional state-space partially observable Markov decision process(POMDP) and provided structural properties of the value-function, as well as theoptimal policy for the special case of optimal stopping problems.We applied this framework to the AVF preparation timing question posed inChapter 3 by considering two types of patients, patients with slow and fast eGFRprogression. We showed that under data-driven conditions, the optimal policy forstarting the AVF preparation is of a threshold type, i.e., it is optimal to startthe AVF preparation, whenever starting is optimal for higher eGFR values (betterkidney health) or when our belief that the patient is a slow processor is higher.Our numerical results showed that when the patient type is a slow progressor(fast progressor) with certainty, the optimal eGFR threshold beyond which startingthe AVF preparation is optimal is 15 mL/min/1.73m2 (20 mL/min/1.73m2). This isconsistent with the Canadian Society of Nephrology (CSN) guidelines which suggestsstarting the AVF preparation at an eGFR of between 15 and 20 mL/min/1.73m2 [16].Our results sharpens the guidelines by matching the lower bound of 15 with patientsclassified as slow progressors (mean eGFR decline of .64 mL/min/1.73m2 per month)and the upper bound of 20 with the fast progressors (mean eGFR decline of 1.4mL/min/1.73m2 per month).We believe that this framework can be extended to other contexts where aMarkov Decision Process (MPD) is applicable and transition parameters can belearned by observing state transitions. Although we considered two patient typesin this chapter, our results can be extended to cases with multiple patient types.64Chapter 5. Conclusions, Extensions and Further ApplicationsWe also believe that the framework can be applied to other chronic diseases whereheterogeneity in disease progression is present. We considered a special structurefor the state dynamics of MDPs where random variables representing the differencebetween the states in periods t and t+ 1 for each action do not depend on the statein period t. It would be interesting to extend the result to a more general setting forstate transitions. Finally, we only investigated the monotonicity of optimal policiesfor optimal stopping timing problems. As a direction for future research, one mightconsider extending the results to the case with a more general action space.65Bibliography[1] National Institutes of Health. Chronic kidney disease and kidney failure.,2016. Retrieved June 1, 2016, https://report.nih.gov/NIHfactsheets/ViewFactSheet.aspx?csid=34&key=C#C.[2] National Kidney & Urologic Diseases Information Clearinghouse (NKUDIC).Kidney and urologic diseases statistics for the united states, 2016. RetrievedJune 1, 2016, http://kidney.niddk.nih.gov/kudiseases/pubs/kustats/.[3] National Kidney Foundation Vascular Access Work Group. Clinical practiceguidelines for vascular access. American Journal of Kidney Diseases, 48(Sup-plement 1):S248–S273, 7 2006.[4] Harold I Feldman, Marshall Joffe, Sylvia E Rosas, J Eileen Burns, Jill Knauss,and Kenneth Brayman. Predictors of successful arteriovenous fistula matura-tion. American Journal of Kidney Diseases, 42(5):1000–1012, 2003.[5] Hugh C Rayner, Ronald L Pisoni, Brenda W Gillespie, David A Goodkin,Takashi Akiba, Tadao Akizawa, Akira Saito, Eric W Young, and Friedrich KPort. Creation, cannulation and survival of arteriovenous fistulae: data fromthe dialysis outcomes and practice patterns study. Kidney International,63(1):323–330, 2003.[6] Rajnish K Dhingra, Eric W Young, TE Hulbert-Shearon, Sean F Leavey, andFriedrich K Port. Type of vascular access and mortality in US hemodialysispatients. Kidney International, 60(4):1443–1451, 2001.[7] Haimanot Wasse, Nancy Kutner, Rebecca Zhang, and Yijian Huang. Associa-tion of initial hemodialysis vascular access with patient-reported health statusand quality of life. Clinical Journal of the American Society of Nephrology,2(4):708–714, 2007.[8] Eduardo Lacson, J Michael Lazarus, Jonathan Himmelfarb, T Alp Ikizler,and Raymond M Hakim. Balancing fistula first with catheters last. AmericanJournal of Kidney Diseases, 50(3):379–395, 2007.[9] Jeffrey Perl, Ron Wald, Philip McFarlane, Joanne M Bargman, EdwardVonesh, Yingbo Na, S Vanita Jassal, and Louise Moist. Hemodialysis vas-66Bibliographycular access modifies the association between dialysis modality and survival.Journal of the American Society of Nephrology, 22(6):1113–1121, 2011.[10] Ann M O’Hare, Michael Allon, and James S Kaufman. Whether and when torefer patients for predialysis av fistula creation: complex decision making inthe face of uncertainty. Seminars in Dialysis, 23(5):452–455, 2010.[11] Ann M O’Hare, Adam Batten, Nilka R´ıos Burrows, Meda E Pavkov, LeslieTaylor, Indra Gupta, Jeff Todd-Stenberg, Charles Maynard, Rudolph A Ro-driguez, Fliss EM Murtagh, et al. Trajectories of kidney function decline inthe 2 years before initiation of long-term dialysis. American Journal of KidneyDiseases, 59(4):513–522, 2012.[12] I. Gorodetskaya, S. Zenios, C. E. McCulloch, A. Bostrom, C. Hsu, A. B. Bind-man, A. S. Go, and G. M. Chertow. Health-related quality of life and estimatesof utility in chronic kidney disease. Kidney International, 68(6):2801–2808,2005.[13] National kidney foundation K/DOQI. Clinical practice guidelines for vascularaccess: Update 2000. American Journal of Kidney Diseases, 48(Supplement1):S137–S181, 2000.[14] National kidney foundation K/DOQI. Clinical practice guidelines for vascularaccess. American Journal of Kidney Diseases, 48(Supplement 1):S176–S247.,2006.[15] Garabed Eknoyan and Nathan W. Levin. K/DOQI clinical practice guide-lines for chronic kidney disease: Evaluation, classification, and stratification.American Journal of Kidney Diseases, 14(Supplement 2):S1–S246, 2002.[16] K. Jindal, C. T. Chan, C. Deziel, D. Hirsch, S. D. Soroka, M. Tonelli, andB. F. Culleton. Hemodialysis Clinical Practice Guidelines for the CanadianSociety of Nephrology: (Chapter 4) Vascular access . Journal of the AmericanSociety of Nephrology, 17(3 Supplement 1):S16–S23, 2006.[17] M Tonelli, N Wiebe, G Knoll, A Bello, S Browne, D Jadhav, S Klarenbach,and J Gill. Systematic review: kidney transplantation compared with dial-ysis in clinically relevant outcomes. American Journal of Transplantation,11(10):2093–2109, 2011.[18] UNOS. United Network for Organ Sharing, 2014. Retrieved Feb 1, 2014,http://www.unos.org/.[19] Annual Data Report: 2013 Atlas of CKD and ESRD: United States RenalData System, 2013.67Bibliography[20] Fistula First. Fistula first breakthrough initiative, 2014. Retrieved Feb 1,2014, http://www.fistulafirst.org/.[21] Jean Ethier, David C Mendelssohn, Stacey J Elder, Takeshi Hasegawa, TadaoAkizawa, Takashi Akiba, Bernard J Canaud, and Ronald L Pisoni. Vascu-lar access use and outcomes: an international perspective from the dialysisoutcomes and practice patterns study. Nephrology Dialysis Transplantation,23(10):3219–3226, 2008.[22] C. E. Lok, M. Allon, L. Moist, M. J. Oliver, H. Shah, and D. Zimmerman. Riskequation determining unsuccessful cannulation events and failure to matura-tion in arteriovenous fistulas. Journal of the American Society of Nephrology,17(11):3204–3212, 2006.[23] W. J. Peterson, J. Barker, and M. Allon. Disparities in fistula maturation per-sist despite preoperative vascular mapping. Clinical Journal of the AmericanSociety of Nephrology, 3(2):437–441, 2008. id: 135.[24] Prabir Roy-Chaudhury, Vikas P Sukhatme, and Alfred K Cheung. Hemodial-ysis vascular access dysfunction: a cellular and molecular viewpoint. Journalof the American Society of Nephrology, 17(4):1112–1127, 2006.[25] Aicha Radoui, Zineb Lyoussfi, Intissar Haddiya, Zoubair Skalli, RedouaneEl Idrissi, Hakima Rhou, Fatima Ezzaitouni, Naima Ouzeddoun, AbbesEl Mesnaoui, Rabea Bayahia, et al. Survival of the first arteriovenous fistula in96 patients on chronic hemodialysis. Annals of Vascular Surgery, 25(5):630–633, 2011.[26] C. Basile and C. Lomonte. Pro: The arteriovenous fistula is a blessing of God.Nephrology Dialysis Transplantation, 27(10):3752–3756, 2012.[27] R. Amerling. Con: On cardiovascular outcomes and the arteriovenous fistula:lesser of evils. Nephrology Dialysis Transplantation, 27(10):3756–3757, 2012.[28] N. Lameire and W. Van Biesen. Moderator’s view: A ‘secular’ view on vascularaccess in haemodialysis. Nephrology Dialysis Transplantation, 27(10):3758–3761, 2012.[29] L. M. Moist, Charmaine E. Lok, T. J. Vachharajani, W. Xi, A. AlJaishi, K. R.Polkinghorne, M. Vazquez, and T. C. Lee. Optimal hemodialysis vascularaccess in the elderly patient. Seminars in Dialysis, 25(6):640–648, 2012.[30] M.J. Barry and S. Edgman-Levitan. Shared decision making-the pinnacleof patient-centered care. New England Journal of Medicine, 366(9):780–781,2012.68Bibliography[31] M. K. Tamura, J. C. Tan, and A. M. O’Hare. Optimizing renal replacementtherapy in older adults: a framework for making individualized decisions.Kidney International, 82(3):261–269, 2011.[32] H.C. Sox and S. Greenfield. Comparative effectiveness research: A report fromthe Institute of Medicine. Annals of Internal Medicine, 151:203–205, 2009.[33] L Boulware. Comparative effectiveness research in kidney disease: a nationalpriority. American Journal of Kidney Diseases, 61(1):9–12, 2013.[34] O. Alagoz, L. M. Maillart, A. J. Schaefer, and M. S. Roberts. The optimal tim-ing of living-donor liver transplantation. Management Science, 50(10):1420–1430, 2004.[35] S. M. Shechter, M. D. Bailey, A. J. Schaefer, and M. S. Roberts. The optimaltime to initiate HIV therapy under ordered health states. Operations Research,56(1):20–33, 2008.[36] L. M. Maillart, J. S. Ivy, S. Ransom, and K. Diehl. Assessing dynamic breastcancer screening policies. Operations Research, 56(6):1411–1427, 2008.[37] T. Ayer, O. Alagoz, and N. K. Stout. OR Forum-A POMDP approach to per-sonalize mammography screening decisions. Operations Research, 60(5):1019–1034, 2012.[38] Chris P. Lee, Glenn M. Chertow, and Stefanos A. Zenios. Optimal initiationand management of dialysis therapy. Operations Research, 56(6):1428–1449,2008.[39] Jingyu Zhang, Brian T Denton, Hari Balasubramanian, Nilay D Shah, andBrant A Inman. Optimization of prostate biopsy referral decisions. Manufac-turing & Service Operations Management, 14(4):529–547, 2012.[40] Mehmet US Ayvaci, Oguzhan Alagoz, and Elizabeth S Burnside. The effect ofbudgetary restrictions on breast cancer diagnostic decisions. Manufacturing& Service Operations Management, 14(4):600–617, 2012.[41] Hui Xue, Eduardo Lacson, Weiling Wang, Gary C Curhan, and Steven MBrunelli. Choice of vascular access among incident hemodialysis patients: adecision and cost-utility analysis. Clinical Journal of the American Society ofNephrology, 5(12):2289–2296, 2010.[42] Swapnil Hiremath, Greg Knoll, and Milton C Weinstein. Should the arteriove-nous fistula be created before starting dialysis?: a decision analytic approach.PloS one, 6(12):e28453, 2011.69Bibliography[43] S. M. Shechter, Skandari M. R., and N. Zalunardo. Timing of arteriovenousfistula creation in patients with CKD: A decision analysis. American Journalof Kidney Diseases, 63(1):95–103, 2014.[44] D. Drew, C. Lok, J. Cohen, M. Wagner, N. Tangri, and D. Weiner. Vascularaccess choice in incident hemodialysis patients: A decision analysis. Journalof the American Society of Nephrology, 2014. Forthcoming.[45] M. Kurella, K. E. Covinsky, A. J. Collins, and G. M. Chertow. Octogenariansand nonagenarians starting dialysis in the United States. Annals of InternalMedicine, 146(3):177–183, 2007.[46] Raymond M Hakim and Jonathan Himmelfarb. Hemodialysis access failure:a call to action revisited. Kidney international, 76(10):1040–1048, 2009.[47] M.R. Gold, J.E. Siegel, L.B. Russell, and M.C. Weinstein. Cost-Effectivenessin Health and Medicine. Oxford University Press, USA, 1996.[48] Institute of Medicine. Insuring America’s Health: Princi-ples and Recommendations, 2004. Retrieved October 15, 2013,http://www.iom.edu/Reports/2004/Insuring-Americas-Health-Principles-and-Recommendations.aspx.[49] Allan J Collins, Robert N Foley, David T Gilbertson, and Shu-Chen Chen.The state of chronic kidney disease, esrd, and morbidity and mortality in thefirst year of dialysis. Clinical Journal of the American Society of Nephrology,4(Supplement 1):S5–S11, 2009.[50] William F Owen. Patterns of care for patients with chronic kidney disease inthe United States: dying for improvement. Journal of the American Societyof Nephrology, 14(Supplement 2):S76–S80, 2003.[51] Douglas C Montgomery, Cheryl L Jennings, and Murat Kulahci. Introductionto time series analysis and forecasting. John Wiley & Sons, 2015.[52] Emre Erdogan, Sheng Ma, Alina Beygelzimer, and Irina Rish. Statisticalmodels for unequally spaced time series. In Proceedings of the Fifth SIAMInternational Conference on Data Mining. SIAM, 2004.[53] Garrett M Fitzmaurice, Nan M Laird, and James H Ware. Applied longitudinalanalysis, volume 998. John Wiley & Sons, 2012.[54] Rockwell Automated Technologies. Arena (version 10), 2010.[55] Laura M Dember, Gerald J Beck, Michael Allon, James A Delmez, Bradley SDixon, Arthur Greenberg, Jonathan Himmelfarb, Miguel A Vazquez, Jen-nifer J Gassman, Tom Greene, et al. Effect of clopidogrel on early failure of70Bibliographyarteriovenous fistulas for hemodialysis: a randomized controlled trial. Jama,299(18):2164–2171, 2008.[56] Wenjie Wang, Brendan Murphy, Serdar Yilmaz, Marcello Tonelli, JenniferMacRae, and Braden J Manns. Comorbidities do not influence primary fistulasuccess in incident hemodialysis patients: a prospective study. Clinical Journalof the American Society of Nephrology, 3(1):78–84, 2008.[57] Carrie A Schinstock, Robert C Albright, Amy W Williams, John J Dillon,Eric J Bergstralh, Bernice M Jenson, James T McCarthy, and Karl A Nath.Outcomes of arteriovenous fistula creation after the fistula first initiative. Clin-ical Journal of the American Society of Nephrology, 6(8):1996–2002, 2011.[58] Dember, Laura M., and Bradley S. Dixon. Early fistula failure: Back to basics.American Journal of Kidney Diseases, 50(5):696–699, 2007. id: 127.[59] Barry M Brenner, Mark E Cooper, Dick de Zeeuw, William F Keane,William E Mitch, Hans-Henrik Parving, Giuseppe Remuzzi, Steven M Snap-inn, Zhonxin Zhang, and Shahnaz Shahinfar. Effects of losartan on renal andcardiovascular outcomes in patients with type 2 diabetes and nephropathy.New England Journal of Medicine, 345(12):861–869, 2001.[60] Lawrence G Hunsicker, Sharon Adler, Arlene Caggiula, Brian K England,Tom Greene, John W Kusek, Nancy L Rogers, Paul E Teschan, Gerald Beck,Modification of Diet in Renal Disease Study Group, et al. Predictors of theprogression of renal disease in the Modification of Diet in Renal Disease Study.Kidney International, 51(6):1908–1919, 1997.[61] E. Arias. United states life tables. National Vital Statistics Reports, 58(21):1—40, 2010.[62] Pietro Ravani, Suetonia C Palmer, Matthew J Oliver, Robert R Quinn, Jen-nifer M MacRae, Davina J Tai, Neesh I Pannu, Chandra Thomas, Brenda RHemmelgarn, Jonathan C Craig, et al. Associations between hemodialysis ac-cess type and clinical outcomes: a systematic review. Journal of the AmericanSociety of Nephrology, 24(3):465–473, 2013.[63] Martin Ferring, Martin Claridge, Steven A Smith, and Teun Wilmink. Routinepreoperative vascular ultrasound improves patency and use of arteriovenousfistulas for hemodialysis: a randomized trial. Clinical Journal of the AmericanSociety of Nephrology, pages 282–310, 2010.[64] United States Renal Data System. 2012 atlas of CKD & ESRD., 2012. Re-trieved June 1, 2016, http://www.usrds.org/atlas12.aspx”.[65] Andrew H Briggs. Handling uncertainty in cost-effectiveness models. Phar-macoeconomics, 17(5):479–500, 2000.71Bibliography[66] Peter Doubilet, Colin B Begg, Milton C Weinstein, Peter Braun, and Bar-bara J McNeil. Probabilistic sensitivity analysis using monte carlo simulation.a practical approach. Medical Decision Making: an International Journal ofthe Society for Medical Decision Making, 5(2):157–177, 1984.[67] Sylvia E Rosas and Harold I Feldman. Synthetic vascular hemodialysis accessvs native arteriovenous fistula: A cost-utility analysis. Annals of Surgery,255(1):181, 2012.[68] Manjula Kurella Tamura, Kenneth E Covinsky, Glenn M Chertow, KristineYaffe, C Seth Landefeld, and Charles E McCulloch. Functional status ofelderly adults before and after initiation of dialysis. New England Journal ofMedicine, 361(16):1539–1547, 2009.[69] Lorien S Dalrymple, Ronit Katz, Bryan Kestenbaum, Michael G Shlipak,Mark J Sarnak, Catherine Stehman-Breen, Stephen Seliger, David Siscov-ick, Anne B Newman, and Linda Fried. Chronic kidney disease and the riskof end-stage renal disease versus death. Journal of general internal medicine,26(4):379–385, 2011.[70] Tushar J Vachharajani, Shahriar Moossavi, Jean R Jordan, Vidula Vachhara-jani, Barry I Freedman, and John M Burkart. Re-evaluating the fistula firstinitiative in octogenarians on hemodialysis. Clinical Journal of the AmericanSociety of Nephrology, 6(7):1663–1667, 2011.[71] BO Eriksen and OC Ingebretsen. The progression of chronic kidney disease:a 10-year population-based study of the effects of gender and age. KidneyInternational, 69(2):375–382, 2006.[72] Adeera Levin, Ognjenka Djurdjev, Monica Beaulieu, and Lee Er. Variabilityand risk factors for kidney disease progression and death following attainmentof stage 4 ckd in a referred cohort. American Journal of Kidney Diseases,52(4):661–671, 2008.[73] Liang Li, Brad C Astor, Julia Lewis, Bo Hu, Lawrence J Appel, Michael S Lip-kowitz, Robert D Toto, Xuelei Wang, Jackson T Wright, and Tom H Greene.Longitudinal progression trajectory of gfr among patients with ckd. AmericanJournal of Kidney Diseases, 59(4):504–512, 2012.[74] Sujha Subramanian, Michelle Klosterman, Mayur M Amonkar, and Timo-thy L Hunt. Adherence with colorectal cancer screening guidelines: a review.Preventive Medicine, 38(5):536–550, 2004.[75] John M Inadomi, Sandeep Vijan, Nancy K Janz, Angela Fagerlin, Jennifer PThomas, Yunghui V Lin, Roxana Mun˜oz, Ma Somsouk, Najwa El-Nachef,72Bibliographyand Rodney A Hayward. Adherence to colorectal cancer screening: a ran-domized clinical trial of competing strategies. Archives of internal medicine,172(7):575–582, 2012.[76] Turgay Ayer, Oguzhan Alagoz, Natasha K Stout, and Elizabeth S Burn-side. Heterogeneity in womens adherence and its role in optimal breast cancerscreening policies. Management Science, 2015.[77] M Romeo, F Martinelli-Boneschi, M Rodegher, F Esposito, V Martinelli, andG Comi. Clinical and mri predictors of response to interferon-beta and glati-ramer acetate in relapsing–remitting multiple sclerosis patients. EuropeanJournal of Neurology, 20(7):1060–1067, 2013.[78] Mariel S Lavieri, Martin L Puterman, Scott Tyldesley, and William J Morris.When to treat prostate cancer patients based on their psa dynamics. IIETransactions on Healthcare Systems Engineering, 2(1):62–77, 2012.[79] Anahita Khojandi, Lisa M Maillart, Oleg A Prokopyev, Mark S Roberts, Tim-othy Brown, and William W Barrington. Optimal Implantable CardioverterDefibrillator (ICD) Generator Replacement. INFORMS Journal on Comput-ing, 26(3):599–615, 2014.[80] Aharon Ben-Tal and Arkadi Nemirovski. Robust convex optimization. Math-ematics of operations research, 23(4):769–805, 1998.[81] Jay K Satia and Roy E Lave Jr. Markovian decision processes with uncertaintransition probabilities. Operations Research, 21(3):728–740, 1973.[82] Joel Goh, Mohsen Bayati, Stefanos A Zenios, Sundeep Singh, and DavidMoore. Data uncertainty in markov chains: Application to cost-effectivenessanalyses of medical innovations. Working Paper, 2015.[83] James John Martin. Bayesian Decision Problems and Markov Chains. Wiley,1967.[84] Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analyticsolution to discrete bayesian reinforcement learning. In Proceedings of the 23rdinternational conference on Machine learning, pages 697–704. ACM, 2006.[85] Mauricio Araya-Lo´pez, Olivier Buffet, Vincent Thomas, and Franc¸ois Charpil-let. Active learning of MDP models. In Recent Advances in ReinforcementLearning, pages 42–53. Springer, 2011.[86] Arthur Guez, David Silver, and Peter Dayan. Efficient Bayes-adaptive rein-forcement learning using sample-based search. In Advances in Neural Infor-mation Processing Systems, pages 1025–1033, 2012.73Bibliography[87] Richard D Smallwood and Edward J Sondik. The optimal control of par-tially observable markov processes over a finite horizon. Operations Research,21(5):1071–1088, 1973.[88] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Plan-ning and acting in partially observable stochastic domains. Artificial intelli-gence, 101(1):99–134, 1998.[89] Michael O’Gordon Duff. Optimal Learning: Computational procedures forBayes-adaptive Markov decision processes. PhD thesis, University of Mas-sachusetts Amherst, 2002.[90] Vikram Krishnamurthy. Partially Observed Markov Decision Processes: FromFiltering to Controlled Sensing. Cambridge University Press, 2016.[91] M. Mundhenk. The complexity of planning with partially-observable MarkovDecision Processes. PhD thesis, Friedrich-Schiller-Universitdt., 2001.[92] William S Lovejoy. Some monotonicity results for partially observed markovdecision processes. Operations Research, 35(5):736–743, 1987.[93] William S Lovejoy. Technical noteon the convexity of policy regions in partiallyobserved systems. Operations Research, 35(4):619–621, 1987.[94] Jonathan E Helm, Mariel S Lavieri, Mark P Van Oyen, Joshua D Stein, andDavid C Musch. Dynamic forecasting and control algorithms of glaucomaprogression for clinician decision support. Operations Research, 63(5):979–999, 2015.[95] Diana M Negoescu, Kostas Bimpikis, Margaret L Brandeau, and Dan A Iancu.Dynamic learning of patient response types: An application to treating chronicdiseases. Working Paper, 2015.[96] Turgay Ayer, Oguzhan Alagoz, and Natasha K Stout. OR Forum-A POMDPapproach to personalize mammography screening decisions. Operations Re-search, 60(5):1019–1034, 2012.[97] Martin L Puterman. Markov Decision Processes: Discrete Stochastic DynamicProgramming. John Wiley & Sons, 2014.[98] M. Shaked and J. G. Shanthikumar. Stochastic orders. Springer Verlag, 2007.[99] SeChan Oh. Optimal Stopping Problems in Operations Management. PhDthesis, Stanford University, 2010.[100] A. Mu¨ller and D. Stoyan. Comparison methods for stochastic models and risks.Wiley, 2002.74[101] R. C. Gupta and D. M. Bradley. Representing the mean residual life in terms ofthe failure rate. Mathematical and Computer Modelling, 37(12-13):1271–1280,2003.[102] Felix Belzunce, Carolina Martinez Riquelme, and Julio Mulero. An Introduc-tion to Stochastic Orders. Academic Press, 2015.75Appendix AChapter 2 Mathematical ProofsSection A.1 provides some general results that are used in Section 2.4, which contains the proofof the analytical results of chapter 2.A.1 General ResultsLemma A.1. Each of the following are equivalent to X ≤hr Y :1. Xt ≤st Yt,∀t2.FX(t)FY(t)is decreasing in t.Proof. For (1) see Equation 1.B.7 in [98]. For (2), see Theorem 1.3.3 in [100]. Lemma A.2 (Closure of stochastic order under mixture). Let X, Y be two random variablessuch that for all realizations of the random vector Z, we have [X|Z = z] ≤st [Y |Z = z]. Then,X ≤st Y .Proof. The proof directly follows Theorem 1.2.15 in [100]. Lemma A.3. Assumption 2.4 is equivalent to having thatFC(t)FA(t)is a log-convex function oft.Proof. Note that ddt ln FX(t) = −rX(t). Since ddt lnFC(t)FA(t)= ddt ln FC(t)− ddt ln FC(t) = rA(t)−rC(t), the result follows from Assumption 2.4 and the fact that a differentiable function isconvex if and only if its derivative is increasing. Lemma A.4. Assume that g is a differentiable and log-convex function. Then, g(x)g(x+a) isdecreasing in x for any a ≥ 0.Proof. It suffices to show that ln g(x)g(x+a) = ln g(x) − ln g(x + a) is decreasing in x. DefineG := ln g, a convex function by assumption. Since ddx lng(x)g(x+a) =ddxG(x) − ddxG(x + a) ≤ 0,based on the fact that the derivative of a convex function is increasing, we have that ln g(x)g(x+a)is decreasing in x. 76A.2. Analytical ResultsLemma A.5. If the random variable X is IFR, then Xt is stochastically decreasing in t.Proof. Choose t ≤ t′ and s ≥ 0, arbitrarily. We have rXt′ (s) = rX(t′ + s) and rXt(s) =rX(t + s). Since X has the IFR property, we have ∀s, rXt(s) ≤ rXt′ (s). Thus, Xt′ ≤hr Xt bydefinition which implies Xt′ ≤st Xt, because hazard rate order implies the stochastic order (seeLemma A.1). Lemma A.6. If FX(t) is differentiable, then the mean residual lifetime of a random variableX is differentiable. Moreover, we have:ddtEXt = rX(t)EXt − 1.Proof. See [101]. A.2 Analytical ResultsWe provide proofs in three sections. Proof of Theorem 2.1 is given in the first section. Thesecond section includes proofs for Theorems 2.2-2.5, Corollaries 2.1-2.2, and Proposition 2.1.The final section provides proofs for Theorems 2.6 and 2.7.A.2.1 Proof of Theorem 2.1:We first prove a preliminary lemma that facilitates proving the main results.Lemma A.7. Assumptions 2.3-2.5 apply to At and Ct as well. In other words, for all s, t ≥ 0,we have Ct ≤hr At, rCt(s)− rAt(s) is decreasing in s, and rAt(s), rCt(s) are increasing in s.Proof. The result follows by noting that rXt(s) = rX(t+ s) for any random variable X, andt, s ≥ 0. In what follows, we let Ki denote the lifetime of the ith AVF, i.e., Ki = 0, if the ith AVFdoes not mature and Ki = Zi, if otherwise.Proof of Theorem 2.1. We prove Theorem 2.1 for all realization of Mi, and Ki for i =1, . . . , N , where N is the total number of AVF chances. Since AVF creation variables are notaffected by the policy in use based on Assumption 2.6, the result generalizes using Lemma A.2(closure of stochastic order under mixture).Let L(t, n) denote a patient’s residual lifetime at t, given n remaining AVF chances, underthe optimal policy (one that maximizes a patient’s survival function probability for each timet). Suppose one could set the AVF use time (rather than setting the surgery time) at t + u.Let L(u) be a patient’s residual lifetime at time t when we plan to use current AVF at t + uand follow the optimal policy for the subsequent n−1 AVF chances. We prove that FL(u)(a) isdecreasing in u (for any a). Since for y, the surgery time, we have y = u−mi, this is equivalentto proving that the residual lifetime stochastically decreases in y.→ Base case: n=1: Based on Assumption 2.2 on a patient’s survival, we can calculateFL(u)(a) for different values of u, a, k as follows (see Figure A.1).77A.2. Analytical ResultstCVCuCVCAVFkcase 1 case 2 case 3 Figure A.1: Possible cases for FL(u)(a). Case 1: a ≤ u: We have FL(u)(a) (∗)=== P[Ct > a] = FCt(a). Case 2: [a− k]+ ≤ u ≤ a: We haveFL(u)(a) = P[Ct > u,At+u > a− u] = P[Ct > u]P[At+u > a− u∣∣Ct > u](∗)=== P[Ct > a]P[At+u > a− u] (∗)=== P[Ct > u].P[At > a|At > u] = FCt(u)FAt(a)FAt(u) Case 3: 0 ≤ u ≤ [a− k]+: We have:FL(u)(a) = P[Ct > u,At+u > a− u,Ct+u+k > a− (u+ k)]= P[Ct > u].P[Au+t > k∣∣Ct > u].P[Cu+k > a− (u+ k)∣∣At+u > k,Ct > u](∗)=== P[Ct > u].P[At > k + u∣∣At > u].P[Ct > a∣∣Ct > u+ k](∗)=== FCt(u).FAt(k + u)FAt(u)FCt(a)FCt(u+ k)= FCt(a).FCt(u)FAt(u)/FCt(u+ k)FAt(u+ k)in which (∗) represents implication by Assumption 2.2. Note that FL(u)(a) is continuous withineach range, and its value on the boundary points coincides. Therefore, it suffices to prove that ineach range, FL(u)(a) is decreasing. In Case 1, the function is constant and thus the result holdstrivially. In Case 2, since Ct ≤hr At according to Lemma A.7 (which requires Assumptions2.3-2.5), the function is decreasing using Lemma A.1. In Case 3, Lemma A.7 and Lemma A.3imply thatFCt (u)FAt (u)is log-convex in u. Using Lemma A.4, we have that FL(u)(a) is decreasingin u.Let L(t, n, u) be the patient’s residual lifetime at t when we use the current AVF chanceat t + u and follow the optimal policy for the subsequent AVF chances. We now present theinduction step:→ Induction step: Assume L(t, n− 1, u2) ≤st L(t, n− 1, u1), for all u1 ≤ u2. We prove thatif u1 ≤ u2, then L(t, n, u2) ≤st L(t, n, u1).To calculate the lifetime of the patient for the case of multiple AVF chances, we assumethat AVFs are created sequentially and never in parallel (supported by Assumption 2.1). Sincestochastic order is a partial order, using the transitivity property we can instead prove thatL(u2) ≤st L′ and L′ ≤st L(u1), in which L′ is the lifetime under a hypothetical situation similar78A.2. Analytical Resultsto L(u1) with the difference that the decision to use the subsequent AVF is delayed until u2+k(see Figure A.2).u1CVCtu1CVCtAVFAVFCVCu1CVCtAVFkL(t, n, u1)L(t+u1+k, n-1)L(t+u2+k, n-1)L'L(t, n, u2)L(t+u1+k, n-1, u2-u1)≥st≥stu2-u1ku2-u1 kFigure A.2: Induction step and the hypothetical random variable L′. L(u2) ≤st L′: For x ≤ u2 + k, we have that FL(u2)(x) = FL(t,1,u2)(x) and FL′(x) =FL(t,1,u1)(x). Thus the result follows from induction base. Otherwise, we haveFL(u2)(x) = FL(u2)(u2 + k).FL(u2+k,n−1)(x− [u2 + k]),FL′(x) = FL′(u2 + k).FL(u2+k,n−1)(x− [u2 + k]).Based on the previous result, we have FL(u2)(u2 + k) ≤ FL′(u2 + k), and thus we get theresult. L′ ≤st L(t, n, u1). For x ≤ u1 + k, we have that FL(u1)(x) = FL′(x) = FL(t,1,u1)(x). Forx ≥ u1 + k,FL(u1)(x) = FL′(u1 + k).FL(u1+k,n−1,0)(x− [u1 + k]),FL′(x) = FL′(u1 + k).FL(u1+k,n−1,u2−u1)(x− [u1 + k]).Using the induction hypothesis, we have L(u1 + k, n− 1, u2 − u1) ≤st L(u1 + k, n− 1, 0),and thus we have the desired result.A.2.2 Proofs of Theorems 2.2, 2.3-2.5, Corollaries 2.1-2.2, and Proposition2.1:We prove the optimality of threshold policies (Theorem 2.3) in three steps. First in PropositionA.1, we prove the existence of an optimal HD-duration threshold policy for the case n = 1.Next, we prove Theorems 2.4-2.5 and Corollary 2.1 for the special case n = 1. Finally, usingthese results, we prove that the same threshold policy formed in Proposition A.1 is optimal forthe case n > 1, as well.79A.2. Analytical ResultsWe use the following notations in what follows. vpi(NF, n, t): the value function (the remaining QALE of a patient) at state (NF, n, t)under an arbitrary policy pi v(NF, n, t, y): the value function of the policy consisting of surgery planned at t+ y forthe current AVF chance and then the optimal policy for the subsequent decisions. v(NF, n, t): the optimal value function at state (NF, n, t).Note that we supposed Assumptions 2.1-2.2 and 2.6 in defining the dynamic programmingmodel (see Section 2.3.4). Let pi0 denote the policy of using CVC for the rest of the patient’slife (hereafter referred to as the “no-AVF” policy). Under this policy, the patient remains on aCVC until she dies, and since her residual lifetime under this policy is Ct, her QALE is qcE[Ct],i.e., we have ∀NF, n : vpi0(NF, n, t) = qcECt. Since vpi0(NF, n, t) = qcECt for any NF and n,we use vpi0(., ., t) to denote this independence.Let s denote a general patient state. Note that the value function of an arbitrary policypi, i.e., vpi(s), is the expected quality adjusted lifetime of a patient under that policy. In whatfollows, we let vpi(s|E) represent the value function of the policy pi conditional on an event E . Forinstance,(vpi1(s)− vpi2(s)∣∣Ct ≤ y) denotes the QALE difference between two arbitrary policiespi1 and pi2 conditional on the event Ct ≤ y. We use Lemmas A.8-A.10 to prove PropositionA.1.AVF𝑡 𝑦 𝑀𝑛𝑣(𝑁𝐹, 𝑛, 𝑡 + 𝑦, 0)𝐾𝑛𝑣(𝑁𝐹, 𝑛, 𝑡, 𝑦)𝑣(𝑁𝐹′, 𝑡′, 𝑛 − 1)CVC𝑡 𝑦𝑣𝜋0(𝑁𝐹, 𝑛, 𝑡) CVC𝑣𝜋0(𝑁𝐹, 𝑛, 𝑡 + 𝑦)𝑡′Figure A.3: Linking v(NF, n, t, y), v(NF, n, t+ y, 0), and vpi0(NF, n, t+ y).Lemma A.8. The following equality holds for v(NF, n, t, y).v(NF, n, t, y) = FCt(y)[v(NF, n, t+ y, 0)− vpi0(., ., t+ y)]+ vpi0(., ., t)Proof. Consider Figure A.3. We want to prove that the difference between the value functionsof the policy consisting of surgery planned at t + y for the current AVF chance and thenthe optimal policy for the subsequent decisions and the no-AVF policy, i.e., v(NF, n, t, y) −vpi0(., ., t), equals FCt(y)[v(NF, n, t+ y, 0)− vpi0(., ., t+ y)].If Ct ≤ y, then the patient dies before the AVF surgery, in which case there is no differencebetween the two policies. If Ct > y, which happens with probability FCt(y), then the differencebetween the two policies equals the difference between the value function at the state (NF, n, t+80A.2. Analytical Resultsy) when we follow the policy consisting of immediate surgery for the current AVF chance andthen the optimal policy for the subsequent decisions and that of the same state but followingthe no-AVF policy, i.e., v(NF, n, t+ y, 0)− vpi0(., ., t+ y). Therefore, we have v(NF, n, t, y)−vpi0(., ., t) = FCt(y)[v(NF, n, t+ y, 0)− vpi0(., ., t+ y)] and thus the result. For Lemma A.9, let w(t,m, k) denote the residual HD utility adjusted lifetime expectancyof a patient at time t (which is the patient’s QALE without subtracting the AVF creationdisutility) under a scenario in which the patient undergoes the surgery at t for her only AVFchance and the AVF maturation time and AVF lifetime are deterministically set at m and k,respectively.Based on Assumption 2.2 on a patient’s survival, we can calculate w(t,m, k) as follows:w(t,m, k) = qc∫ m0xfCt(x)dx+FCt(m)[qcm+qa∫ k0xfAt+m(x)dx+ FAt+m(k)[qak + qcECt+m+k]]. (A.1)We can express v(NF, 1, t, 0) and vpi0(., ., t) using w(t,m, k) as follows:v(NF, 1, t, 0) = −d+ EM,K|NF [w(t,m, k)], (A.2)vpi0(., ., t) = w(t,m, 0) : ∀m. (A.3)We will use these equalities in later proofs.Lemma A.9. Suppose Assumptions 2.2-2.5 and 2.8. We have ∂∂kw(t,m, k) is non-negative anddecreasing in t and m.Proof. To have differentiability of w in k, it suffices to assume that FA(x) and FC(x) aredifferentiable at all values of x because they in turn imply that FAt(x) and FCt(x) (as a directresult) and ECx (using Lemma A.6) are differentiable functions in x.We have:∂∂kw(t,m, k) =FCt(m)[ddkqa∫ k0xfAt+m(x)dx+ FAt+m(k)ddk[qak + qcECt+m+k]+[ ddkFAt+m(k)][qak + qcECt+m+k]](A.4)= FCt(m)[qakfAt+m(k) + FAt+m(k){qa + qc[rCt+m(k)ECt+m+k − 1]}− fAt+m(k)[qak + qcECt+m+k]](A.5)= FCt(m)FAt+m(k)[qa − qc + qcECt+m+k[rCt(m+ k)− rAt(m+ k)]](A.6)81A.2. Analytical Resultswhere Equation A.4 follows from Equation A.1 using the product rule in calculating the deriva-tives of products of two functions, Equation A.5 follows from Equation A.4 by using LemmaA.6, and finally Equation A.6 follows from Equation A.5 by rearranging terms.We can prove that ∂∂kw(t,m, k) is decreasing in t and non-negative by showing that it is aproduct of the following three non-negative decreasing functions:1. FCt(m): This is decreasing in t, since Ct is stochastically decreasing in t based on LemmaA.5 and that C is IFR by Assumption 2.5.2. FAt+m(k): This is decreasing in t, since At+m is stochastically decreasing in t based onLemma A.5 and that A is IFR by Assumption 2.5.3. qa − qc + qcECt+m+k[rCt(m+ k)− rAt(m+ k)]: non-negative: We have that qa ≥ qc by Assumption 2.8. Also, rCt(m + k) ≥rAt(m+ k) based on Lemma A.7 (which requires Assumptions 2.3-2.5). decreasing: ECt+m+k is decreasing in t, because Ct+m+k is stochastically decreasingin t by Lemma A.5 and the fact that C is IFR by Assumption 2.5. Also, rCt(m +k)− rAt(m+ k) is decreasing in t based on Lemma A.7.Using the same logic, we can show that ∂∂kw(t,m, k) is decreasing in m.Lemma A.10. Suppose Assumptions 2.2-2.6 and 2.8. For any NF , we have v(NF, 1, t, 0) −vpi0(., ., t) is decreasing in t.Proof. Choose t1 ≤ t2, arbitrarily. We have that ∀m : ∂∂k [w(t2,m, k)−w(t1,m, k)] ≤ 0 by thelinearity of the differential operator and Lemma A.9 (which requires Assumptions 2.2-2.5 and2.8). This implies that∀k,m : w(t2,m, k)− w(t1,m, k) ≤ w(t2,m, 0)− w(t1,m, 0).But by Equation A.3 we have: ∀m, t : w(t,m, 0) = vpi0(., ., t). Thus,∀k,m : w(t2,m, k)− w(t1,m, k) ≤ vpi0(., ., t2)− vpi0(., ., t1).Taking expectation from both sides with respect to M,K|NF and Equation A.2 gives us:v(NF, 1, t2, 0)− v(NF, 1, t1, 0) ≤ vpi0(., ., t2)− vpi0(., ., t1).Note that taking expectation is justified based on Assumption 2.6. By rearranging the termsin the above inequality, we obtain the desired result. 82A.2. Analytical ResultsProof of Theorem 2.2. This result is in fact a corollary to Lemma A.10. By assumingd = 0, M = 0, and K = ∞ with probability 1, we obtain v(NF, 1, t, 0) = qAEAt. Sincevpi0(., ., t) = qCECt by definition, we have that qAE[At]− qCE[Ct] is decreasing in t. The resultthen follows by assuming qA = qC = 1. Proposition A.1 (Existence of Threshold Policies for n = 1). Assume n = 1 and fix NF ,arbitrarily. Under Assumptions 2.2-2.6 and 2.8, there exists a threshold policy τ∗(NF ) thatmaximizes the QALE of the patient.Proof. Fix t, and NF , arbitrarily. Assume that we plan the surgery at t+ y. By Lemma A.8,we have:v(NF, 1, t, y) = FCt(y)[v(NF, 1, t+ y, 0)− vpi0(., ., t+ y)]+ vpi0(., ., t)For this decision to be an optimal action, it is necessary that surgery at t+ y is no worse thanthe no-AVF policy, i.e., v(NF, 1, t+ y, 0) ≥ vpi0(., ., t+ y).Since v(NF, 1, t + y, 0) − vpi0(., ., t + y) is decreasing in y by Lemma A.10 (which requiresAssumptions 2.2-2.6 and 2.8), and FCt(y) is decreasing in y, then v(NF, 1, t, y) is decreasing iny for all y that satisfy the necessary condition. Thus, the optimal action is to perform surgeryat t if v(NF, 1, t, 0) ≥ vpi0(., ., t), and no surgeries, if otherwise.Now, we form the policy τ∗ as follows based on whether v(NF, 1, 0, 0) ≤ vpi0(., ., 0) or not. v(NF, 1, 0, 0) ≤ vpi0(., ., 0): we have that for ∀t : v(NF, 1, 0, 0) ≤ vpi0(., ., 0), sincev(NF, 1, t, 0) − vpi0(., ., t) is decreasing in t by Lemma A.10. As a result, the no AVFsurgery (i.e., “CVC forever”) is optimal for all t. Choose τ∗(NF ) = 0 in this case. v(NF, 1, 0, 0) > vpi0(., ., 0): we have that ∃t′ ≤ ∞ such that for t < t′, we have v(NF, 1, 0, 0) >vpi0(., ., 0), and v(NF, 1, 0, 0) ≤ vpi0(., ., 0) for t ≥ t′ because v(NF, 1, t, 0) − vpi0(., ., t) isdecreasing in t. For t < t′, surgery at t is optimal, and for t ≥ t′, the patient shouldremain on a CVC, i.e., the no surgery policy is optimal. Choose τ∗(NF ) = t′ in this case.The policy τ∗(NF ) is optimal for n = 1 by construction. Now that we have achieved the first step in proving the optimality of threshold policies,we prove Theorems 2.4-2.5 and Corollary 2.1 for the special case n = 1. Once we prove theoptimality of τ∗(NF ) for all n in Theorem 2.3, which requires Assumptions 2.1-2.8, these resultsalso generalize.Proof of Theorem 2.4 for n = 1. Based on the way we constructed τ∗(NF ; d) in Proposi-tion A.1, we have:t ≥ τ∗(NF ; d) ⇐⇒ v(NF, 1, t, 0; d) ≤ vpi0(., ., t). (A.7)83A.2. Analytical ResultsDefine dcr(NT, t) as follows:dcr(NF, t) := EM,K|NF [w(t,m, k)]− vpi0(., ., t). (A.8)Note that by Equation A.2, we have:dcr(NF, t) = d+ v(NF, 1, t, 0)− vpi0(., ., t). (A.9)By Equation A.7 and the above equality, we have:t ≥ τ∗(NF ) ⇐⇒ dcr(NF, t) ≤ d (A.10)Thus, dcr(NF, t) is indeed a critical value for AVF creation disutility in determining the optimaldecision. Proof of Theorem 2.5 for n = 1. We have:dcr(NF, t) = P[K = 0|NF ]EM [w(t,m, 0)] + P[K > 0|NF ]EM,Z [w(t,m, z)]− vpi0(., ., t) (A.11)= P[K = 0|NF ]vpi0(., ., t) + P[K > 0|NF ]EM,Z [w(t,m, z)]− vpi0(., ., t) (A.12)= P[B = 1|NF ](EM,Z [w(t,m, z)]− vpi0(., ., t)), (A.13)where Equation A.11 follows from the definition of dcr(NF, t) in Equation A.8, the law of totalprobability and definitions of K and Z, Equation A.12 follows from Equation A.11 by usingthe fact vpi0(., ., t) = w(t,m, 0) (see Equation A.3), and Equation A.13 follows from EquationA.12 by rearranging terms. Note that we can use Equation A.13 to numerically calculate the critical disutility bycalculating EM,Z [w(t,m, z)], either by Monte-Carlo simulation or analytically, and vpi0(., ., t)using the equality vpi0(., ., t) = qcECt.Proof of Corollary 2.1 for n = 1.By Equation A.13, we have dcr(NF, t) = P[B = 1|NF ](EM,Z [w(t,m, z)] − vpi0(., ., t)). ByAssumption 2.7, the AVF surgery success probability is decreasing in NF . Therefore, we havethat the critical disutility is decreasing in NF for any t.Choose NF1 ≤ NF2, arbitrarily. Let ti = τ∗(NFi) for i = 1, 2. By Equation A.10, wehave dcr(NF1, t1) ≥ d (substitute t1 for t and NF1 for NF ). Since the critical disutility isdecreasing in NF for any t, we have dcr(NF2, t1) ≤ d, as well. By Equation A.10, we havet2 ≤ t1 (substitute t2 for t and NF2 for NF ). Before proving Theorem 2.3, we show the following property for τ∗.Proposition A.2. For τ∗, we have:1. ∀n,NF, t : vτ∗(NF, n, t) ≥ vpi0(., ., t),84A.2. Analytical Results2. ∀n,NF : vτ∗(NF, n, t)− vpi0(., ., t) is decreasing in t.Proof. Fix NF , arbitrarily. We prove the result by induction on n as follows: n = 1: We have:vτ∗(NF, 1, t)− vpi0(., ., t) ={v(NF, 1, t, 0)− vpi0(., ., t) : t < τ(NF )0 : o.w.The function is decreasing for t < τ(NF ) by Lemma A.10, and for t ≥ τ(NF ) trivially.It suffices to have that vτ∗(NF, 1, t) ≥ vpi0(., ., t), which follows from the fact that τ∗ isoptimal for n = 1. Assume the result holds for n = 1, . . . , l. We prove that it holds for n = l + 1.For t ≥ τ∗(NF ), we have vτ∗(NF, l + 1, t) = vpi0(., ., t), since the two policies coincide.For t < τ∗(NF ), fix M = m, and K = k for the current AVF chance, arbitrarily. Theresult generalizes by taking expectation. Let t′ = t+m+ k and NF ′ = NF + 1, if k = 0,and NF ′ = NF , otherwise. We have:vτ∗(NF, l + 1, t)− vτ∗(NF, 1, t) = S(t, t′)[vτ∗(NF ′, l, t′)− vpi0(., ., t′)]. (A.14)where S(t, t′) represent the probability of survival of the patient until time t′. We canexplain Equation A.14 as follows. The difference, in terms of QALE, between the states(NF, l + 1, t) and (NF, 1, t) under the policy τ∗ does not start until t′, which is realizedonly if the patient survives until t′ with probability S(t, t′). At t′, the patient receivesvτ∗(NF ′, l, t′) for the case we start by l + 1 AVF chances, whereas for the case we startby one AVF chance, the patient switches to a CVC forever at t′ and receives vpi0(., ., t′).Therefore, we havevτ∗(NF, l + 1, t) ≥ vτ∗(NF, 1, t) ≥ vpi0(., ., t),where the first inequality results from Equation A.14 and that vτ∗(NF ′, l, t′) ≥ vpi0(., ., t′)by induction assumption, and the second inequality results from induction basis. Thisproves the first property.Since vτ∗(NF, 1, t) − vpi0(., ., t) is decreasing in t, in order to prove the second property,it suffices to prove that the right-hand side of Equation A.14 is decreasing in t. We proveit by showing that it is the product of the following two non-negative and decreasingfunctions:1. S(t′, t): The probability is non-negative by definition. First we compute S(t′, t) as85A.2. Analytical Resultsfollows:S(t, t′) =P[Ct > m,At+m > k] = P[Ct > m]P[At+m > k|Ct > m]= FCt(m)FAt+m(k),where the last equality follows from Assumption 2.2. Both FCt(m) and FAt+m(k)are decreasing in t because At+x and Ct+x are stochastically decreasing in t, for anyx ≥ 0 by Lemmas A.5 and A.7.2. vτ∗(NF ′, l, t′)− vpi0(., ., t′): This term is non-negative and decreasing in t using theinduction assumption.Proof of Theorem 2.3. We prove the optimality of τ∗(NF ) formed in Proposition A.1 byinduction on n. Note that Proposition A.1 required Assumptions 2.2-2.6 and 2.8. The proofadditionally requires Assumption 2.7 to use Corollary 2.1 and Assumption 2.1 regarding decisionpoints in the model. n = 1: The policy is optimal for n = 1 by construction. Assume the optimality of τ∗(NF ) for n = 1, . . . , l. We prove it for n = l + 1.Fix NF , arbitrarily. We prove the optimality of τ∗ based on whether t ≥ τ∗(NF ) or notas follows.→ t ≥ τ∗(NF ): The policy suggests no more surgeries. We argue its optimality asfollows.We argue that the last l AVF chances will not be used. Note that these AVFs’possible use time will be at some t′ ≥ t and for some NF ′ ≥ NF . Since τ∗ isoptimal for n ≤ l, τ∗(NF ′) ≥ τ∗(NF ) (by Corollary 2.1), and that t′ ≥ t ≥τ∗(NF ) ≥ τ∗(NF ′), these AVF chances will not be used . Thus, we are left withone AVF chance. Similarly, we should not use that chance, either. Thus, the nosurgery decision is optimal in this case.→ t < τ∗(NF ): The policy suggests surgery at t. We argue that it is optimal asfollows.Assume the surgery is planned at t′ := t + y. Note that no surgeries should beperformed later than τ∗(NF ) (using the logic explained in the first case). Thus, werestrict our attention to t′ < τ∗(NF ). For all such t′, we have that v(NF, n, t′, 0) =vτ∗(NF, n, t′), based on the induction assumption. By this equality and Lemma A.8,we havev(NF, n, t, y) = FCt(y)[vτ∗(NF, n, t+ y)− vpi0(., ., t+ y)]+ vpi0(., ., t)86A.2. Analytical ResultsWe conclude the proof by showing v(NF, n, t, y) is decreasing in y. Since FCt(y) isdecreasing in y and non-negative, it suffices to have that vτ∗(NF, n, t+y)−vpi0(., ., t+y) is non-negative and decreasing in y which holds by Proposition A.2, respectively.Proof of Proposition 2.1. Fix NF , arbitrarily. Based on the way the optimal policy isformed in Proposition A.1, we have that for all t ∈ (0, τ∗(NF )), v(NF, 1, t, 0) > vpi0(., ., t) andfor all t ∈ [τ∗(NF ), tmax], we have v(NF, 1, t, 0) ≤ vpi0(., ., t). Since v(NF, 1, t, 0)− vpi0(., ., t) isa decreasing continuous function, we can find τ∗(NF ) using a binary search over [0, tmax]. Proof of Corollary 2.2. By Equation A.9 and Lemma A.10, we have that dcr(NF, t) is de-creasing in t. The result then directly follows Equation A.10. A.2.3 Proofs of Theorems 2.6, 2.7:Proof of Theorem 2.6. Fix Ψ = ψ arbitrarily. The result generalizes using Lemma A.2. LetLT (y) and L(y) be a patient’s residual lifetime at t when the AVF surgery is planned at ywith and without a potential transplant at t = ψ, respectively. We prove that FLT(y)(a) isdecreasing in y for any a. Let Tr(ψ) be the patient’s residual lifetime on transplant at ψ. Wehave:FLT(y)(a) ={FL(y)(a) : a ≤ ψ (A.15a)FL(y)(ψ)FTr(ψ)(a− ψ) : o.w. (A.15b)Equation A.15a follows from the fact that transplant benefits a patient’s survival after thetransplant, and Equation A.15b follows our assumption that the lifetime of a patient on trans-plant does not depend on HD history. The result then follows by Theorem 2.1, which indicatesthat FL(y)(x) is decreasing in y. We use Lemma A.11 to prove Theorem 2.7.Lemma A.11. Consider the random variable Y , a function of the continuous random variableX, defined for X = x as follows:Y (x) ={g(x) : x ≤ θ;g(θ) + U : x > θ.where g is a linear function. Then, we have EY (X) = Eg(X) + FX(θ)[U − Eg(Xθ)]87A.2. Analytical ResultsProof. We haveE[Y (X)− g(X)] = FX(θ)E[Y (X)− g(X)∣∣X > θ] = FX(θ)E[U + g(θ)− g(θ +Xθ)], (A.16)= FX(θ)E[U + g(θ)− g(θ)− g(Xθ)] = FX(θ)E[U − g(Xθ)], (A.17)where the first equality in Equation A.16 follows the total law of probability and the factthat Y = g for X ≤ θ, the second equality follows using the definition of Y and the identityX|X > θ = Xθ + θ, and finally the first equality in Equation A.17 follows by the linearity ofg. Proof of Theorem 2.7. In order to prove the theorem, we only show that Lemma A.10holds under the extended model as well. The rest of the proof follows similar steps taken forProposition A.1 and Theorem 2.3, which require Assumptions 2.1-2.8.Let ν(NF, 1, t, 0) and νpi0(., ., t) be the equivalents of v(NF, 1, t, 0) and vpi0(., ., t), respec-tively, under the model with the transplant option. Since monotonicity preserves under expec-tation, it suffices to prove the result under all possible scenarios (i.e., we use a sample pathargument). Under the scenario where the transplant is canceled, we have v(.) = ν(.) and thusthe result follows using Lemma A.10. Now we consider the case of no cancellation. If the AVFdoes not mature, we have ν(NF, 1, t, 0) − νpi0(., ., t) = −d, as the only difference in QALE isthe AVF creation disutility. It remains to prove the result for the case of a successful AVFcreation.Fix M = m arbitrarily. Let t′ := t+m be the time of switching to the matured AVF, andU be the lump-sum QALE the patient receives from transplant. Using Lemma A.11 and byconsidering g(x) = qAx, X = At′ , and θ = ψ− t′, we can show that the QALE residual at t′ fora patient who is on an AVF from t′ until transplant equals qAEAt′ + FAt′ (ψ− t′)(U − qAEAψ).Similarly, we can show that the QALE residual at t′ for a patient on the CVC equals qCECt′ +FCt′ (ψ − t′)(U − qCECψ).We can calculate ν(NF, 1, t, 0)− νpi0(., ., t) as follows:ν(NF, 1, t, 0)− νpi0(., ., t) =FCt(m)[{qAEAt′ + FAt′ (ψ − t′)(U − qAEAψ)}−{qCECt′ + FCt′ (ψ − t′)(U − qCECψ)}]− d (A.18)Equation A.18 can be explained as follows. The patient experiences a QALE difference startingfrom t′ (AVF maturation time), but only if she survives until then. Therefore the QALEdifference after t′ is discounted by FCt(m). Since FCt(m) is decreasing in t (see the proof ofLemma A.9), it suffices to prove that the term in the brackets, henceforth denoted by ∆, isnon-negative and decreasing in t (or equivalently t′). By rearranging terms, we obtain :∆ =[qAEAt′ − qCECt′]+[FCt′ (ψ − t′)qCECψ − FAt′ (ψ − t′)qAEAψ]+ U[FAt′ (ψ − t′)− FCt′ (ψ − t′)].88A.2. Analytical ResultsWe have:∆ ≥[qAEAt′ − qCECt′]+[FCt′ (ψ − t′)qCECψ − FAt′ (ψ − t′)qAEAψ]≥[qAEAt′ − qCECt′]+ FAt′ (ψ − t′)[qCECψ − qAEAψ]≥ 0.where the first inequality follows since U ≥ 0 and FAt′ (ψ− t′) ≥ FCt′ (ψ− t′) as a consequenceof Lemma A.7, and the second inequality follows because again FAt′ (ψ−t′) ≥ FCt′ (ψ−t′). Wehave qAEAt′ − qCECt′ ≥ FAt′ (ψ − t′)[qAEAψ − qCECψ]and thus the last inequality, becausebased on Theorem 2.2, qAEAt − qCECt is decreasing in t, t′ ≤ ψ, and FAt′ (ψ − t′) ≤ 1.Finally, by rearranging terms in ∆, we can show that it equals the sum of the followingdecreasing functions: qAEAt′ − qCECt′ : This term is decreasing based on Theorem 2.2. −FCt′ (ψ− t′)[qAEAψ−qCECψ]: Since FC(t′) is decreasing and −FCt′ (ψ− t′) = −FC(ψ)FC(t′)by definition, we have that −FCt′ (ψ−t′) is decreasing. Since Cψ ≤st Aψ based on LemmaA.7, then using Assumption 2.8 we can show that qAEAψ ≥ qCECψ. Therefore, the term−FCt′ (ψ − t′)[qAEAψ − qCECψ]is decreasing.[FAt′ (ψ − t′) − FCt′ (ψ − t′)](U − qAEAψ): This term is decreasing because FAt′ (ψ −t′)−FCt′ (ψ − t′) is decreasing based on the theorem assumption, and U ≥ qAEAψ sincewe assume that a patient’s residual QALE on transplant is higher than on HD.89Appendix BChapter 4 Mathematical ProofsB.1 General ResultsLemma B.1. If v(x) increases with x, we have Ev(X) ≤ Ev(Y ), for any X ≤st Y .Proof. See Theorem 2.2.5. in [102]. Lemma B.2. Assume an increasing function f and independent random variables Xi, Yi withXi ≤st Yi. We have f(X1, . . . , Xn) ≤st f(Y1, . . . , Yn).Lemma B.3. The random vectors X and Y satisfy X ≤st Y if, and only if, there exist tworandom vectors Xˆ and Yˆ , defined on the same probability space, such that Xˆ =st X andYˆ =st Y , and we have Xˆ ≤ Yˆ with probability 1.Proof. See Theorem 6.B.1. in [98]. Lemma B.4. LetX1, X2, . . . , Xm be a set of independent random variables and let Y1, Y2, . . . , Ymbe another set of independent random variables. If Xi ≤st Yi for i = 1, . . . ,m, then for anyincreasing function ψ : Rm → R, one has ψ(X1, . . . , Xm) ≤st ψ(Y1, . . . , Ym). In particular,∑Xi ≤st∑Yi.Proof. See Theorem 1.A.3. in [98]. Lemma B.5. Let X, Y be two random variables such that for all realizations of the randomvariable Z, we have [X|Z = z] ≤st [Y |Z = z]. Then, X ≤st Y .Proof. See Theorem 1.2.15 in [100]. Lemma B.6. Let Xa be a family of real-valued random variables parametrized by a ∈ R suchthat Xa ≤st Xa′ whenever a ≤ a′. Then, we have XZ ≤st XY whenever Z ≤st Y .Proof. By Lemma B.3, there exist random variables Y˜ and Z˜ defined on the same samplespace such that Y˜ =st Y , Z˜ =st Z, and Z˜ ≤ Y˜ with probability 1. We also have XZ˜ ≤st XY˜since Xa increases in a in the usual stochastic order and Z˜ ≤ Y˜ with probability 1. Therefore,we have XZ =st XZ˜ ≤st XY˜ =st XY , and thus the result. 90B.2. Analytical ResultsB.2 Analytical ResultsB.2.1 Proofs of Results in Section 4.2:Proof of Proposition 4.1.We prove the result by induction on t. For t = T + 1, we have vt(x) = R(x). Therefore, theinduction basis holds by assumption (b). We now show that the result holds for t, if it hold fort+ 1. Let a∗ be an optimal action under state x at t. Consider any state x′ ≥ x. We have:vt(x) = ra∗t (x) + βEvt+1(x˜a∗t (x))≤ ra∗t (x′) + βEvt+1(x˜a∗t (x))≤ ra∗t (x′) + βEvt+1(x˜a∗t (x′))≤ maxa∈A{rat (x′) + βEvt+1(x˜at (x′))}= vt(x′),where the first inequality follows by assumption (a), the second inequality follows since vt+1(x)increases with x (by induction assumption) and x˜a∗t (x) ≤st x˜a∗t (x′) (assumption (c)). In what follows, define function ∆t(x) for any optimal stopping problem by∆t(x) := rt(x) + βEvt+1(x˜t(x))−Rt(x).Note that in period t stopping is optimal at x if and only if we have ∆t(x) ≤ 0 (for maximizationproblems).Proof of Proposition 4.3:Let ∆it(x) be the ∆t(x) function for problem i, i.e., let∆it(x) := rit(x) + βEvit+1(x˜it(x))−Rit(x).We can show that it suffices to prove that ∆1t (x) ≤ ∆2t (x) as follows. Assume that stoppingis optimal for problem 2 at x in period t, i.e., we have ∆2t (x) ≤ 0. We then have ∆1t (x) ≤∆2t (x) ≤ 0, and therefore, stopping is optimal for problem 1 as well.We now show that ∆1t (x) ≤ ∆2t (x). For t = T , we have ∆it(x) = δit(x) by the definition ofone-step benefit function. Therefore, the induction basis holds by assumption (a). As shownin [99], we can show that ∆t(x) = δt(x) + βEmax[0,∆t+1(x˜t(x))] as follows.∆t(x) =rt(x) + Evt+1(x˜t(x))−Rt(x)=rt(x)−Rt(x) + Emax[rt+1(x˜t(x)) + Evt+2(x˜t+1(x˜t(x))), Rt+1(x˜t(x))] (B.1)=rt(x)−Rt(x) + Emax[Rt+1(x˜t(x)) + ∆t+1(x˜t(x)), Rt+1(x˜t(x))] (B.2)=rt(x)−Rt(x) + ERt+1(x˜t(x)) + Emax[∆t+1(x˜t(x)), 0] (B.3)=δt(x) + βEmax[0,∆t+1(x˜t(x))]. (B.4)91B.2. Analytical Resultswhere the Eq. B.1 follows the definition of vt+1(.), Eq. B.2 follows by the definition of ∆t+1(.),Eq. B.3 follows by re-arranging terms, and Eq. B.4 follows by the definition of δt+1(.).Therefore, we have:∆1t (x) = δ1t (x) + Emax[0,∆1t+1(x˜1t (x))]≤ δ2t (x) + Emax[∆1t+1(x˜1t (x)), 0]≤ δ2t (x) + Emax[∆2t+1(x˜2t (x)), 0] = ∆2t (x),where the first inequality follows since δ1t (x) ≤ δ2t (x) (assumption (a)), and the second inequal-ity follows since ∆1t+1(x) ≤ ∆2t+1(x) for any x (by induction assumption), and x˜1t (x) =st x˜2t (x)for any x (assumption (b)). Proof of Proposition 4.4:It suffices to show that ∆t(x) increases in xi. By the definitions of ∆t and σt we have:∆t(x) = σt(x) + βEvt+1(x˜t(x)).The first term is increasing in xi by assumption (c). The second term increases with xi by thedefinition of stochastic order of random vectors since vt+1 increases with x (assumption (b))and x˜t(x) increases with xi in the usual stochastic order (assumption (a)). B.2.2 Proof of Results in Section 4.3Proof of Lemma 4.1. By re-arranging terms in Eq. 4.1, we obtain:Ba(d, p) =[1 +1− ppfaw(d)fab (d)]−1.Since1− ppdecreases with p andfaw(d)fab (d)increases with d by the definition of the MLR order,the result follows. We use the following lemma to prove Lemma 4.2.Lemma B.7. Assume that we have γab ≤st γaw, for action a. Then, γap decreases in p in theusual stochastic order.Proof. We have fp = pfab + (1− p)faw. Therefore, P[γap ≤ x] = pP[γab ≤ x] + (1− p)P[γaw ≤ x].Since by assumption P[γaw ≤ x] ≤ P[γab ≤ x], we have that P[γap ≤ x] increase with p, andtherefore, the result follows. Proof of Lemma 4.2. We show that x˜a(e, p) stochastically increases with e and p as follows.92B.2. Analytical Results x˜a(e, p) ≤st x˜a(e, p′) for p ≤ p′.Since the MLR order implies the usual stochastic order, we have γab ≤st γaw. Therefore byLemma B.7, we have that γap decreases with p in the usual stochastic order. By LemmaB.3, there exist random variables γ and γ′ defined on the same probability space suchthat γ =st γap , γ′ =st γap′ , and γ′ ≤ γ with probability 1. Therefore, we have that thefollowing holds with probability 1:[e− γ,Ba(γ, p)] ≤ [e− γ′,Ba(γ′, p)] ≤ [e− γ′,Ba(γ′, p′)], (B.5)where the first and second inequalities hold since Ba(d, p) decreases with d and increaseswith p by Lemma 4.1.Since x˜a(e, p) =st [e − γ,Ba(γ, p)] and x˜a(e, p′) =st [e − γ′,Ba(γ′, p′)], the result followsby Lemma B.3 and Eq. B.5. x˜a(e, p) ≤st x˜a(e′, p) for e ≤ e′.We have the following:x˜a(e, p) =st [e− γap ,Ba(γap , p)] ≤ [e′ − γap ,Ba(γap , p)] =st x˜a(e′, p). (B.6)where the inequality holds with probability 1. Therefore, the result follows by LemmaB.3.Proof of Proposition 4.5. We use Proposition 4.1 to show the result. By Lemma 4.2 andassumption (c), we have x˜a(e, p) ≤st x˜a(e′, p′), and hence assumption (c) of Proposition 4.1holds. We have ra(e, p) = prab (e)+(1−p)raw(e). Since rm(e) increases with m and e (assumption(a)), we have ra(e, p) increases with (e, p), and hence assumption (a) of Proposition 4.1 holds.Similarly using assumption (b), we can show that assumption (b) of Proposition 4.1 holds, andthe result follows. Proof of Proposition 4.6. We use Proposition 4.4 to prove the result. By Lemma 4.2 wehave x˜(e, p) increases with e and p in the usual stochastic order. Therefore, assumption (a)of Proposition 4.4 holds. Also, by Proposition 4.5 and assumptions (a,b,c), the value functionincreases with (e, p). Therefore, assumption (b) of Proposition 4.4 holds. We have:σ(e, p) = pσb(e) + (1− p)σw(e)Therefore, σ(e, p) increases with p and e by assumption (d), and thus, assumption (c) ofProposition 4.4 holds. Proof of Proposition 4.7. We use Proposition 4.2 to prove the result. By Lemma 4.2 wehave x˜(e, p) increases with e and p in the usual stochastic order. Therefore, assumption (b) ofProposition 4.2 holds.93B.2. Analytical ResultsIt remains to show that assumption (a) of Proposition 4.2 holds. To that end, it sufficesto show that δ(e, p) = pδb(e) + (1− p)δw(e), since then we have δ(e, p) increases with p and esince δm(e) increases with m and e (assumption (a)).Let [piw(d, p), pib(d, p)] = [1−B(d, p),B(d, p)] and [piw, pib] = [1− p, p] be posterior and priorbelief vectors. We have:δ(e, p) = r(e, p) + Eγp [R(e− γp,B(γp, p))]−R(e, p) (B.7)=∑m=w,bpimrm(e) + Eγp[ ∑m=w,bpim(γp, p)Rm(e− γp)]−∑m=w,bpimRm(e) (B.8)=∑m=w,bpim[rm(e)−Rm(e)] + Eγp[ ∑m=w,bpim(γp, p)Rm(e− γp)](B.9)=∑m=w,bpim[rm(e)−Rm(e)] +∫ [ ∑m=w,bpimfm(x)fp(x)Rm(e− x)fp(x)]dx (B.10)=∑m=w,bpim[rm(e)−Rm(e)] +∫ [ ∑m=w,bpimfm(x)Rm(e− x)]dx (B.11)=∑m=w,bpim[rm(e)−Rm(e)] +∑m=w,bpim∫fm(x)Rm(e− x)dx (B.12)=∑m=w,bpim[rm(e)−Rm(e)] +∑m=w,bpimERm(e− γm) (B.13)=∑m=w,bpimδm(e), (B.14)where Eq. B.7 and Eq. B.8 follow by the definition of δ(x), Eq. B.9 follows by re-arrangingterms, Eq. B.10 follows by the definition of the posterior belief, Eq. B.11 follows by re-arranging terms, Eq. B.12 follows by exchanging the order of expectation and summation,Eq. B.13 follows by the definition of expectation, and Eq. B.14 follows by the definition ofδm(s). Proof of Proposition 4.8. We have:δw(e) = rw(e) + ERw(e− γw)−Rw(e) = rb(e) + ERb(e− γw)−Rb(e)≤ rb(e) + ERb(e− γb)−Rb(e) = δb(e),where the first and last equality follow the definition of δ, the second equality follows assumption(a,b), and the inequality follows by Lemma B.1 since Rb(e) increases with e, and e−γw ≤st e−γb(assumption (c)). 94B.2. Analytical ResultsB.2.3 Proof of Results in Section 4.4:In what follows let δm(e) be the one-step benefit function for patient type m at health state e,i.e., let δm(e) := Edm(e− γm)− dm(e). In Lemma B.9, we show that under the assumptions ofProposition 4.9, δm(e) decreases with e and m. We use Lemma B.8 to prove Lemma B.9.Lemma B.8. Tme increases with e and m in the usual stochastic order, whenever γb ≤st γw.Proof. We need to show that P[Tme ≤ t] decreases with e and m. Let γim be i.i.d. samplesfrom γm. By definition we have:P[Tme ≤ t] = P[ t∑i=1γim ≥ e− ed].It is trivial that P[Tme ≤ t] decreases with e. By Lemma B.4, we have that γb ≤st γw implies∑ti=1 γib ≤st∑ti=1 γiw. Therefore, P[Tme ≤ t] decreases with m, as well. Lemma B.9. Assume:(a) γb ≤r γw,(b) Function c is convex.then, δm(e) decreases with e and m.Proof. Let e′ be the eGFR value in the subsequent period when the eGFR is currently at eand the decision is to continue, i.e., let e′ := e− γm. Also, define L′ as a random variable thatis smaller than L (the lead time until the AVF is ready) by one month, i.e., let L′ := L − 1.Recall that L− Tme represents the difference between the time an AVF becomes available andwhen HD commences. We have L − Tme =st L′ − Tme′ , since after one month, the eGFR hastransitioned from e to e′, and thus we have Tme′ until HD starts, and one month of the AVFpreparation has passed. Therefore, we have dm(e) = Ec(L′− Tme′ ). Let g(x) := c(x)− c(x− 1).We have:δm(e) = Edm(e− γm)− dm(e) = Ec(L− Tme−γm)− Ec(L− 1− Tme−γm)= Eg(L− Tme−γm), (B.15)where the first follows by the definition of δ, the second equality follows by the definition ofdm(e) and the observation above that L − Tme =st L′ − Tme′ , and the last equality follows bythe definition of g. Since the MLR order implies the usual stochastic order, by assumption(a) we have γb ≤st γw, and therefore by Lemma B.8, Tme increases with e and m in the usualstochastic order. Also, since c is convex (assumption (b)), g increases with x.We now prove the claim:→ Monotonicity in e:Choose e ≤ e′ arbitrarily. Since Tme increases with e in the usual stochastic order, we95B.2. Analytical Resultshave:[Tme−γm |γm = d] ≤st [Tme′−γm |γm = d].By Lemma B.5 , we have Tme−γm ≤st Tme′−γm . Therefore,δm(e) = Eg(L− Tme−γm) ≥ Eg(L− Tme′−γm) = δm(e′),where the inequality follows by Lemma B.1 since L−Tme−γm ≥st L−Tme′−γm , and g increaseswith x.→ Monotonicity in m:We first show Twe−γw ≤st T be−γb as follow. Since Tme increases with e in the usual stochasticorder, and γb ≤st γw, we obtain that T be−γw ≤st T be−γb by Lemma B.6. Since for any e,Twe ≤st T be , we have:[Twe−γw |γw = d] ≤st [T be−γw |γw = d].Therefore, by Lemma B.5, we have Twe−γw ≤st T be−γw , and as a result, Twe−γw ≤st T be−γb .We have:δb(e) = Eg(L− T be−γb) ≤ Eg(L− Twe−γw) = δw(e),where the inequality follows by Lemma B.1 since L−Twe−γw ≥st L−T be−γb , and g increaseswith x.We now prove Proposition 4.9.Proof of Proposition 4.9.By Lemma B.9 and assumptions (a,b), we have that δm(e) decreases with e and m. Therefore,the result follows the infinite-horizon, cost minimization version of Proposition 4.7. Proof of Proposition 4.10.We use the infinite horizon version of Proposition 4.3 to show the result. Let δi(e, p) be theone-step benefit function for problem i. Since x˜1(e, p) = x˜2(e, p) by assumption, it sufficesto show that δ1(e, p) ≤ δ2(e, p). In the proof of proposition 4.7, we show that δi(e, p) =pδib(e) + (1− p)δiw(e). Therefore, it suffices to show that δ1m(e) ≤ δ2m(e). By Eq. B.15, we haveδim(e) = Eg(Li − Tme−γm) for some increasing function g. We have:δ1m(e) = Eg(L1 − Tme−γm) ≤ Eg(L2 − Tme−γm) = δ2m(e),where the inequality follows by Lemma B.1 since g is increasing and L1 ≤st L2. 96
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- Optimal treatment planning under consideration of patient...
Open Collections
UBC Theses and Dissertations
Featured Collection
UBC Theses and Dissertations
Optimal treatment planning under consideration of patient heterogeneity and preparation lead-time Skandari, Mohammad Reza 2016
pdf
Page Metadata
Item Metadata
Title | Optimal treatment planning under consideration of patient heterogeneity and preparation lead-time |
Creator |
Skandari, Mohammad Reza |
Publisher | University of British Columbia |
Date Issued | 2016 |
Description | This thesis comprises three chapters with applications of stochastic optimization models to vascular access planning for patients with chronic kidney disease (CKD). Hemodialysis (HD) is the most common treatment for patients with end-stage renal disease, the last stage of CKD. There are two primary types of vascular accesses used for HD, arteriovenous fistula (AVF), and central venous catheter (CVC). An AVF, which is created via a surgical procedure, is often considered the gold standard for delivering HD due to better patient survival and higher quality of life. However, there exists a preparation lead-time for establishing a functional AVF since it takes several months to know whether the surgery was successful, and a majority of AVF surgeries end in failure. In this thesis, we address the question of whether and when to perform AVF surgery on patients with CKD with the aim of finding individualized policies that optimize patient outcomes. In Chapter 2 we focus on vascular access planning for HD dependent patients. Using a continuous-time dynamic programming model and under data-driven assumptions, we establish structural properties of optimal policies that maximize a patient's probability of survival and quality-adjusted life expectancy. We provide further insights for policy makers through our numerical experiments. In Chapter 3 we develop a Monte-Carlo simulation model to address the timing of AVF preparation for progressive CKD patients who have not yet initiated HD. We consider two types of strategies based on approaches suggested in recently published guidelines. We evaluate these strategies over a range of values for each strategy, compare them with respect to different performance metrics (e.g., percentage of patients with an unnecessary AVF creation), and provide policy recommendations. Our simulation results suggest that the timing of AVF referral should be guided by the individual rate of CKD progression. Motivated by our findings in Chapter 3, we develop a dynamic programming model in Chapter 4 that incorporates patient heterogeneity in disease progression when making clinical decisions. We then apply this modeling framework to the case of the AVF preparation timing problem introduced in Chapter 3 and provide recommendations that consider patient heterogeneity in CKD progression. |
Genre |
Thesis/Dissertation |
Type |
Text |
Language | eng |
Date Available | 2016-09-07 |
Provider | Vancouver : University of British Columbia Library |
Rights | Attribution-NonCommercial-NoDerivatives 4.0 International |
DOI | 10.14288/1.0314138 |
URI | http://hdl.handle.net/2429/59079 |
Degree |
Doctor of Philosophy - PhD |
Program |
Business Administration - Management Science |
Affiliation |
Business, Sauder School of |
Degree Grantor | University of British Columbia |
GraduationDate | 2016-11 |
Campus |
UBCV |
Scholarly Level | Graduate |
Rights URI | http://creativecommons.org/licenses/by-nc-nd/4.0/ |
AggregatedSourceRepository | DSpace |
Download
- Media
- 24-ubc_2016_november_skandari_mohammadreza.pdf [ 3.54MB ]
- Metadata
- JSON: 24-1.0314138.json
- JSON-LD: 24-1.0314138-ld.json
- RDF/XML (Pretty): 24-1.0314138-rdf.xml
- RDF/JSON: 24-1.0314138-rdf.json
- Turtle: 24-1.0314138-turtle.txt
- N-Triples: 24-1.0314138-rdf-ntriples.txt
- Original Record: 24-1.0314138-source.json
- Full Text
- 24-1.0314138-fulltext.txt
- Citation
- 24-1.0314138.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.24.1-0314138/manifest