Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Yaghoubi, Marjan
Master of Science - MSc
University of British Columbia
Data sets are becoming massive with ever increasing advances in data collection technologies and are altering the nature of biomedical research. With many techniques, these huge data sets can be challenging, or even impossible, to accurately analyse. In biomedical settings, data sets are frequently heterogeneous, with samples representing various subtypes of diseases that are thought to have variations with respect to underlying biology. A motivating example is the study of progressive diseases such as Alzheimer's disease (AD). While there is a significant increase in the number of studies that concentrate on regression modeling of the disease progression, they ignore the fact that the pattern change are profoundly different for patients with various initial pro les. Estimating separate models for each subgroup is extremely difficult due to small sample sizes in the high dimensional setting, but may obtain results that are more accurate and reliable. Moreover, recognizing homogeneous subgroups of predictors can be cumbersome in high-dimensional regression analysis over subgroups of samples. This thesis attempts to improve upon an established method of regularized regression for group-structured datasets by using a linear combination of two penalty functions to select predictive clusters of correlated variables, and to allow for subgroup-specific parameter estimates. In order to showcase the performance of the suggested methodology, we conducted a series of experiments on Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset including three groups of Cognitively Normal Controls (CN), Late Mild Cognitive Impairment (LMCI), and Alzheimer's disease (AD) subjects to estimate Mini- Mental State Examination (MMSE) scores in multiple future time points. Results reveal the effectiveness of the suggested method in terms of Root Mean Square Error (RMSE) over several available well-known statistical methods in two subgroups, AD and LMCI. However, in CN group, our proposed method performed better than other methods at two time points. We also investigated the prediction performance of our proposed method with multiple multi-task learning regression methods.
Subgroup-Specific Regression ModelsbyMarjan YaghoubiB.Sc., University of Science and Culture, 2010M.Sc., Iran University of Science and Technology, 2014A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Mathematics)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)May 2021© Marjan Yaghoubi, 2021The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Subgroup-Specific Regression Modelssubmitted by Marjan Yaghoubi in partial fulfilment of the requirementsof the degree of Master of ScienceDr. Javad Tavakoli, Irving K. Barber Faculty of ScienceSupervisorDr. Paramjit Gill, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Jeffrey Andrews, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Robert Lalonde, Irving K. Barber Faculty of ScienceUniversity ExamineriiAbstractData sets are becoming massive with ever increasing advances in data col-lection technologies and are altering the nature of biomedical research. Withmany techniques, these huge data sets can be challenging, or even impossi-ble, to accurately analyse. In biomedical settings, data sets are frequentlyheterogeneous, with samples representing various subtypes of diseases thatare thought to have variations with respect to underlying biology. A mo-tivating example is the study of progressive diseases such as Alzheimer’sdisease (AD). While there is a significant increase in the number of studiesthat concentrate on regression modeling of the disease progression, they ig-nore the fact that the pattern change are profoundly different for patientswith various initial profiles. Estimating separate models for each subgroupis extremely difficult due to small sample sizes in the high-dimensional set-ting, but may obtain results that are more accurate and reliable. Moreover,recognizing homogeneous subgroups of predictors can be cumbersome inhigh-dimensional regression analysis over subgroups of samples. This thesisattempts to improve upon an established method of regularized regressionfor group-structured datasets by using a linear combination of two penaltyfunctions to select predictive clusters of correlated variables, and to allowfor subgroup-specific parameter estimates. In order to showcase the perfor-mance of the suggested methodology, we conducted a series of experimentson Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset includingthree groups of Cognitively Normal Controls (CN), Late Mild Cognitive Im-pairment (LMCI), and Alzheimer’s disease (AD) subjects to estimate Mini-Mental State Examination (MMSE) scores in multiple future time points.Results reveal the effectiveness of the suggested method in terms of RootMean Square Error (RMSE) over several available well-known statisticalmethods in two subgroups, AD and LMCI. However, in CN group, our pro-posed method performed better than other methods at two time points. Wealso investigated the prediction performance of our proposed method withmultiple multi-task learning regression methods.iiiLay SummaryThe great progress in data acquisition has resulted in major changesin scientific fields. Biomedical areas are especially fascinating to look at,as data sets in this field are becoming increasingly heterogenous, with sam-ples spanning multiple subgroups that may be expected to have variations inpatterns of association between observed measurements and a response of in-terest. As a motivating example, we consider a study of Alzheimer’s disease(AD). Regression is a method of analysis that is ubiquitous in biomedicalstudies. With regression methods being so prevalent, it is crucial that it isbeing done efficiently and with enhanced accuracy. Along with the rise inthe number of group-structured datasets, it is necessary to develop efficientregression methods to attain subgroup-specific parameters and structure.We begin our work by reviewing several previously presented supervised re-gression frameworks. Then, we modify a current regression method, whichconsider high-dimensional regression in the group-structured setting. Fi-nally, we compare the performance of the model with multiple availablewell-known models in terms of root mean square error and correlation coef-ficients for Alzheimer’s disease dataset.ivTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . vList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.1 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . 21.1.2 Gradient-based Optimization . . . . . . . . . . . . . . 31.2 Subdifferential Calculus . . . . . . . . . . . . . . . . . . . . . 31.3 Proximal Gradient Methods . . . . . . . . . . . . . . . . . . . 41.3.1 FISTA . . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . 61.4.1 Bias and Variance . . . . . . . . . . . . . . . . . . . . 8Chapter 2: Literature Review . . . . . . . . . . . . . . . . . . . 102.1 Linear regression and least squares . . . . . . . . . . . . . . . 102.2 Regularized Least Squares . . . . . . . . . . . . . . . . . . . . 112.2.1 Least absolute shrinkage and selection operator . . . . 112.2.2 Ridge Regression . . . . . . . . . . . . . . . . . . . . . 132.2.3 Elastic Net . . . . . . . . . . . . . . . . . . . . . . . . 132.3 Ensemble Learning . . . . . . . . . . . . . . . . . . . . . . . . 14vTABLE OF CONTENTS2.3.1 Decision Tree . . . . . . . . . . . . . . . . . . . . . . . 142.3.2 Random Forest . . . . . . . . . . . . . . . . . . . . . . 142.3.3 Gradient Boosting Algorithm . . . . . . . . . . . . . . 15Chapter 3: Structured Linear Regression . . . . . . . . . . . . 173.1 Fused Lasso and its generalization . . . . . . . . . . . . . . . 173.2 Joint Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.3 Generalized Joint Lasso (Our Model) . . . . . . . . . . . . . . 193.4 Multi-Task Learning . . . . . . . . . . . . . . . . . . . . . . . 223.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . 25Chapter 4: Data Analysis Results . . . . . . . . . . . . . . . . . 264.1 Current Statistics of Alzheimer’s Disease . . . . . . . . . . . . 264.1.1 Stages of AD . . . . . . . . . . . . . . . . . . . . . . . 274.1.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.2 Problem Description . . . . . . . . . . . . . . . . . . . . . . . 294.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . 304.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . 36Chapter 5: Conclusions . . . . . . . . . . . . . . . . . . . . . . . 38Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44Appendix A: Proximity Operator . . . . . . . . . . . . . . . . . . . 45Appendix B: R Source Code . . . . . . . . . . . . . . . . . . . . . . 46Appendix C: List of Variables . . . . . . . . . . . . . . . . . . . . . 60Appendix D: Degrees of freedom . . . . . . . . . . . . . . . . . . . 63viList of TablesTable 4.1 Summary of subjects demographics considered for thisstudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Table 4.2 Performance comparison of various methods in termsof RMSE for AD class . . . . . . . . . . . . . . . . . . 31Table 4.3 Performance comparison of various methods in termsof RMSE for CN class . . . . . . . . . . . . . . . . . . 32Table 4.4 Performance comparison of various methods in termsof RMSE for LMCI class . . . . . . . . . . . . . . . . . 32Table 4.5 Performance comparison of various methods in termsof CC for AD class . . . . . . . . . . . . . . . . . . . . 33Table 4.6 Performance comparison of various methods in termsof CC for CN class . . . . . . . . . . . . . . . . . . . . 33Table 4.7 Performance comparison of various methods in termsof CC for LMCI class . . . . . . . . . . . . . . . . . . . 33Table 4.8 Performance comparison of various methods in termsof RMSE for AD class . . . . . . . . . . . . . . . . . . 34Table 4.9 Performance comparison of various methods in termsof RMSE for CN class . . . . . . . . . . . . . . . . . . 34Table 4.10 Performance comparison of various methods in termsof RMSE for LMCI class . . . . . . . . . . . . . . . . . 34Table 4.11 Performance comparison of various methods in termsof CC for AD class . . . . . . . . . . . . . . . . . . . . 35Table 4.12 Performance comparison of various methods in termsof CC for CN class . . . . . . . . . . . . . . . . . . . . 36Table 4.13 Performance comparison of various methods in termsof CC for LMCI class . . . . . . . . . . . . . . . . . . . 36Table C.1 List of variables . . . . . . . . . . . . . . . . . . . . . . 60Table C.2 List of variables . . . . . . . . . . . . . . . . . . . . . . 61Table C.3 List of variables . . . . . . . . . . . . . . . . . . . . . . 62viiLIST OF TABLESTable D.1 Degrees of freedom for LMCI class . . . . . . . . . . . 63Table D.2 Degrees of freedom for AD class . . . . . . . . . . . . . 63Table D.3 Degrees of freedom for CN class . . . . . . . . . . . . . 63viiiList of FiguresFigure 1.1 Machine Learning Cycle. . . . . . . . . . . . . . . . . 7Figure 4.1 The change patterns of MMSE score for different studygroups of AD. . . . . . . . . . . . . . . . . . . . . . . 29ixAcknowledgementsI would like to express my sincere gratefulness to my supervisor Dr.Javad Tavakoli for his expert advice, guidance, and support throughout mystudies. I will be forever grateful for the opportunity of being his studentand work under his supervision. I would also express my gratitude to Dr.Paramjit Gill, Dr. Jeffrey Andrews, and Dr. Robert G. Lalonde for takingtime to evaluate my work.xDedicationI would like to thank my husband and family for their support, patience,and motivation.xiChapter 1IntroductionIn this chapter, an overview of two main concepts in mathematics andstatistics is given. The first one is the concept of mathematical optimization,as many recent applied statistics problems can be presented in the contextof convex optimization. The second one is the field of machine learning,which is closely related to the optimization problem.1.1 OptimizationOptimization is so prevalent in real-world application as many of thestrategies adopted by humans seek to minimize a certain cost, or maximize acertain profit for a specific task. An optimization problem involves searchingfor the best element of a certain space (X) with respect to some constraints.The standard form of an optimization problem defined as:minimizex∈X{f(x)} subject to x ∈ C, (1.1)where x are the optimization variables, the objective function f : X →R∪{∞} denotes the cost of selecting x ∈ X, and C ⊂ X is the feasible region,where every acceptable solution must lie in C. The solution of Problem 1.1is the element xop ∈ C such that for every x ∈ C, f(xop) ≤ f(x) is satisfied.Problem 1.1 is a general formulation of a constrained problem, that isthe solution must satisfy a specific criterion. However, it can be relaxed tothe unconstrained problemminimizex∈X{f(x)}. (1.2)In general, every constrained problem can be converted into an uncon-strained problem by making proper adjustments to its objective function[Bec17].In optimization setting, two major concerns are: 1) the problem shouldbe properly defined in terms of the variables, the objective function, and thefeasible region. 2) the optimization problem should be manageable. This11.1. OPTIMIZATIONactively demonstrates that, a balance between the problem’s complexity andthe cost of solving it needs to be found.The space over which the optimization problems discussed in this thesisis a Euclidean space E with inner product, denoted by 〈., .〉 and norm‖.‖. ‖.‖pdenotes the `p norm of its argument. In addition, the objective functions arerestricted to functions on E which can take any real value and∞ ( extendedreal-valued functions).1.1.1 Basic ConceptsIn this section some general concepts associated with convex optimiza-tion are presented. Most of these concepts are included in [Bec17, BV04].Definition 1.1. (Distance Function). The distance function dC : E −→[0,∞) to a nonempty set C ⊂ E is defined by:dC(x) = infc∈C‖x− c‖ . (1.3)Definition 1.2. (Projection). The projection of x ∈ E onto a closed,nonempty set C ⊂ E is the point z = PC(x) ∈ C such that dC(x) =‖x− z‖.Definition 1.3. (Indicator Function). The indicator function of a closedset C ⊂ E is defined as:δC(x) ={0 if x ∈ C,∞ if x /∈ C. (1.4)As stated above, every constained optimization problem can also be ex-pressed as an unconstrained optimization problem using the indicator func-tion:minx∈E{f(x)} s.t. x ∈ C ≡ minx∈E{f(x) + δC(x)}. (1.5)Definition 1.4. (Lower Semi-Continuous Function). f : E −→ R ∪ {∞} islower semi-contininuous or lsc at x ∈ E if the following inequality is hold forevery sequence {xn} −→ x:f(x) ≤ limn→∞ inf f(xn). (1.6)Definition 1.5. (Convex Function). A function f : E −→ R∪{∞} is convexif for all λ ∈ [0, 1], the following inequality is satisfied:(∀x, y ∈ domf) f((1− λ)x+ λy) ≤ (1− λ)f(x) + λf(y). (1.7)21.2. SUBDIFFERENTIAL CALCULUSThe objective functions of the minimization problems tackled in thisthesis are all convex, since a local minimum is also a global minimum in thistype of functions.Definition 1.6. (Strictly Convex Function). A function f : E −→ R∪{∞}is strictly convex if for all λ ∈ (0, 1), the following inequality is satisfied:(∀x, y ∈ domf) f((1− λ)x+ λy) < (1− λ)f(x) + λf(y). (1.8)1.1.2 Gradient-based OptimizationIn many optimization problems, the assumption is that the objectivefunctions are convex and smooth (continuous and differentiable). This allowsto solve problems in the form of (1.2) by methods which are based on thegradient. In this case, a necessary and sufficient optimality condition is∇f(xop) = 0. (1.9)Thus, a solution of Equation 1.9 is equivalent to solving the minimizationproblem in (1.2). In many cases, Equation 1.9 has to be solved numericallyby using iterative algorithms, such as Gradient Descent Methods (GDMs).According to [BV04], the outline of a GDM is given in Algorithm 1. Thisalgorithm alternates between two steps: finding a descent direction, andthe choice of a step size. A natural choice for the descent direction is thenegative gradient d = −∇f(x). The GDM algorithm runs until a stoppingcondition, usually of the form∥∥∇f(x)∥∥2≤ ν, where ν > 0, is satisfied.Algorithm 1: Gradient Descent MethodResult: xop ' arg minx∈X{f(x)}Input: a convex and smooth function f ;Initialization: a starting point x[0] ∈ X;for i = 0, 1, · · · doset a descent direction d[i] ∈ X ;choose a step size t[i] > 0;update x[i+1] := x[i] + t[i]d[i];end1.2 Subdifferential CalculusMany optimization problems of interest are non-differentiable. In suchcases, classical gradient-based techniques cannot be implemented and other31.3. PROXIMAL GRADIENT METHODSmethods such as Proximal Gradient Methods (PGMs) [Bec17] will need tobe utilized. In particular, PGMs generalize the concept of gradient by thoseof subgradient.Definition 1.7. (Subgradient). If f : E → (−∞,∞] is a proper (domf 6= ∅) function, then a subgradient of f at x ∈ E is a vector g ∈ E suchthat(∀y ∈ E) f(x) + 〈g, y − x〉 ≤ f(y). (1.10)Definition 1.8. (Subdifferential). If f : E→ (−∞,∞] is a proper function,then the subdifferntial is defined as:∂f(x) = {g ∈ E| (∀y ∈ E) f(x) + 〈g, y − x〉 ≤ f(y)}. (1.11)According to [Roc96], several properties of the subdifferential are:1. ∂f(x) is either empty or a closed convex set, for x ∈ E.2. for x ∈ E, f is differentiable at x ⇔ ∂f(x) = {∇f(x)}.3. ∂(λf)(x) = λ∂f(x), ∀λ > 0.Theorem 1.9. [Roc96, Fermat’s Rule] The point x ∈ E is a global minimizerof a proper convex function f : E→ (−∞,∞] if and only if 0 ∈ ∂f(x).Proof. Indeed,0 ∈ ∂f(x)⇔ (∀y ∈ E) f(x) + 〈0, y − x〉 ≤ f(y)⇔ (∀y ∈ E) f(x) ≤ f(y)⇔ x is a global minimizer of f.1.3 Proximal Gradient MethodsIn the following, the concept of the prox(imity) operator [Mor65] is de-scribed, which is the fundamental concept in the PGM [BT09].Definition 1.10. (Prox Operator). The proximal mapping or prox operatorof a function f : E→ (−∞,∞] at point x ∈ E is given byproxf (x) = arg minu∈E{12‖u− x‖2 + f(u)}. (1.12)An altenative definition of the prox operator isproxf (x) = (Id + ∂f)−1(x). (1.13)41.3. PROXIMAL GRADIENT METHODSIt is worth noting that, the zeros of the subdifferntial of a proper convexfunction are the fixed points of its prox operators. Therefore, by Fermat’srule, we have:x = arg minu∈Ef(u)⇔ 0 ∈ ∂f(x)⇔ x = proxf (x). (1.14)In the following the PGMs are introduced. Consider the following generalcomposite problemminx∈E{F (x) := f(x) + g(x)}, (1.15)where f : E → (−∞,∞] is a proper closed convex and L-smooth function,and g : E → (−∞,∞] is just proper closed and convex function. PGM isdescribed in Algorithm 2.Algorithm 2: Proximal Gradient MethodResult: x[i+1] ' arg minx∈E{F (x)}initialization: pick a starting point x[0] ∈ E;for i = 0, 1, · · · dochoose a step size t = 1L ;update x[i+1] := prox 1Lg(x[i] − t∇f(x[i]));endPGM consists of a gradient step followed by a prox operator, and theupdate step can be compactly written asx[i+1] = T f,gL (x[i]),where T is a prox-grad operator defined byT f,gL (x) = prox 1Lg(x−1L∇f(x)).1.3.1 FISTAFast Iterative Shrinkage- Thresholding Algorithm (FISTA) [BT09] is alsoused to minimize the sum of a smooth and a non-smooth functions. Inparticular, the objective function is F = f1 + f2, where the smooth term f1is a convex extended real-valued function on E, with Lipschitz gradient ∇f1and Lipschitz constant L. The non -smooth term f2 is a convex, proper andlsc function. Therefore, the optimization problem is:minx∈E{F (x)}= minx∈E{f1(x) + f2(x)}. (1.16)51.4. MACHINE LEARNINGBy Fermat’s rule, xopt is a minimizer of F if and only if0 ∈ ∂F (xopt) = ∂ (f1 + f2) (xopt)= ∇f1(xopt) + ∂f2(xopt)⇔ xopt ∈ xopt + δ∇f1(xopt) + δ∂f2(xopt),∀δ > 0⇔ xopt − δ∇f1(xopt) ∈ xopt + δ∂f2(xopt)⇔ xopt ∈ (Id+ δ∂f2)−1(xopt − δ∇f1(xopt))⇔ xopt = proxδf2(xopt − δ∇f1(xopt)).FISTA summarized in Algorithm 3.Algorithm 3: Fast Iterative Shrinkage- Thresholding AlgorithmResult: x[i+1] ' arg minx∈E{F (x)}Input: f1 with ∇f1 and a Lipschitz constant L , f2;Initialization: x[0] ∈ E;y[0] = x[0];t[0] = 1;for i = 0, 1, · · · dox[i+1] := prox 1Lf2(y[i] − 1L∇f1(y[i]));t[i+1] =1+√1+4(t[i])22 ;y[i+1] = x[i+1] + t[i]−1t[i+1](x[i+1] − x[i]);endBeck and Teboulle in [BT09] showed that the sequence x[i+1] convergesto the minimum of Problem 1.16.1.4 Machine LearningIn statistical and mathematical communities, machine learning (ML)concerns the development and study of systems that can learn from data.According to [DHS01], the design of a ML framework involves several tasks.Figure 1.1 represents the general ML lifecycle, which includes the followingsteps:61.4. MACHINE LEARNINGFigure 1.1: Machine Learning Cycle.Data Acquisition. The data is collected, cleaned, and preprocessedin this step. An instance of this stage is feature normalization.Variable Selection. The selection of significant features for the prob-lem at hand is included in this step. In particular, this step is intended forhigh-dimensional data with a relatively limited number of samples.Model Selection. Model selection is an important step when creatingany sort of statistical model after data collection and feature selection, asthe complexity of the model should be sufficient to learn the undelying datastructure. If the model is too basic, then the relationship between the inputand the output can not be learned. On the other hand, if the model istoo complex, then the model will also learn noisy observation, and for newand unknown observations it will generalize very poorly. These effects arereferred to under-fitting and over-fititing, respectively.Model training. The model that learns from the data is developedthrough a process called training. This step is usually formulated as anoptimization problem.Model Evaluation. In this step, the quality of the model is evaluatedbased on some metrics.In particular, the focus of this thesis is on supervised learning (where aresponse variable is measured), in which a function/process has to be deter-mined from a set of input-output pairs, also known as a training set, thatmap a specific input data to the output . Afterwards, it can be applied toestimate the output for unobserved data. The building blocks of supervisedlearning are model, parameters, and objective function.Model. The model refers to the mathematical framework in which theoutput (prediction) is made from the input. The linear regression model,71.4. MACHINE LEARNINGwhere the prediction is given as a linear combination of weighted inputfeatures, is a common example.Parameters. The parameters are those that should be learnt fromthe data. For instance, in linear regression, the parameters are regressioncoefficients.Objective Function. In order to train the model, we need to definethe objective function to measure how well the model fits to the data. Theobjective functions are usually separated into two terms, a loss function (L),and a penalty (regularization) term (R):Obj(.) = L(.) + γR(.),L measures how predictive our model is with respect to the training data,such as Mean Squared Error (MSE) for regression problems. R measuresand controls the complexity of the model, include additional informationabout the ground truth of the problem. γ is a regularization parameter andthe performance of the model depends on this value. It is responsible for thebalance between the complexity and the accuracy of the model. Therefore,a good choice of γ is important and usually determined by cross validationprocedure.1.4.1 Bias and VarianceBias and variance are the fundamental concepts in the field of ML. Highvariance related to over-fitting, and large bias related to under-fitting. Theseterms, bias and variance or “bias-variance tradeoff” [GBD92], are used todescribe the performance of a model. The inability for an ML method tocapture the true structure that exists in the training data is called bias, andthe difference in fits between datasets is called variance. In practice, theideal algorithm has low bias and low variability. This is done by finding abalance between a simple model and a complex model. Three widely usedmethods for achieving this are regularization, boosting and bagging (theyare further detailed in Chapter 2).The remainder of the thesis is organized as follows:- Chapter 2 provides a literature review for linear regession models, whereregularized learning models play a pivotal role, indicating also how to solvethem under a common approach based on PGMs. Furthermore, there is afocus on ensemble learning models.- Chapter 3 presents details on the structured linear regression models.Moreover, a new regularized approach, called the generalized joint lasso, is81.4. MACHINE LEARNINGproposed. Also, multiple multi-task learning methods present in this chap-ter, which later in chapter 4, we show how these models are successfullyapplied to the real-world, health-related problem, where they can give ac-curate predictions.- Chapter 4 provides results from the proposed method to Alzheimer’sDisease Neuroimaging Initiative (ADNI) data set.- Chapter 5 provides a summary and ideas for future work.9Chapter 2Literature ReviewIn statistical and machine learning communities, linear regression, alsoknown as least squares, is commonly used to predict a numerical response/outputvariable from a set of predictors/features. The output is estimated as aweighted linear combination of the predictors. A line is fit to the data usingleast squares, or in other words, a line that results in the minimum sum ofsquared residuals is found.NotationSuppose a sample of n cases, each represented by a p-dimensional inputvector and a 1-dimensional output vector. Let X = (x1, · · · , xp) ∈ Rn×prepresent the input matrix (matrix of predictors), and let Y ∈ Rn×1 denotethe output vector (response vector). The vector of coefficients of a linearmodel is denoted by β ∈ Rp×1. For the sake of brevity, all the modelsdiscussed in the following section are considered without intercept term.More precisely, the predictors are standardized via removal of the meanand scaling to unit variance and the respose variable is centered such that∑ni=1 yi = 0,∑ni=1 xij = 0, and∑ni=1 x2ij = 1 for j = 1, · · · , p.2.1 Linear regression and least squaresThe least squares approach is commonly used in linear regression prob-lem in order to fit a line to the data. In linear regression problem, we seekto find a coefficient vector β∗ ∈ Rp×1 such thatβ∗ ∈ arg minβ{12‖Y −Xβ‖22}. (2.1)To solve this minimization problem we can take the derivative of (2.1) withrespect to β and set it equal to zero to obtain the normal equationsXTXβ∗ = XTY. (2.2)102.2. REGULARIZED LEAST SQUARESSolving Equation 2.2 for β∗ is equivalent to solving the minimization problemin (2.1). Many methods, both direct and iterative, exist to solve Equation2.2 [Bjo96].Since least squares approach has no control on model complexity, itsperformance will depend on the relation between the dimensionality of thedata (the number of features p) and the sample size n. If the sample size isinsufficient with respect to number of features to estimate these coefficients,then the model cannot be fit naively.In many fields, such as biomedical studies, the problem is often of largedimension (p > n), so direct solvers will not suffice and iterative methodswith regularization will need to be utilized. Regularization is when addi-tional information is included about the desired solution. One option forregularization is to add a penalty term to the function being minimizedwhere the penalty term has a specific structure in order to enforce a de-sired outcome. For example, the `1 norm can be used if sparse solutions aredesired. If a penalty was added to (2.1), we would obtainβ∗ = arg minβ{12‖Y −Xβ‖22 + λR(β)}, (2.3)for some penalty term R(β). A common method for solving Equation 2.3 isthe Fast Iterative Shrinkage-Thresholding Algorithm (FISTA).2.2 Regularized Least SquaresRegularizations are the set of techniques that attempts to reduce thevariance of the model by fitting a function appropriately on the given train-ing set [Fri89]. Common regularized regression techniques include Lasso[Tib96], Ridge [HK70], and Elastic Net [ZH05].2.2.1 Least absolute shrinkage and selection operatorLeast absolute shrinkage and selection operator (Lasso) is a popular typeof regularized regression analysis method, which uses `1 penalty function inorder to alleviate the overfitting problem of least squares and impose spar-sity structure on the vector of regression coefficients. Lasso model enhancesthe prediction accuracy of the resulting statistical model by shrinking theregression coefficients toward zero and set the regression coefficients of irrel-evant variables to zero [HTF09]. Lasso, however, has a tendency to selectonly one variable from a group of highly correlated variables, so it is unable112.2. REGULARIZED LEAST SQUARESto discover grouping information [ZH05]. The Lasso model is defined as thesolution of the optimization problemβˆlasso = arg minβ{12‖Y −Xβ‖22}, s.t. ‖β‖1 ≤ t, (2.4)where t > 0 is a tuning parameter that controls the amount of shrinkage,and‖.‖1 denotes the `1 norm of the coefficients (i.e.,‖β‖1 =∑pj=1 |βj |). Thesmaller value of t will make more of the coefficients to be exactly zero andwill shrink coefficients toward the origin. Equation 2.4 can be written in theLagrangian form asβˆlasso = arg minβ{12‖Y −Xβ‖22 + λ‖β‖1}, (2.5)where λ > 0 is a tuning parameter which controls the strength of the con-straint (sparsity level). The greater amount of shrinkage, more coefficientswith zero value, can be obtained by setting λ to a large value, and thereforea simpler model can be achieved. On the other hand, when λ is small, Lassowould produce similar parameter estimates as least squares approach. Thevalue for λ is determined using cross-validation procedure.For Equation 2.5, the objective function is non-differentiable, hence it fitsnicely in the framework of PGMs; in particular FISTA. In order to applyFISTA, Equation 2.5 is divided into a smooth termf1 and a non-smoothterm f2:minβ{12‖Y −Xβ‖22︸ ︷︷ ︸f1(β)+λ‖β‖1︸ ︷︷ ︸f2(β)}. (2.6)The gradient of f1(β) is ∇f1(β) = (XT y−XTXβ). The Lipschitz constantL is given by the largest eigenvalue of XTX. Thus, FISTA can be appliedonce the prox operator of f2 is obtained. Prox operator of the `1 norm issoft- thresholding (see Appendix A) applied element-wise:softλ(β) =sgn(β1)[|β1| − λ]+sgn(β2)[|β2| − λ]+...sgn(βp)[|βp| − λ]+ .The operator [y]+ := max(y, 0) represents the nonnegative part of y.122.2. REGULARIZED LEAST SQUARES2.2.2 Ridge RegressionRidge regression model is similar to Lasso where the `1 penalty function(∑pj=1 |βj |) is replaced by the `2 penalty (∑pj=1 β2j ). Ridge regression tendsto shrink the correlated predictors toward each other. The correspondingoptimization problem is:βˆridge = arg minβ{‖Y −Xβ‖22 + λ‖β‖22}, (2.7)where ‖.‖22 denotes the Euclidean (`2) norm of the coefficients (i.e., ‖β‖22 =∑pj=1 β2j ), and λ > 0 is a regularization parameter.This problem has a closed-form solution asβˆridge = (XTX + λI)−1XTY, (2.8)where I ∈ Rp×p is the identity matrix. As λ → 0, the solution of Equation2.8 tends to the least squares one . Practically, the `2 norm shrinks thecoefficients towards zero. In particular, when XTX = I, we haveβˆridge =11 + λβ∗.Both Lasso and Ridge regression are ideally suited for high-dimensional data[ZH08] wherein some kind of penalties have been applied to restrict thenumber of predictors entering the model. The principal difference betweenthese two is the geometric shape of the constraint, `1 norm is square and `2norm is circular [Fu98].2.2.3 Elastic NetElastic net (ElNet) combines `1 and `2 penalty functions for group pre-dictor selection, based on the fact that ridge regression tends to shrinkthe correlated predictors toward each other. The Elastic net estimate ofβˆ = (βˆ1, · · · , βˆp) is equivalent to the following constrained optimizationproblemβˆEnet = arg minβ{‖Y −Xβ‖22}, s.t. αp∑j=1|βj |+ (1− α)p∑j=1β2j ≤ t. (2.9)The function α∑pj=1 |βj | + (1 − α)∑pj=1 β2j is called the elastic netpenalty, which is a convex combination of the Lasso and Ridge penalty.t and α ∈ [0, 1] are tuning parameters.132.3. ENSEMBLE LEARNINGEquation 2.9 can be written in the Lagrangian formβˆEnet = arg minβ{‖Y −Xβ‖22 + λ(α‖β‖1 + λ(1− α)‖β‖22)}. (2.10)This optimization problem is also non-differentiable due to the `1 norm.Since `2 norm is differentiable, it can be included ino the smooth term f1,and thus we haveminβ{‖Y −Xβ‖22 + λα‖β‖22︸ ︷︷ ︸f1(β)+λ(1− α)‖β‖1︸ ︷︷ ︸f2(β)}. (2.11)Subsequently, the gradient of f1(β) can be found easily, and the prox oper-ator of f2(β) can be found following the same philosophy discussed in theLasso problem. Therefore, FISTA can be applied.2.3 Ensemble LearningEnsemble learning is an ML paradigm in which an optimal predictivemodel is generated by combining the predictions of multiple weak learn-ers (or base models), resulting in predictions that are more reliable thanany single model. There are two types of Ensemble learning, Boosting andBootstrap Aggregation (or Bagging).2.3.1 Decision TreeDecision trees (DTs) [Bel59] are very popular weak learners for ensemblemethods. Every individual tree has several branches, nodes, and leaves. Itbegins at the very top with the root node that represents entire sample. Thisnode then splits into a left and right nodes (sub-nodes or decision nodes orinternal nodes), and how far this splitting goes is known as the tree depth.The last node in each branch (sub-tree) is called the leaf or terminal node.At the end of the leaf node, the average of the value of the dependent variableis the final prediction.2.3.2 Random ForestRandom Forests (RF) [Bre01] are built from DTs that combine the sim-plicity of decision trees with flexibility which results in substantial improve-ment in prediction accuracy. RF uses straightforward algorithm as follows:142.3. ENSEMBLE LEARNING1. A bootstrapped dataset of the same size as the training set is createdfrom the training dataset.2. A decision tree is formed using the bootstapped dataset. However, arandom subset of features is used when building the tree.3. Repeat step 1 and 2, say 100 times. At this step a wide variety of treesare created, which makes RF more effective and robust than individualdecision trees.4. Estimate the accuracy of the Random Forest.2.3.3 Gradient Boosting AlgorithmOne of the most effective and powerful boosting techniques for buildingpredictive models is Gradient Boosting (GB) [Fri00]. GB for regressionproceeds in the following fashion:1. Input: training set {(xi, yi)}ni=1, and a differentiable Loss FunctionL(yi, F (x))2. Initialize model with a constant value:F0(x) = arg minγn∑i=1L(yi, γ)3. for m = 1 to M :(A) Compute the pseudo- residuals:rim = −[∂L(yi, F (xi))∂F (xi)]F (x)=Fm−1(x)for i = 1, · · · , n.(B) Fit a regression tree to the pseudo-residuals and create terminalnodes Rjm, for j = 1, · · · , Jm(C) For j = 1, · · · , Jm compute:γjm = arg minγ∑xi∈RijL(yi, Fm−1(xi) + γ)(D) Update the model:Fm(x) = Fm−1(x) + νJm∑j=1γmI(x ∈ Rjm).152.3. ENSEMBLE LEARNING4. Output FM (x).The xi’s refer to each row of measurements that is used to predict theresponse variable and the yi’s refer to the response variable measured foreach subject in the dataset. n is the number of subjects in the dataset. Lossfunction L(yi, F (x)) evaluates how well the response variable is predicted,where F (x) is a function that gives the predicted values. In step 3 of GBalgorithm, M trees are built, then the derivative of the loss function withrespect to the predicted value is calculated. In rim, i is the sample numberand m is the index for the tree that is built. In part (B) of step 3, terminalnodes Rj,m are refered to the leaves in our regression tree, where j is theindex for each leaf in the tree. In part (C) of step 3, the output valuesγjm for each leaf is determined by solving the minimization problem wherethe previous prediction is taken into account. xi ∈ Rij means all elementsin that particular leaf node. In practice, the output values are always theaverage of the residuals that end up in the same leaf. In part (D), a newpredictions for each sample is made. These new predictions are based onthe previous predictions, the learning rate ν ∈ [0, 1], and the output valuesfrom the newest tree. A small ν reduces the effect each tree has on the finalprediction, and this improves accuracy in the long run.16Chapter 3Structured Linear RegressionThis chapter introduces the Fused Lasso model under the same paradigmof regularized learning techniques. Furthermore, a modification of a cur-rent regularized regression model, the so-called joint lasso, is developed.This model is based on the Generalized Fused Lasso but for heterogenousdatasets. The corresponding solver algorithm for this model is also included.Finally, this chapter provides details on multi-task learning method.3.1 Fused Lasso and its generalizationTibshirani et al. [TSR+05] introduced Fused Lasso (FL) that imposessparsity in the coefficients, and also accounts for spatial association of pre-dictors. In order to reflect this structure in the model, in addition to the`1 norm, another regularizer is added to the final model, which encouragessimilarity between coefficients corresponding to adjacent features. FL yieldsestimates using the following optimization problemβˆFL = arg minβ12‖Y −Xβ‖22 , s.t.p∑j=1|βj | ≤ t andp−1∑j=1|βj+1 − βj | ≤ s,(3.1)where t and s are tuning parameters. The second constraint (fusion penalty)encourages sparsity in the differences of coefficients. The fusion penalty canbe written as follows:‖Hβ‖1 =p−1∑j=1|βj+1 − βj |, (3.2)where H ∈ R(p−1)×p denotes the differencing matrix defined as:hij =−1 if i = j1 if j = i+ 10 otherwise.173.1. FUSED LASSO AND ITS GENERALIZATIONA natural extension of the FL is to remove the ordering restriction on fea-tures. She in [She10] presented a generalization of the FL, called the Clus-tered Lasso (Classo), to reflect this behaviour. Classo solves the followingoptimization problemβˆclasso = arg minβ12‖Y −Xβ‖22 , s.t.p∑j=1|βj | ≤ t andp∑j<k|βj − βk| ≤ s.(3.3)FL and Classo methods can be unified in the framework of the generalizedlasso (genlasso) [TT11]:βˆgenlasso = arg minβ12‖Y −Xβ‖22 , s.t. ‖Hβ‖1 ≤ t, (3.4)where H ∈ Rm×p is a specified penalty matrix.Authors in [JLL+15] present a variant of the genlasso, called HexagonalOperator for Regression with Shrinkage and Equality Selection (HORSES),that selects positively correlated predictors in high dimensional setting. Infact, HORSES finds a homogeneous subgroup structure within the featurespace. In contrast to the Elnet that puts negatively correlated predictors intothe same group, the regularization term formulation in HORSES encour-ages grouping of positively correlated variables. The shape of the constraintregion for HORSES is hexagonal, which encourages similarity between co-efficients only in y = x direction. The corresponding optimization problemis:βˆHORSES = arg minβ12‖Y −Xβ‖22 , s.t. αp∑j=1|βj |+(1−α)∑j<k|βj−βk| ≤ t,(3.5)where p−1/2 ≤ α ≤ 1. Equation 3.5 can be written in the Lagrangian formas follows:βˆHORSES = arg minβ12‖Y −Xβ‖22 + λα p∑j=1|βj |+ (1− α)∑j<k|βj − βk| .(3.6)HORSES is superior to other current methods in a variety of ways:1. It selects groups of predictors, rather than randomly select one torepresent the entire collinear group as the Lasso does.2. Positively correlated variables are grouped together rather than bothpositively and negatively correlated variables.183.2. JOINT LASSO3.2 Joint LassoIn many applied fields, samples are divided into subgroups that may notbe similar in terms of the underlying regression models. Such concerns canbe seen in biomedical fields, where observations represent disease subtypesin terms of underlying biology, and hence do not have similar patterns ofassociation between predictors and a response variable. If such a structureexists, the model should reflect it. Therefore, Dondelinger et al. in [DM18]proposed joint lasso (JL), which is used to jointly estimate subgroup-specificregression coefficients whilst producing a sparse solution and encouragingsimilarity between subgroup-specific regression coefficients. Joint lasso fitsnicely in heterogenous datasets (group-structured datastes), as it providessubgroup-specific sparsity patterns.Each subgroup k ∈ {1, · · · ,K} considers the same set of p features, withdifferent subgroup sample size nk. Joint lasso yields estimates usingBˆ = arg minB=[β1···βK ]K∑k=1 12nk ‖yk −Xkβk‖22 + λ‖βk‖1 + γ∑k′>kτk,k′‖βk − βk′‖1 ,(3.7)and a variant with an `2 penalty in the last term. λ, γ and τ as tuningparameters and the role of τ is to control the extent of fusion across specificsubgroups. By default, all τ ’s are set to unity.The sum over k in joint lasso is necessary to account for different samplesizes in different groups. Total sample size is n =∑Kk=1 nk. In the aboveformulation, Xk ∈ Rnk×p is the feature matrix for subgroup k and thecorresponding vector of responses, yk ∈ Rnk×1. B = [β1 · · ·βK ] is a p ×Kmatrix that collects all coefficients together. βk ∈ Rp denote subgroup-specific coefficients.3.3 Generalized Joint Lasso (Our Model)Grouping positively correlated variables during variable selection pro-cedure when analysing a system is beneficial. The particular reason forthis is that the variables within each category behave similarly, implyingthat they determine features that affect the system through the same path-ways [JLL+15]. In this thesis, we therefore propose “generalized joint lasso”(GJL) as a modified version of the method outlined in the previous section.In order to simultaneously identify homogeneous subgroups of variables and193.3. GENERALIZED JOINT LASSO (OUR MODEL)jointly estimate the regression coeficients in group-structured data, we in-troduce generalized joint lasso asBˆ = arg minB=[β1···βK ]K∑k=1 12nk ‖yk −Xkβk‖22 + λα‖βk‖1 + (1− α) ∑k′>kτk,k′‖βk − βk′‖1 ,(3.8)where p−1/2 ≤ α ≤ 1. Problem 3.8 is non-differentiable due to having anon-smooth penalty function. [CKL+10] described a proximal relaxationof this kind of problems in multi-task regression setting that introducesadditional smoothing to transform the non-smooth objective function intoa continuously differentiable function. Motivated by their work in [DM18,CKL+10], it is straightforward to adapt their optimization method for theGJL model.First, an undirected graph G = (V,E) with vertex set V = {1, ...,K}corresponding to the subgroup and edges between all vertices is created.The Problem 3.8 can then be formulated asfL(B) =∑k{12nk‖yk −Xkβk‖22}+‖BC‖1 , (3.9)where C = (λ1I, λ2H) is a K × (K + |E|) matrix, with λ1 = λα , λ2 =λ(1 − α), and I ∈ RK×K the identity matrix . The last term in (3.9)includes both `1 and fusion penalties. The matrix H defined asHk,e =τm,l if e = (m, l) and k = m−τm,l if e = (m, l) and k = l0 otherwise.Because of duality between `1 and `∞, the overall penalty in Equa-tion 3.9 can be written as ‖BC‖1 = max‖A‖∞≤1〈A,BC〉, where A ∈ Q ={A | ‖A‖∞ ≤ 1, A ∈ Rp×(K+|E|)} is an auxiliary matrix, and ‖.‖∞ is thematrix `∞ norm, defined as the maximum absolute value of all entries. Fol-lowing Chen et al. in [CKL+10], a smooth approximation of ‖BC‖1 can bewritten asfµ(B) = max‖A‖∞≤1〈A,BC〉 − µd(A), (3.10)where µ > 0, and d(A) ≡ 12‖A‖2F , with ‖.‖F the Frobenius norm. Theorem1 in [CKL+10] shows that fµ(B) is smooth with gradient ∇fµ(B) = A∗CT ,where A∗ is the optimal solution of Problem 3.10, and the closed-form ex-pressions of it are introduced in the following lemma.203.3. GENERALIZED JOINT LASSO (OUR MODEL)Lemma 3.1. Let A∗ be the optimal solution of Equation 3.10:A∗ = S(BC/µ),whereS(x) =x if −1 < x < 11 if x ≥ 1−1 if x ≤ −1.Substituting ‖BC‖1 with fµ(B) in Equation 3.9, we havef˜L(B) =∑k{12nk‖yk −Xkβk‖22}+ fµ(B), (3.11)which is smooth with∇f˜L(B) =∑k{1nkXTk (Xkβk − yk)}+A∗CT . (3.12)Moreover, ∇f˜L(B) is Lipschitz continuous with an upper bound of the Lip-schitz constantLU = λmax(XTk Xk) +λ21 + 2 ∗ λ22 maxk∈V dkµ, (3.13)where λmax(XTk Xk) is the largest eigenvalue of (XTk Xk) and dk =∑Kk′ τk,k′ .The result of the aforementioned procedure is that a smooth lower boundof fL(B), namely f˜L(B), is derived. Finally, the Nesterov’s method [Nes05]can be applied for optimizing (3.11) as shown in Algorithm 4. For moredetails on this optimization approach see [CKL+10].Algorithm 4: Proximal gradient method for the generalized jointlassoResult: Bˆ = B[i]Input:X,Y, λ1, λ2, G, µ;Initialization: buid C = (λ1I, λ2H); find LU according to (3.13);W [0] = 0 ∈ Rp×K ;for i = 0, 1, · · · until covergence of B[i] doCompute ∇f˜(W [i]) according to Equation 3.12;Compute the gradient descent step: B[i] = W [i] − 1LU∇f˜(W [i]) ;Set Z [i] = − 1LU∑im=0m+12 ∇f˜(W [i]) ;Set W [i+1] = i+1i+3B[i] + 2i+3Z[i];end213.4. MULTI-TASK LEARNING3.4 Multi-Task LearningA standard methodology in ML is learning one task at a time (single-taskanalysis). However, multi-task learning (MTL) [Car98] has gained growingdegree of attention to solve multiple learning tasks simultaneously. Thegoal of MTL is to learn the shared information among related tasks in orderto enhance prediction accuracy [LJY09]. This section will discuss multiplemulti-task learning techniques for predicting multiple related outcomes froma common set of predictors.Consider a multi-task learning setting with t response variables (tasks).Suppose that p is the number of predictors, which is shared across all thetasks, and n is the number of samples. Let X ∈ Rn×p indicate the matrixof predictors, Y ∈ Rn×t represent a matrix of t responses over the sameset of observations. B ∈ Rp×t denote the parameter matrix, with columnβ.i ∈ Rp corresponds to task i, i = 1, · · · , t, row βj. ∈ Rt corresponds tofeature j = 1, · · · , p, and yk denotes the k-th column of Y .The representative multitask learning methods used for comparison inChapter 4 include the following models:1. Multi-task feature learning with `2,1 norm: [LJY09] introduced anMTL framework using a regularization based on the `2,1 norm‖B‖2,1 =p∑j=1∥∥βj.∥∥2 ,where `2 encourages grouping (refers to the grouping effect of `2 norm)of the weights corresponding to the j-th feature across several tasks.Thus, the complete MTL model via the `2,1-norm regularization is:minB{12‖Y −XB‖2F + λ‖B‖2,1}. (3.14)The formulation of the `2,1 norm encourages multiple input featuresfrom different tasks to share similar sparsity patterns for parameters,and the selection of predictors is based on the joint strength of alltasks jointly.2. Multi-task ElasticNet [PVG+11]: The objective function for this modelisminB{12‖Y −XB‖2F + αρ‖B‖2,1 + α(1− ρ)/2‖B‖2F }, (3.15)where 0 < ρ ≤ 1. The penalty function of Multi-task ElasticNet is acombination of `2,1 and `2.223.4. MULTI-TASK LEARNING3. Trace-Norm Regularized Multi-Task Learning: [JY09] proposed a for-mulation of MTL based on trace norm in order to discover the low-rank common subspace among different tasks. Hence, the optimizationproblem becomesminB{12‖Y −XB‖2F + λ‖B‖∗}, (3.16)where ‖.‖∗ = denotes the trace norm defined as the sum of all thesingular values of B. The effect of trace norm regularization is toforce B to have a low rank [PTJY09]. The low-rank regularization is auseful technique for obtaining the common low-dimensional subspacefor several tasks [HSH16].4. Multi-task learning with network incorporation: [EMP05] introducedthe regularization functionJ(β) =12T∑s,t=1Ast‖βs − βk‖2 ,where T is the number of tasks and A is the graph adjacency matrix tocapture the task similarities. Thus, the optimization problem becomesminB12{‖Y −XB‖2F + λJ(β)}. (3.17)5. Graph-Guided Fused Lasso (GFLasso) [KSX09]: One potential struc-ture on the response variables that would be useful to capture is somecorrelation structure that relates correlated responses. The behaviourof the coefficients should reflect this structure, in the sense that thecoefficients corresponding to highly correlated responses should be sim-ilar. The large limitation of techniques like Lasso is that they mustbe repeated over every single response variable. In other words, Lassofor multiple correlated responses is equivalent to solving multiple in-dependent regressions for each response. Thus, the information acrossmultiple responses is not integerated into the model such that the es-timates reflect the possible relatedeness in the regression coefficientsfor those correlated responses [KSX09]. In the GFLasso, the correla-tion structure of tasks is represented as a graph, with each task as anode, and the relationship between two tasks as an edge. This methodlinks several response variables in a single regularized linear regressionframework, and jointly analyze them.233.4. MULTI-TASK LEARNINGThe correlation structure over the set of t tasks is presented by anundirected graph G = (V,E), where V represents the set of nodes,each representing one of the t tasks, and E refers to the set of edges.Each edge ek,k′ ∈ E corresponds to an edge from the k-th task to thek′-th task, and |rk,k′ | encodes the strength of the correlation betweenthese two nodes. The pairwise Pearson correlation coefficients for eachpair of tasks is computed, and then two nodes are linked with anedge only if their correlation coefficient is above a given threshold τ .The authors introduced two variants of the GFLasso: GcFLasso andGwFLasso. The first version uses unweighted graph and the secondone takes into account the edge weights information in G.If two tasks are connected with an edge in the graph G, their variationacross observations might be explained by the same set of featureswith similar strength. In GcFLasso, two regression coefficients β.k andβ.k′ for each feature m are fused if tasks k and k′ are connected in G.The GcFLasso solvesBˆ = arg minB{12‖Y −XB‖2F + λ1‖B‖1 + λ2∑(k,k′)∈E∥∥β.k − sign(rk,k′)β.k′∥∥1},(3.18)where λ1 and λ2 are regularization parameters, and the last term (thesummation term) is called a fusion penalty. A natural extension ofGcFLasso is GwFLasso wherein the edge weights is taken into account.Also, in GwFLasso, the amount of correlation controls the amount offusion. Thus, the complete optimization problem for GwFLasso is:Bˆ = arg minB{12‖Y −XB‖2F + λ1‖B‖1+ λ2∑(k,k′)∈Ef(rkk′)∥∥β.k − sign(rk,k′)β.k′∥∥1}. (3.19)Possible choices for f(rkk′) could be |rk,k′ | and r2k,k′ . For comparisonresults in Chapter 4, we used |rk,k′ |.For implementation of Multi-task ElasticNet model, we use Scikit-learn[PVG+11]. For other aformentioned multi-task methods, we make use of theRMTL package [CZS18] available for use in the R programming language.243.5. CHAPTER SUMMARY3.5 Chapter SummaryThis chapter has introduced Generalized Joint Lasso (GJL) for high-dimensional regression in the group-structured setting that simultaneouslyselects positively correlated variables and provides subgroup-specific coef-ficient estimates. The corresponding optimization procedure for solvingthis model was also introduced. Finally, this chapter has reviewed severalmulti-task models, which are later used to show the usefulness of multi-tasklearnign methods in biomedical problems.25Chapter 4Data Analysis ResultsThis chapter will discuss a real-world application of the regularized linearregression models, ensemble models, and multi-task regression models inthe field of Alzheimer’s disease research. It is shown how these previouslypresented models can be used to predict the progression of the disease. Inthis setting, these models fit nicely, since 1) the dimensionality of the featurespace can be very large relative to the number of observations; and 2) thesparse models might be preferred, since they are more interpretable thanother more complicated models. For implementation of these models, wemake use of the R programming language [Tea20] and Scikit-learn machinelearning library in Python [PVG+11]. The source code for all models isprovided in Appendix B.4.1 Current Statistics of Alzheimer’s DiseaseThe most prevalent type of dementia in adults aged 65 and above isAlzheimer’s disease (AD), which predominantly affects memory and cogni-tive functions. In the United States, it is ranked the fifth leading cause ofdeath for those age 65 and older [Ass17]. AD has brought a major burden tothe health care system and economy. According to Alzheimer’s Disease In-ternational (ADI) Report [Int19], the global prevalence of AD is expected toincrease to 152 million by 2050. Moreover, dementia including Alzheimer’sdisease, is expected to cost more than 1.1 trillion by 2050 for the UnitedStates alone [Ass17]. These statistics may seem surprising, but as the pop-ulation of seniors is increasingly growing, they are projected to increase astime goes on.AD was first discovered by a German clinical psychiatrist and neu-roanatomist, Dr. Alois Alzheimer, in the early 20th century [HN03]. ADis characterized by gradual loss of cognitive functioning, namely thinking,remembering, and reasoning, which impeded the ability of an individual toperform activities of daily living [oA20]. AD is known to progress over timeby destroying memory cells in the brain, resulting in loss of memory. Todaymore than ever it is crucial to understand the stages of the AD in order to264.1. CURRENT STATISTICS OF ALZHEIMER’S DISEASEimprove the lives of those affected by the disease and establish a clear planon how its prevelance can be predicted and detected.4.1.1 Stages of ADHuman brains start shrinking and losing weight as they begin to age.The brain not only shrinks in Alzheimer’s patients, but also starts to formknots. These knots are chemical-releasing twisted protein fibers that destroythe nerve cells in the brain. Such changes affect the ability of a person tolearn and recall [Naz11]. The disease advances from mild to moderate to se-vere cognitive impairment and the speed of symptoms vary from individualto individual [Ass17]. While several factors such age, genetics, and diet havebeen hypothetically linked to the onset of AD, so far, there are no reliablebiomarkers used to diagnose the early stages of AD [Arm13]. Scientists sys-tematically seek to validate these hypotheses by testing disease progression,with the goal of improving cognitive function of those affected by the diseaseand developing successful medications and treatments, at least, to delay theprogression of the disease [Ass17, Int19].Conducting brain scans, such as magnetic resonance imaging (MRI) andpositron emission tomography (PET) have already been demonstrated toprovide support in the direct observation and inspection of brain abnor-malities such as cerebral atrophy [CRS10]. A convenient clinical diagnos-tic technique is to perform neuropsychological examinations, such as theMini-Mental State Examination (MMSE), and the Alzheimer’s Disease As-sessment Scale cognitive total score (ADAS) that can be used to recognizedisease-related abnormalities. The reliable link between these clinical scoresand AD prognosis has been shown in several studies [DCG+09]. Predictingcognitive performance of patients from neuroimaging feautures is importantfocuse of the study of AD and is recieving a growing degree of attentionrecently.4.1.2 DataAlzheimer’s Disease Neuroimaging Initiative (ADNI)1 database is a lon-gitudinal study designed to collect demographic, imaging, clinical assess-ment, and genetic data from participants who are tracked and reassessedover time to follow the pathology of the progression of the disease. Sev-eral researchers carried out studies [LCW+19, LGC+17] using the ADNIdatabase to identify regions of interest (ROI) in the brain that should be1http://adni.loni.usc.edu/274.1. CURRENT STATISTICS OF ALZHEIMER’S DISEASErevelant to the detection of AD. The first phase of ADNI (ADNI1) startedin 2004 with a follow-up period of five years, ADNI-GO phase began in2009 and followed participants for two years, in 2011 ADNI2 started. Re-cently, ADNI3, the fourth phase, began in 2016 and includes scientists at59 research clinics in the United States and Canada [ADN20]. Progressionand prediction of AD at multiple time points are more important than everbefore. Techniques to describe AD progression can allow clinicians to cre-ate new treatments and track their effectiveness. In ADNI, all participantsreceived 1.5-T structural MRI. This dataset encourages scientists to designnew strategies for reliably predicting patients’ future status. At baseline, pa-tients were classified as either cognitively normal (CN), early mild cognitiveimpairment (EMCI), late mild cognitive impairment (LMCI), and severecognitive impairment (dementia or AD). The MMSE score range of [0, 30]is commonly used as a disease progression indicator, where lower scores in-dicate greater cognitive disability [FFM75]. During the MMSE, a medicalexpert asks a patient a series of questions and tests designed to assess avariety of daily mental abilities, such as time and place orientation, focusand calculation, immediate and delayed recall of words, and language func-tions [LCW+19]. The MMSE scores of patients are measured repeatedly atmultiple time points. A MMSE score of 20 to 24 suggests EMCI, 13 to 20suggests LMCI, and less than 12 indicates AD [Ass17]. Table 4.1 summa-rizes the demographic characteristics of all subjects, including age, gender,and education at baseline.In this thesis we use the baseline MRI features (see Appendix C for de-tailed list of predictors) processed by a team from UCSF using FreeSurfer im-age analysis suite (https://surfer.nmr.mgh.harvard.edu/) and baseline MMSEas inputs and MMSE scores at multiple time points as output variables. Thedataset that we use in this work includes CN, LMCI and AD subgroups. ForCN and LMCI groups, data is availabe for all time points, but for AD group,no data is available for M36.Table 4.1: Summary of subjects demographics considered for this studySubjects CN LMCI ADNumber (F/M) 110/119 141/257 91/100Age(y, mean ± sd) 75.87 ±5.01 74.73 ± 7.38 75.26 ± 7.46Education (y, mean ± sd) 16.06 ± 2.85 15.63 ± 3.03 14.69 ± 3.15284.2. PROBLEM DESCRIPTION4.2 Problem DescriptionThe main goal of this thesis is to accurately predict the progressionof Alzheimer’s disease at the different stages of the disease. Inferring thetrajectories of AD progression over time is important than ever before andis the basis for the implementation of appropriate and prompt therapeuticstrategies that may be subject-specific. However, the majority of studies onAD have focused on the prediction of disease progression at a single pointin time. These studies ignore the dependency structure that exists betweensubsequent time points [TAS+18]. Since the patterns of AD progressionrely heavily on the time of diagnosis, the demographic information of thepatients, and several other features, a single regression model imposed onall of the patients with wide ranges of baseline data may be mis-specified.Fig 4.1 illustrates the change patterns of MMSE scores for several patientsin three different study groups (CN, LMCI , and AD) over the three-yeartime period. As one can see from the figure below, it is obvious that theprogression of the disease does not follow a steady pattern, making it difficultto develop an effective model.Figure 4.1: The change patterns of MMSE score for different study groupsof AD.Additionally, to handle missing data, we calculate the Euclidean distance294.3. RESULTS AND DISCUSSIONbetween the observations with missing values and all other observations andthen select the first five closest subjects to those with missing values andreplace the missing value with the average measurements of those subjects.4.3 Results and DiscussionIn this work, we investigate whether baseline MRI features can be usedto predict AD progression measured by MMSE scores. Specifically, we usethe baseline MRI features and baseline MMSE score to predict MMSE scoresin the next 3 years. We further performed the following preprocessing steps:− exclude features with more than 15 percent missing entries;− remove patients without baseline MRI data;− exclude patients with missing value of MMSE scores;− complete the missing entries using the technique we discussed in Sec-tion 4.2.Since neurodegeneration in AD occurs years before symptoms appear andclinical intervention is more successful in the early stages of the disease, it isnecessary to accurately predict cognitive scores, such as MMSE [ZYLY11].So, our objective is to predict the MMSE scores at different time points withour method, and compare the relative performance of various approaches.The speed of the progression of AD have a strong dependence on the initialstages of a patient. If so, establishing a single regression model on all ofthe subjects with different initial patterns may be mis-specified. To addressthis issue, we adopted a framework which models each subgroup of theAD separately. We evaluate the performance of our proposed method withother regression techniques, including Lasso, Ridge, Elastic net, RF, GB,and JL. The results are presented in Tables 4.2 to 4.7. We indicate eachtime point in the tables by the duration starting from the baseline. Forexample, M06 denotes 6 months after the baseline, M12 indicates 12 monthsafter the baseline, and so on. We use MRI at baseline to predict MMSEscores at four time points: M06, M12, M24, M36. 10-fold cross validationis used to evaluate the performance of the models. The data were z-scoredbefore applying the regression methods. For the quantitative performanceevaluation, we employed the metric of Root Mean Squared Error (RMSE)and Correlation Coefficient (CC) between the predicted MMSE scores andthe target MMSE scores for each regression task. Smaller values of RMSE304.3. RESULTS AND DISCUSSIONand larger values of CC indicate better regression performance. From theexperimental results in Tables 4.2, 4.3, and 4.4, we observe the following:1. For AD group, GJL and JL both demonstrated an improved perfor-mance over the other methods, such as Lasso, Ridge, ElNet, RF, GB,in terms of RMSE , while GJL performed the best among all compet-ing methods for M12 and M24. JL performed better than GJL forM06. Overall, The GJL offers substantial gains compared with othermethods for this subgroup. The biggest gain with GJL method is forthe AD subgroup at M12 and M24.2. For LMCI group, GJL method outperformed JL methods in all timepoints. Moreover, GJL demonstrated an improved performance overLasso, Ridge, ElNet, RF, and GB methods in M12, M24, and M36time points; however, Lasso and ElNet performed better for M06.3. It can be seen that a notable difference between JL and GJL is in theLMCI and AD subgroups, or in other words, the largest improvementin prediction performance is in AD and LMCI subgroups.4. For CN group, GJL performed better than JL in M24 and M36. Itcan be seen that Lasso, Ridge, ElNet, RF, and GB were best suitedto predict CN status than the JL and GJL models for M06, M12.5. Overall, the GJL approach seems to perform slightly better than theJL approach for M24 and M36 in CN group.6. There is a decrease in performance compared to the Lasso, Ridge,ElNet, RF, and GB methods for JL and GJL methods in CN subgroup.Table 4.2: Performance comparison of various methods in terms of RMSEfor AD classMethod M06 M12 M24Lasso 3.38 4.09 4.68Ridge 3.68 4.29 4.96ElNet 3.38 4.11 4.67RF 3.71 4.42 5.46GB 3.63 4.32 5.28JL 2.42 3.01 4.87GJL 2.59 2.84 3.20314.3. RESULTS AND DISCUSSIONTable 4.3: Performance comparison of various methods in terms of RMSEfor CN classMethod M06 M12 M24 M36Lasso 1.00 0.91 1.12 1.30Ridge 1.00 0.90 1.16 1.29ElNet 0.99 0.90 1.11 1.29RF 1.00 0.90 1.15 1.31GB 1.00 0.90 1.14 1.32JL 1.04 1.29 1.11 1.55GJL 1.08 1.30 1.09 1.31Table 4.4: Performance comparison of various methods in terms of RMSEfor LMCI classMethod M06 M12 M24 M36Lasso 2.10 2.52 3.39 3.82Ridge 2.21 2.57 3.58 4.01ElNet 2.10 2.45 3.39 3.87RF 2.26 2.58 3.62 4.02GB 2.14 2.48 3.53 3.64JL 2.41 2.48 3.36 3.47GJL 2.21 2.36 2.80 3.10The correlation coefficients between the actual outcomes and the predic-tions are shown in Tables 4.5, 4.6, and 4.7. For AD class, both JL and GJLdemonstrated an improved performance over the other competing methodsfor M12, while ElNet performed the best among all competing methods forM24. For LMCI group, all other methods performed better than JL andGJL methods for M12, M24, and M36. For M06, both JL and GJL havebetter CC. For CN group, GJL demonstrated an improved performance overthe other competing methods for M06 and M12, while ElNet and Ridge per-formed better than other methods for M24 and M36, respectively.324.3. RESULTS AND DISCUSSIONTable 4.5: Performance comparison of various methods in terms of CC forAD classMethod M06 M12 M24Lasso 0.58 0.55 0.67Ridge 0.43 0.48 0.65ElNet 0.57 0.52 0.69RF 0.44 0.47 0.58GB 0.44 0.49 0.56JL 0.52 0.62 0.51GJL 0.54 0.61 0.60Table 4.6: Performance comparison of various methods in terms of CC forCN classMethod M06 M12 M24 M36Lasso 0.14 0.06 0.21 0.12Ridge 0.09 0.06 0.02 0.15ElNet 0.14 0.058 0.26 0.14RF 0.08 0.12 0.09 0.04GB 0.09 0.05 0.16 0.00JL 0.09 0.01 0.26 0.04GJL 0.16 0.11 0.19 0.03Table 4.7: Performance comparison of various methods in terms of CC forLMCI classMethod M06 M12 M24 M36Lasso 0.52 0.56 0.63 0.64Ridge 0.44 0.54 0.57 0.62ElNet 0.52 0.58 0.63 0.63RF 0.40 0.54 0.55 0.62GB 0.50 0.55 0.59 0.70JL 0.59 0.42 0.39 0.44GJL 0.52 0.50 0.47 0.42Tables 4.8, 4.9, and 4.10 show the Root Mean Squared Error (RMSE)between the predicted MMSE scores and the actual RMSE scores for eachmulti-task regression models discussed in chapter 3. We consider the pre-334.3. RESULTS AND DISCUSSIONdiction of the MMSE score at each time point as a task. We emphasizethat the purpose of the following is to illustrate the usefulness of multi-tasklearning methods for data analysis and prediction in AD.Table 4.8: Performance comparison of various methods in terms of RMSEfor AD classMethod M06 M12 M24MultiElNet 2.99 4.04 4.57L21 3.64 4.46 5.50low rank 3.71 4.64 5.47network structure 4.00 4.85 6.14GFLasso 1.23 1.12 0.94Table 4.9: Performance comparison of various methods in terms of RMSEfor CN classMethod M06 M12 M24 M36MultiElNet 1.03 1.18 1.17 1.26L21 0.80 1.05 0.96 1.52low rank 0.79 1.07 0.99 1.52network structure 0.78 1.06 0.96 1.52GFLasso 1.29 1.36 1.29 1.51Table 4.10: Performance comparison of various methods in terms of RMSEfor LMCI classMethod M06 M12 M24 M36MultiElNet 2.40 2.79 3.77 4.38L21 2.32 2.31 3.09 3.86low rank 2.28 2.32 3.11 3.76network structure 2.75 2.77 3.87 4.46GFLasso 1.09 1.03 1.04 0.98From the exprimental results in Tables 4.2, 4.3, 4.4, 4.8, 4.9, and 4.10,we observe the following:1. In AD subgroup, the GFLasso largely outperformed all other com-peting multi-task learning analyses. Moreover, GFLasso method im-344.3. RESULTS AND DISCUSSIONproved predictive performance over the independent regression meth-ods (Lasso, Ridge, ElNet, RF, GB, JL, and GJL) in this subgroup.2. Our method (GJL) outperformed MultiElNet, L2,1, low rank, and net-work structure models in AD class for all time points.3. There is a decrease in performance compared to all other mutli-tasklearning methods for GFLasso in CN subgroup.4. GJL outperformed GFLasso in CN group at all time points, however, itseems that MultiElNet, L2,1, low rank, and network structure modelsperformed better than GJL in CN group.5. In LMCI subgroup, the GFLasso outperformed all other competingmulti-task learning analyses. Furthermore, GFLasso method improvedpredictive performance over the all independent regression methods inthis subgroup.6. GJL method performed better than MultiElNet, L2,1, low rank, andnetwork structure models at M06, M24, and M36 time points in LMCIsubgroup.Tables 4.11, 4.12, and 4.13 show the corrleation coefficient (CC) betweenthe predicted MMSE scores and the actual RMSE scores.Table 4.11: Performance comparison of various methods in terms of CC forAD classMethod M06 M12 M24MultiElNet 0.49 0.52 0.60L21 0.33 0.31 0.50low rank 0.30 0.28 0.51network structure -0.28 -0.36 0.43GFLasso 0.18 0.14 0.50354.4. CHAPTER SUMMARYTable 4.12: Performance comparison of various methods in terms of CC forCN classMethod M06 M12 M24 M36MultiElNet 0.04 -0.03 0.02 0.08L21 -0.02 0.13 0.14 0.03low rank 0.02 -0.01 0.05 0.07network structure 0.16 0.02 -0.00 -0.04GFLasso 0.03 0.01 0.09 0.07Table 4.13: Performance comparison of various methods in terms of CC forLMCI classMethod M06 M12 M24 M36MultiElNet 0.54 0.53 0.61 0.56L21 0.55 0.52 0.60 0.55low rank 0.56 0.52 0.60 0.60network structure -0.36 -0.28 0.11 0.47GFLasso 0.46 0.48 0.49 0.52It can be seen that MultiElNet method performed the best among allcompeting multi-task methods in terms of CC in AD subgroup for all timepoints. For LMCI group, MultiElNet method performed better than allcompeting methods at M12 and M24. From Tables 4.11 and 4.5 , we observethat all single-task learning methods such as Lasso, Ridge, ElNet, RF, GB,JL, and GJL, peformed better than all multi-task learning methods in termsof CC in AD class .4.4 Chapter SummaryIn this chapter, subjects who underwent MRI, and neuropsychologicaltests such as MMSE have been selected from ADNI dataset. The MMSEscore is the target variable to be predicted with our method. In order toshow the impact of GJL, we compared the predictive performance of ourproposed model with multiple state-of-the-art regression models. We alsoinclude several available well-known multi-task learning methods to showthe usefulness of these learning techniques in health-related problem suchas predicting the cognitive scores of patients with Alzheimer’s disease atmultiple future time points. Overall, the results show that there is not one364.4. CHAPTER SUMMARYsingle method outperforming the others, but instead a group of three topmethods including JL, GJL, and GFlasso that would all be a suitable choice.37Chapter 5ConclusionsLarge datasets can be broken into smaller datasets, that may have simil-larities but not necessarily identically distributed. This thesis proposedand investigated a high-dimensional regression learning framework over sub-groups of observations to support predicting MMSE cognitive scores forsubjects with Alzheimer’s disease at multiple future time points. For eachAlzheimer’s disease stages, separate sets of regression models are trainedto increase the prediction accuracy. More specifically, We modified sev-eral calculations from a previously published approach called ”joint lasso”by Dondelinger and Mukherjee [DM18]. We provided a ”generalized” ver-sion of joint lasso which groups positively correlated variables together andproduces a sparse solution for group-structured datasets, and also providessubgroup-specific regression coefficient estimates.We studied the prediction performance of our proposed method withother linear and nonlinear single-task learning methods such as Ridge, Lasso,Elastic net, Random Forest, Gradient boosting, and joint lasso regressiontechniques. We also investigated the prediction performance of our proposedmethod with several multi-task learning models.It is further noteworthy that the joint lasso and generalzed joint lassohave the advantage of allowing for subgroup-specific sparsity patterns andparameter estimates, which can be of scientific interest. We further note thatif groups in dataset only slightly different, establishing one single regressionmodel may be more effective, or in other words, the choice of pooling or notpooling analysis wil be data-dependent.This work is based on linear models, but methods with nonlinear kernelfunction can be used to model the output scores as nonlinear functionsof input measurements, which can provide additional insights to interpretdata. Moreover, future work should provide details about why GJL appearsto perform better than the other single-task learning methods in later timepoints.38Bibliography[ADN20] ADNI. Alzheimer’s disease neuroimaging initiative(http://adni.loni.usc.edu/) [online]. 2020. → pages 28[Arm13] Richard A. Armstrong. What causes alzheimer’s disease? FoliaNeuropathol, 51(3):169–188, 2013. → pages 27[Ass17] Alzheimer’s Association. Alzheimer’s disease facts and figures.Alzheimer’s & Dementia, 13(4):325–373, 2017. → pages 26, 27,28[Bec17] Amir Beck. First-order methods in optimization. Society forIndustrial and Applied Mathematics, 2017. → pages 1, 2, 4[Bel59] William A. Belson. Matching and prediction on the principle ofbiological classification. Journal of the Royal Statistical Society.Series C (Applied Statistics), 8(2):65–75, 1959. → pages 14[Bjo96] Ake Bjorck. Numerical methods for least squares problems. So-ciety for Industrial and Applied Mathematics, 1996. → pages11[Bre01] Leo Breiman. Random forests. Machine Learning, 45(1):5–32,2001. → pages 14[BT09] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Jour-nal on Imaging Sciences, 2(1):183–202, 2009. → pages 4, 5, 6[BV04] Stephen Boyd and Lieven Vandenberghe. Convex Optimization.Cambridge University Press, 2004. → pages 2, 3[Car98] Rich Caruana. Multitask learning. In Learning to Learn.Springer, pages 95–133, 1998. → pages 22[CKL+10] Xi Chen, Seyoung Kim, Qihang Lin, Jaime G. Carbonell, andEric p. Xing. Graph-structured multi-task regression and an39Bibliographyefficient optimization method for general fused lasso. arXiv,1005.3579, 2010. → pages 20, 21[CRS10] Rudy J. Castellani, Raj K. Rolston, and Mark A. Smith.Alzheimer’s disease? Disease-a-Month, 56(9):484–546, 2010. →pages 27[CZS18] Han Cao, Jiayu Zhou, and Emanuel Schwarz. Rmtl: An r libraryfor multi-task learning. Bioinformatics, 35(10):1797–1798, 2018.→ pages 24[DCG+09] Simon Duchesne, Anna Caroli, Cristina Geroldi, D. LouisCollins, and Giovanni. B. Frisoni. Relating one-year cognitivechange in mild cognitive impairment to baseline mri features.Neuroimage, 47(4):1363–1370, 2009. → pages 27[DHS01] Richard O. Duda, Peter E. Hart, and David G. Stork. PatternClassification. Wiley, 2001. → pages 6[DM18] Frank Dondelinger and Sach Mukherjee. The joint lasso: high-dimensional regression for group structured data. Biostatistics,21(2):219–235, 2018. → pages 19, 20, 38[EMP05] Theodoros Evgeniou, Charles A. Micchelli, and MassimilianoPontil. Learning multiple tasks with kernel methods. Journal ofMachine Learning Research, 6(1):615–637, 2005. → pages 23[FFM75] M. F. Folstein, S. E. Folstein, and P.R. McHugh. “mini-mentalstate” a practical method for grading the cognitive state ofpatients for the clinician. Journal of Psychiatric Research,12(3):189–198, 1975. → pages 28[Fri89] Jerome H. Friedman. Regularized discriminant analysis. Journalof the American Statistical Association, 84(405):165–175, 1989.→ pages 11[Fri00] Jerome H. Friedman. Greedy function approximation: A gradi-ent boosting machine. Annals of Statistics, 29:1189–1232, 2000.→ pages 15[Fu98] Wenjiang J. Fu. Penalized regressions: The bridge versusthe lasso. Journal of Computational and Graphical Statistics,7(3):397–416, 1998. → pages 1340Bibliography[GBD92] Stuart Geman, Elie Bienenstock, and Rene Doursat. Neuralnetworks and the bias/variance dilemma. Neural Computation,4(1):1–58, 1992. → pages 8[HK70] Arthur E. Hoerl and Robert W. Kannard. Ridge regression:biased estimation for nonorthogonal problems. Technometrics,12(1):55–67, 1970. → pages 11[HN03] Hanns Hippius and Gabriele Neundo¨rfer. The discoveryof alzheimer’s disease. Dialogues in Clinical Neuroscience,5(1):101–108, 2003. → pages 26[HSH16] Zhouyuan Huo, Dinggang Shen, and Heng Huang. New multi-task learning model to predict alzheimer’s disease cognitive as-sessment. Medical Image Computing and Computer-Assisted In-tervention, pages 317–325, 2016. → pages 23[HTF09] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. Theelements of statistical learning- data mining, inference and pre-diction. Springer Series in Statistics. Springer, New York, NY,USA, 2009. → pages 11[Int19] Alzheimer’s Disease International. World alzheimer report 2019:Attitudes to dementia. Alzheimer’s Disease International (ADI),2019. → pages 26, 27[JLL+15] Woncheol Jang, Johan Lim, Nicole A. Lazar, Ji Meng Loh, andDonghyeon Yu. Some properties of generalized fused lasso andits applications to high dimensional data. Journal of the KoreanStatistical Society, 44(3):352–365, 2015. → pages 18, 19[JY09] Shuiwang Ji and Jieping Ye. An accelerated gradient methodfor trace norm minimization. In Proceedings of the 26th annualinternational conference on machine learning. ACM, pages 457–464, 2009. → pages 23[KSX09] Seyoung Kim, Kyung-Ah Sohn, and Eric P. Xing. A multivariateregression approach to association analysis of a quantitative traitnetwork. Bioinformatics, 25(12):204–212, 2009. → pages 23[LCW+19] Xiaoli Liu, Peng Cao, Jianzhong Wang, Jun Kong, and DazheZhao. Fused group lasso regularized multi-task feature learningand its application to the cognitive performance prediction of41Bibliographyalzheimer’s disease. Neuroinformatics, 17(4):271–294, 2019. →pages 27, 28, 60[LGC+17] Xiaoli Liu, Andre R. Goncalves, Peng Cao, Dazhe Zhao, andArindam Banerjee. Modeling alzheimer’s disease cognitive scoresusing multi-task sparse group lasso. Computerized Medical Imag-ing and ics, 66:100–114, 2017. → pages 27[LJY09] Jun Liu, Shiwang Ji, and Jieping Ye. Multi-task feature learningvia efficient `2,1-norm minimization. In Proceedings of the 25thconference on uncertainty in artificial intelligence, pages 339–348, 2009. → pages 22[Mor65] J.J. Moreau. Proximite´ et dualite´ dans un espace hilbertien.Bulletin de la S. M. F., 93(2):273–299, 1965. → pages 4[Naz11] Linda Nazarko. Understanding dementia: diagnosis and devel-opment. British Journal of Healthcare Assistants, 5(5):216–220,2011. → pages 27[Nes05] Yurii Nesterov. Smooth minimization of non-smooth functions.Mathematical Programming, 103(1):127–152, 2005. → pages 21[oA20] National Institute of Aging. Alzheimer’s disease fact sheet. Re-trieved from Alzheimer’s Disease Education and Referral Center(ADEAR) website, 2020. → pages 26[PTJY09] Ting Kei Pong, Paul Tseng, Shuiwang Ji, and Jieping Ye. Tracenorm regularization: Reformulations, algorithms, and multi-task learning. SIAM Journal on Optimization, 20(6):3465–3489,2009. → pages 23[PVG+11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau,M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Ma-chine learning in Python. Journal of Machine Learning Research,12:2825–2830, 2011. → pages 22, 24, 26[Roc96] R. Tyrrell Rockafellar. Convex analysis. Princeton UniversityPress (Princeton Landmarks in Mathematics and Physics), 1996.→ pages 442Bibliography[She10] Yiyuan She. Sparse regression with exact clustering. ElectronicJournal of Statistics, 4:1055–1096, 2010. → pages 18[TAS+18] Solale Tabarestani, Maryamossadat Aghili, Mehdi Shojaie,Christian Freytes, and Malek Adjouadi. Profile-specific regres-sion model for progression prediction of alzheimer’s disease usinglongitudinal data. 2018 17th IEEE International Conference onMachine Learning and Applications (ICMLA), pages 1353–1357,2018. → pages 29[Tea20] R Core Team. R: A Language and Environment for StatisticalComputing. R Foundation for Statistical Computing, Vienna,Austria, 2020. → pages 26[Tib96] R. Tibshirani. Regression shrinkage and selection via the lasso.Journal of the Royal Statistical Society. Series B ( Methodolog-ical), 58(1):267–288, 1996. → pages 11[TSR+05] Robert Tibshirani, Michael Saunders, Saharon Rosset, Ji Zhu,and Keith Knight. Sparsity and smoothness via the fused lasso.Journal of the Royal Statistical Society Series B (StatisticalMethodology), 67(1):91–108, 2005. → pages 17[TT11] Ryan J. Tibshirani and Jonathan Taylor. The solution pathof the generalized lasso. Annals of Statistics, 39(3):1335–1371,2011. → pages 18[ZH05] Hui Zou and Trevor Hastie. Regularization and variable selectionvia the elastic net. Journal of the Royal Statistical Society, SeriesB, 67:301–320, 2005. → pages 11, 12[ZH08] Cun-Hui Zhang and Jian Huang. The sparsity and bias of thelasso selection in high-dimensional linear regression. Annals ofStatistics, 36(4):1567–1594, 2008. → pages 13[ZYLY11] Jiayu Zhou, Lei Yuan, Jun Liu, and Jieping Ye. A multi-tasklearning formulation for predicting disease progression. in Pro-ceedings of the 17th ACM SIGKDD International Conference onKnowledge Discovery and Data Mining, pages 814–822, 2011. →pages 3043Appendix44Appendix AProximity OperatorProx operator of the `1 norm has a closed-form expression.If f(x) = λ|x| where λ > 0 on R, then∂f(x) =−λ if x < 0[−λ, λ] if x = 0λ if x > 0proxf (y) = sgn(y) max{|y| − λ, 0} =y − λ if y > λ,0 if |y| ≤ λ,y + λ if y < −λ.This function is called the soft thresholding.From defintion, we knowproxf (y) = arg minx∈E{12‖x− y‖2 + f(x)},andx = proxf (y)↔ y − x ∈ ∂f(x) (∗).There are three cases:Case 1: |y| ≤ λ −→ y − 0 ∈ [−λ, λ] = ∂f(0) and so (∗) holds.Case 2: y > λ , then y− (y−λ) = λ, therefore (∗) holds with x = y−λ.Case 3: y < −λ, similar to case2.45Appendix BR Source CodeThe following is the code used for filling NA values.46Appendix B. R Source CodeThe following code is used for implementing Ridge regression.47Appendix B. R Source CodeThe following code is used for implementing Lasso regression.48Appendix B. R Source CodeThe following code is used for implementing Elastic Net regression.49Appendix B. R Source CodeThe following code is used for implementing Random Forest for regres-sion.50Appendix B. R Source CodeThe following code is used for implementing Gradient Boost for regres-sion.51Appendix B. R Source CodeThe following code is used for implementing JL and GJL models.52Appendix B. R Source Code53Appendix B. R Source Code54Appendix B. R Source CodeThe following code is used for implementing multi-task elastic net re-gression.55Appendix B. R Source CodeThe following code is used for implementing Graph-Guided Fused Lassowith the gflasso R package.56Appendix B. R Source CodeThe following code is used for implementing multi-task learning with `2,1norm with the RMTL R package.57Appendix B. R Source CodeThe following code is used for implementing low rank multi-task learningwith the RMTL R package.58Appendix B. R Source CodeThe following code is used for implementing multi-task learning withnetwork structure with the RMTL R package.59Appendix CList of VariablesThe names of predictors are listed in Tables C.1, C.2, and C.3. TA, TS,SA, and CV stand for thickness average, standard deviation of thickness,surface area, and volume of the cortical and subcortical regions of the brain,respectively. Laterality denotes the various types of features measured for L(left hemisphere), R (right hemisphere), and Bilateral (whole hemisphere)[LCW+19]. For more details on how the FreeSurfer is being used to extarctfeatures of MRI scans see [LCW+19]. The MMSE score at baseline, thecortical thickness average (TA) of left Isthmus Cingulate, and the volume ofright Inferior Parietal play important roles in predicting MMSE at all timepoints.Table C.1: List of variablesName Type LateralityBanks superior temporal sulcus CV, SA, TA, TS L, RCaudal anterior cingulate cortex CV, SA, TA, TS L, RCaudal middle frontal gyrus CV, SA, TA, TS L, RCuneus cortex CV, SA, TA, TS L, REntorhinal cortex CV, SA, TA, TS L, RFrontal pole CV, SA, TA, TS L, RFusiform gyrus CV, SA, TA, TS L, RInferior parietal cortex CV, SA, TA, TS L, RInferior temporal gyrus CV, SA, TA, TS L, RInsula CV, SA, TA, TS L, RIsthmusCingulate CV, SA, TA, TS L, RLateral occipital cortex CV, SA, TA, TS L, RLateral orbital frontal cortex CV, SA, TA, TS L, RLingual gyrus CV, SA, TA, TS L, RMedial orbital frontal cortex CV, SA, TA, TS L, RMiddle temporal gyru CV, SA, TA, TS L, RParacentral lobule CV, SA, TA, TS L, R60Appendix C. List of VariablesTable C.2: List of variablesName Type LateralityParahippocampal gyrus CV, SA, TA, TS L, RPars opercularis CV, SA, TA, TS L, RPars orbitalis CV, SA, TA, TS L, RPars triangularis CV, SA, TA, TS L, RPericalcarine cortex CV, SA, TA, TS L, RPostcentral gyrus CV, SA, TA, TS L, RPosterior cingulate cortex CV, SA, TA, TS L, RPrecentral gyrus CV, SA, TA, TS L, RPrecuneus cortex CV, SA, TA, TS L, RRostral anterior cingulate cortex CV, SA, TA, TS L, RRostral middle frontal gyrus CV, SA, TA, TS L, RSuperior frontal gyrus CV, SA, TA, TS L, RSuperior parietal cortex CV, SA, TA, TS L, RSuperior temporal gyrus CV, SA, TA, TS L, RSupramarginal gyrus CV, SA, TA, TS L, RTemporal pole CV, SA, TA, TS L, RTransverse temporal cortex CV, SA, TA, TS L, RHemisphere SA L, RTotal intracranial volume CV BilateralAccumbens area SV L, RAmygdala SV L, RCaudate SV L, RCerebellum cortex SV L, RCerebellum white matter SV L, RCerebral cortex SV L, RCerebral white matter SV L, RChoroid plexus SV L, RHippocampus SV L, RInferior lateral ventricle SV L, RLateral ventricle SV L, RPallidum SV L, RPutamen SV L, RThalamus SV L, RVentricle diencephalon SV L, RVessel SV L, R61Appendix C. List of VariablesTable C.3: List of variablesName Type LateralityBrain stem SV BilateralCorpus callosum anterior SV BilateralCorpus callosum central SV BilateralCorpus callosum middle anterior SV BilateralCorpus callosum middle posterior SV BilateralCorpus callosum posterior SV BilateralCerebrospinal fluid SV BilateralFourth ventricle SV BilateralNon white matter hypointensities SV BilateralOptic chiasm SV BilateralThird ventricle SV BilateralWhite matter hypointensities SV Bilateral.62Appendix DDegrees of freedomTables D.1, D.2, and D.3 present the number of variables retained foreach sparse method in LMCI, AD, and CN subgroups.Table D.1: Degrees of freedom for LMCI classMethod Lasso ElNet JL GJLdf M06 7 19 24 167df M12 11 11 22 168df M24 9 11 44 44df M36 19 21 39 43Table D.2: Degrees of freedom for AD classMethod Lasso ElNet JL GJLdf M06 16 117 26 166df M12 16 111 29 168df M24 18 150 44 50Table D.3: Degrees of freedom for CN classMethod Lasso ElNet JL GJLdf M06 1 149 13 167df M12 2 1 21 168df M24 11 8 44 54df M36 17 34 39 6963
Thesis/Dissertation
eng
Mathematics
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Tavakoli, Javad
Subgroup-specific regression models
Text