Arts and Social Sciences, Irving K. Barber Faculty of (Okanagan)
Psychology, Department of (Okanagan)
DSpace
UBCO
Armstrong, Graham
Master of Arts - MA
University of British Columbia
Stroke affects 50,000 Canadians every year and millions of people worldwide. Stroke occurs when the brain tissue is damaged by being deprived of essential compounds from the blood stream. Patients with stroke frequently experience a variety of impairments including prolonged cognitive deficits. Neuropsychological assessment is the most effective way of measuring the nature and magnitude of cognitive impairments. One of the main functions of neuropsychological assessment in a rehabilitation setting is to inform clinical decision-making regarding patient treatment trajectories. Recommendations provided by a neuropsychologist depend on the utility of the neuropsychological battery for predicting the patient’s levels of daily functioning. Daily functioning assesses an individual’s ability to complete tasks of daily life. However, this research has lacked the necessary specificity to capture the comprehensive association between cognition and functional abilities. Further, domain specific score practices, regularly employed by clinicians, can lead to substantial levels of misclassification. The goal of the present study was to evaluate machine learning multiple linear regression relative to ordinary least squares (OLS) on a multivariate level as well as at the level of individual predictors, the clinical utility of a comprehensive neuropsychological battery, lastly the potential impact of an adjunctive clinical decision algorithm in a rehabilitation setting was examined. Data were taken from 167 neuropsychological assessments from patients with stroke who participated in rehabilitation at Kelowna General Hospital. The results indicate that a broad neuropsychological assessment accounts for a significant level of post stroke daily functioning scores. Machine learning is a more powerful tool for identifying individual cognitive predictors of stroke than traditional OLS methods. Machine learning did not provide incremental improvement over OLS methods at a multivariate level. The adjunctive clinical decision-making algorithm did not provide sufficient clinical decision input in this setting. Keywords: Stroke, Neuropsychological Assessment, Functional Outcome, Supervised Machine Learning
Neuropsychology’s Machine Assistant: Predicting Functional Outcomes with Machine Learning By Graham Armstrong A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in THE COLLEGE OF GRADUATE STUDIES (Psychology) THE UNIVERSITY OF BRITISH COLUMBIA (Okanagan) January 2023 © Graham Armstrong, 2023 ii The following individuals certify that they have read, and recommend to the College of Graduate Studies for acceptance, a thesis entitled: Neuropsychology’s Machine Assistant: Predicting Functional Outcomes with Machine Learning submitted by Graham Armstrong in partial fulfillment of the requirements of the degree of Master of Arts. Dr. Brian O’Connor, Department of Psychology Supervisor Dr. Maya Libben, Department of Psychology Supervisory Committee Member Dr. Harry Miller, Department of Psychology Supervisory Committee Member Dr. Brian Dalton, School of Health and Exercises Science University Examiner Dr. Zoë Soon, Department of Biology Neutral Chair iii Abstract Stroke affects 50,000 Canadians every year and millions of people worldwide. Stroke occurs when the brain tissue is damaged by being deprived of essential compounds from the blood stream. Patients with stroke frequently experience a variety of impairments including prolonged cognitive deficits. Neuropsychological assessment is the most effective way of measuring the nature and magnitude of cognitive impairments. One of the main functions of neuropsychological assessment in a rehabilitation setting is to inform clinical decision-making regarding patient treatment trajectories. Recommendations provided by a neuropsychologist depend on the utility of the neuropsychological battery for predicting the patient’s levels of daily functioning. Daily functioning assesses an individual’s ability to complete tasks of daily life. However, this research has lacked the necessary specificity to capture the comprehensive association between cognition and functional abilities. Further, domain specific score practices, regularly employed by clinicians, can lead to substantial levels of misclassification. The goal of the present study was to evaluate machine learning multiple linear regression relative to ordinary least squares (OLS) on a multivariate level as well as at the level of individual predictors, the clinical utility of a comprehensive neuropsychological battery, lastly the potential impact of an adjunctive clinical decision algorithm in a rehabilitation setting was examined. Data were taken from 167 neuropsychological assessments from patients with stroke who participated in rehabilitation at Kelowna General Hospital. The results indicate that a broad neuropsychological assessment accounts for a significant level of post stroke daily functioning scores. Machine learning is a more powerful tool for identifying individual cognitive predictors of stroke than traditional OLS methods. Machine learning did not provide incremental improvement over OLS methods at a multivariate level. The adjunctive clinical decision-making algorithm did not provide sufficient clinical decision input in this setting. Keywords: Stroke, Neuropsychological Assessment, Functional Outcome, Supervised Machine Learning iv Lay Summary This thesis explored four research questions, firstly it examined the relative usefulness of applying machine learning statistics compared to common ordinary least squares regression amongst clinical neuropsychological practice in stroke rehabilitation care amongst individual cognitive measures and a neuropsychological assessment battery. Also, this thesis examined the clinical utility of a neuropsychological assessment battery for predicting post stroke daily functioning. Lastly, the potential clinical usefulness of using Machine Learning to identify specific cognitive predictors of how a patient with stroke functions daily was examined. The results indicate that machine learning does not provide incremental utility above ordinary least squares in this research context but could be a useful tool for identifying individual cognitive prognosticators of post stroke functioning. This thesis confirms the clinical utility of a neuropsychological assessment battery amongst stroke patients in a rehabilitation setting. This research indicates the potential of statistical decision making to impact stroke rehabilitation practices. v Table of Contents Abstract .............................................................................................................................. iii Lay Summary ..................................................................................................................... iv Table of Contents ................................................................................................................ v List of Tables .................................................................................................................... vii List of Figures .................................................................................................................. viii Acknowledgements ............................................................................................................ ix Chapter 1: Introduction ....................................................................................................... 1 Stroke .......................................................................................................................... 1 Rehabilitation .............................................................................................................. 1 Cognitive Assessment ................................................................................................. 2 Neuropsychological Assessment ................................................................................ 3 Functional Outcomes .................................................................................................. 5 Previous Literature on Neuropsychological Assessment and Functional Outcomes .. 7 Optimizing Neuropsychological Assessment ........................................................... 11 Machine Learning and Supervised Learning ............................................................ 15 The Present Study ..................................................................................................... 19 Chapter 2: Body of Thesis ................................................................................................ 20 Method .......................................................................................................................... 20 Participants ................................................................................................................ 20 Procedure .................................................................................................................. 21 Measures ................................................................................................................... 21 The Functional Independence Measure. ............................................................... 21 Model Predictors. .................................................................................................. 22 Missing Data ............................................................................................................. 25 Data Analysis ............................................................................................................ 25 Random Forest. ................................................................................................. 32 Ridge Regression. ............................................................................................. 32 Lasso Regression. ............................................................................................. 32 Elastic Net Regression. ......................................................................................... 32 Results ........................................................................................................................... 33 Missing Data ............................................................................................................. 35 Multiple Regression .................................................................................................. 37 Individual Predictor Analysis ................................................................................... 38 Multivariate Predictive Utility .................................................................................. 40 Chapter 3: Conclusion....................................................................................................... 41 vi Discussion ..................................................................................................................... 41 Research Aim and Findings ...................................................................................... 41 Missing Data ............................................................................................................. 42 Neuropsychological Battery Predicting Functional Outcome .................................. 43 Machine Learning ..................................................................................................... 44 Machine Learning Versus OLS Regression .............................................................. 45 Individual Cognitive Facets ...................................................................................... 46 Adjunctive Clinical Decision Algorithm .................................................................. 48 Limitations and Future Research .............................................................................. 50 Bibliography ................................................................................................................. 54 vii List of Tables Table 1. Descriptive Statistics of the sample, mean and standard deviation for age adn education as well as sex compisition. ........................................................................................... 33 Table 2. The battery of neuropsychological measures as well as teh cognitive domain that each test assesses........................................................................................................................... 34 Table 3. Mean scores and fraction of missing information scores for variances on each predictor. ....................................................................................................................................... 36 Table 4. Results of the different regression models, including Ridge, Lasso, Elastic Net, Random Forest and Ordinary Least Squares methods. ................................................................. 38 viii List of Figures Figure 1. Percentage of missing data within each predictor. ........................................................ 35 Figure 2. RMSE across levels of lambda, the penalty term for penalized ridge regression. ........ 37 Figure 3. Individual factor coefficients across linear methods with original FIM score outcome…………………………………………………………………………………………..40 Figure 4. Coefficients across linear methods with second FIM reading as outcome. .................. 39 ix Acknowledgements I would like to acknowledge Dr. Brian O’Connor for his contributions across a variety of domains especially statistical methodology. I would also like to acknowledge the work of Dr. Maya Libben and Dr. Harry Miller in contributing to the clinical elements of this work. Chapter 1: Introduction Stroke is a highly prevalent disorder, both in Canada and globally, which can significantly impact physical and cognitive functioning. Typically, patients who have experienced a stroke participate in rehabilitative care soon after. A neuropsychologist is a key member of any stroke rehabilitation team (Miller et al., 2010). A significant component of a neuropsychologist’s role in this capacity is a cognitive evaluation, which includes a battery of norm-based neuropsychological (NP) assessments. The NP assessment is used to evaluate the patient’s cognitive status across multiple domains; one of its functions is to predict the patient’s level of post-stroke functioning. However, most studies in this space of cognition and predicting functional outcomes have not employed cognitive assessments which are broad enough to capture the full range of cognitive facets. Further, there could be a dearth of research that investigates the optimization of NP assessment battery compositions and score interpretation practices for the rehabilitation environment. For instance, the composition of a standard NP assessment battery has not been optimized to predict a patient’s daily functioning abilities. Additionally, score interpretation practices could be undermining the clinical utility of a NP assessment in the rehabilitation environment. NP assessment scores can be subject to bias, partly because there is no formal agreement among neuropsychologists about the definition of abnormality (i.e., cut off scores). For instance, the minimum distance which defines abnormality between a patient’s score and the mean of a normative distribution can vary across methods (i.e., 1 SD vs 2 SD). This evaluation issue can lead to differential classification of patients who have similar abilities. The goal of this thesis was to explore the potential clinical and research impacts of machine multiple regression algorithms when NP assessments are used to predict post stroke functioning, as measured by the Functional Independence Measure (FIM; MACHINE LEARNING AND NEUROPSYCHOLOGICAL ASSESSMENT Keith, Granger, Hamilton & Sherwin, 1987), amongst rehabilitation patients. This introduction will be structured in three sections. Firstly, background information will be provided on stroke, rehabilitation and cognitive assessment. Second, the literature related to this research will be discussed followed by information on machine learning (ML). Lastly, the methods of this proposed study will be outlined, including the participants, procedure, measures and data analysis methods. Stroke Each year approximately 50,000 Canadians, one million Americans, and millions more people worldwide are hospitalized due to a stroke (Mozaffarian et al., 2015). Stroke is one of the main forms of non communicable disease (NCD) along with heart disease, cancer, diabetes and chronic respiratory disease (Beaglehole et al., 2011). The NCD action group is a substantial, but informal, collaboration of academics, practitioners, and civil society organizations. In 2011, they presented the spread of NCD’s as a global crisis. The Center for Disease Control states that stroke, or a cerebrovascular accident (CVA), occurs when blood flow to the brain is interrupted. The resulting deprivation of essential compounds can cause tissue damage which varies based on stroke characteristics (CDC, 2022). The two main types of stroke are ischemic, which are caused by the blockage of a blood vessel, and hemorrhagic, which result from the rupture of a blood vessel. Our knowledge of stroke has increased substantially in recent years. A better understanding of the disease has led the National Collaborating Center for Chronic Conditions (NCCCC) to call for attention in specific areas of stroke care (2008). They emphasized an immediate need for a better understanding of the processes of care that contribute to improved post stroke outcomes. This thesis focused on the stroke care process by analyzing the relationship between cognitive impairment and functional outcome in a rehabilitation settingMACHINE LEARNING AND NEUROPSYCHOLOGICAL ASSESSMENT Rehabilitation The initial goals of stroke care during an inpatient hospital stay are to stabilize the patient, deliver acute treatments and perform preventive and prophylactic measures. Subsequently, rehabilitation services are the primary care mechanism by which functional recovery and the achievement of independence are promoted. In the United States, more than two thirds of stroke survivors receive rehabilitation services after hospitalization (Buntin, Colla, Deb, Sood, Escarce, 2010). Rehabilitation services vary considerably in care settings, intensity, duration, interventions and support staff (Winstein et al., 2016). Powerful and well-designed studies support the overall effectiveness of existing stroke rehabilitation practices (National Collaborating Center for Chronic Conditions, 2008) but major gaps persist in some areas of the literature. Where evidence does exist in these spaces, smaller trials of less rigorous design provide the only available data (Winstein et al., 2016). One specific domain which requires attention is research that informs clinical practice. The NCCCC has stated that evidence from research studies needs to be put into practice in order for it to improve patient outcomes (National Collaborating Center for Chronic Conditions, 2008). There is clear evidence that stroke rehabilitation is an effective and justified medium of medical intervention post-stroke. However, there is a need for clinically informing studies that further support the efficacy of rehabilitative care among patients with stroke. Stroke rehabilitation care is highly sensitive to budgetary constraints, due to the staff-intensive nature of the specialty (Thomson & Taylor, 2005). Some have argued that rehabilitative care should be deprioritized relative to other areas of health care, due to its high costs (Winstein et al., 2016). However, a patient who does not participate in rehabilitation has a greater risk of post-stroke medical morbidity resulting from immobility, depression, loss of autonomy, and reduced functional independence (Winstein et al., 2016). These issues can 2 cause unnecessary pain and suffering in patients and they increase stroke related costs on the healthcare system. Therefore, the net effect of stroke care services needs to be considered when evaluating the feasibility of a care program. Overall, rehabilitative care for patients with stroke is effective and a net positive financially. However, given the inherent financial burden associated with the specialty, cost efficiency should be considered when evaluating research that informs clinical practice (Thomson & Taylor, 2005; Frontera et al., 2005). Damage to the brain resulting from a stroke can lead to both cognitive and physical impairments, which range in their nature and severity (Nys et al., 2007; Barker Collo & Feigin, 2006). Cognitive impairments differ from physical impairments in their prognostic implications. Therefore, it has been recommended that the evaluation of both domains be routine in the clinical care of patients with stroke (Winstein et al., 2016). Multiple studies indicate that rehabilitation is effective for remediating cognitive issues (Barnes, Conner, Legault, Reznickova & Harrison-Felix, 2004; Paker, Bugdayci, Tekdos, Kaya & Dere, 2010). A neuropsychologist is a key member of any stroke care rehabilitation team (Miller et al., 2010). Neuropsychologists specialize in brain behavior relationships and have extensive training in anatomy, physiology, and neuropathology. They identify and treat cognitive and neurobehavioral dysfunction, and through assessment also monitor recovery and thereby enhance community reintegration (American Psychological Association, 2020). In a rehabilitation context, the role of the neuropsychologist is to inform recommendations for goal setting, treatment and discharge (Lezak, 1995). Cognitive assessment is a crucial step in stroke rehabilitation, since establishing a baseline assessment of cognitive abilities proximal to a stroke event and follow up deficits is essential to informing treatment recommendations. Cognitive Assessment As mentioned, cognitive impairments are common amongst patients with stroke. They 3 affect more than one third of stroke survivors at 3- and 12-months post stroke (McClure et al., 2012). These impairments persist in many individuals for years (Patel, Coshal, Rudd, Wolfe, 2002; Patel, Coshal, Rudd, Wolfe, 2003) and are associated with higher disability, greater institutionalization rates and poor long-term survival. Long-term cognitive decline in CVA survivors occurs at a rate of 12-56% versus 5-10% in normal adults (Xu, Ren, Prakash, Vijayadas & Kumar, 2013). However, research related to cognitive sequelae is relatively limited compared to the more extensive body of literature on physical sequelae (Lees, Fearon, Harrison, Broomfield & Quinn, 2012). Despite shortcomings in the cognitive aspect of the stroke care literature, the clinical value of a cognitive assessment is widely recognized (Winstein et al., 2016). International stroke guidelines recommend cognitive assessments for all stroke survivors, due to heterogeneity amongst both cognitive impairment and patient recovery trajectories from these issues (Miller, Murray, Richards, et al., 2010). A cognitive evaluation is an essential component of the post stroke diagnostic process. In a recent comprehensive review of the stroke rehabilitation process, Winstein et al. (2016) recommended that all patients with stroke should undergo a formal assessment, including a cognitive evaluation, to inform their rehabilitation needs. Cognitive tests are regularly administered by psychologists, educators, physicians and rehabilitation specialists. However, neuropsychologists are uniquely qualified in the conceptualization and assessment of cognitive issues. Neuropsychological Assessment NP assessment is the foremost cognitive evaluation method used by neuropsychologists in stroke rehabilitation. A NP assessment involves a thorough investigation of abilities and skills that are linked to brain function. The cognitive testing element of an NP assessment typically relies on a series of performance-based evaluations. These measures are norm-based, 4 meaning that the patient’s score is compared to a distribution of scores from testing a group of healthy individuals on the same task. Demographic parameters are employed in order to control for extraneous contributors to an individual’s score (i.e., age, education, etc.). NP tests target primary areas of cognition including attention, working memory, processing speed, memory recall and consolidation, language, visual perception and executive function (EF). NP assessment is widely accepted as the gold standard of cognitive evaluation (Bertolin, Patten, Greif, & Fucetola, 2017). The DSM-5 recommends that the determination of cognitive impairment should be “preferably documented by NP testing” (p.602). The neuropsychologist’s expertise is fundamental to the validity of the assessment. The manual for Advanced Clinical Solutions (ACS) for Wechsler Adult Intelligence Scale (WAIS)-IV and Wechsler Memory Scale(WMS)-IV states that, “When ACS is to be used for NP assessment, the examiner should have appropriate training in neuropsychology and NP assessment” (Pearson, p. 8). A NP assessment is far more comprehensive than a stand-alone cognitive assessment battery. Neuropsychologists must have “the ability to integrate NP test findings with neurologic and other medical data, psychosocial and other behavioral data, and knowledge in the neurosciences,” as well as “an appreciation of social, cultural and ethical issues” (Boake & Block 2015, Paragraph 5). As mentioned, the broad and comprehensive scope of the NP assessment is underpinned by a rich cognitive evaluation. The breadth of cognitive facets measured by a NP assessment is central to its advantage over other forms of cognitive measurement. The DSM-5 states that a NP evaluation is advantageous over all other forms of cognitive testing since it provides “quantitative assessment of all relevant domains” (p.610). To date, research with NP assessment has investigated cognitive profiles across an expansive range of populations, including patients with stroke. Research employing NP assessment to investigate domain specific cognitive profiles 5 among patients with stroke indicates that cognitive disruption is multifaceted. Pioneering work by Tatemichi and colleagues identified attention, memory, language and orientation to be most affected by stroke, with marked deficits in visuospatial skills and abstract reasoning as well (Tatemichi et al., 1994). However, a more recent body of literature indicates that stroke affects EF and attention to a greater degree than memory (Cumming, Marshal, & Lazar, 2013). Measuring the nature and severity of such deficits can play a major role in stroke care. In fact, addressing cognitive deficits was labeled a top priority in dealing with post-stroke outcomes according to surveyed stroke survivors, caregivers, and health professionals (Pollock, George, Fenton, & Firkins, 2012). In a rehabilitation setting, it is essential to examine cognitive functioning in relation to downstream daily functioning. In doing so, the clinician can estimate how the patient will perform upon discharge. Consequently, they can provide the necessary support and interventions to maximize the patient’s downstream functioning. Several studies indicate that the use of NP tests to evaluate specific cognitive domains is warranted for a more thorough understanding of the brain-behaviour relationship and to infer post-stroke daily functioning (Barker-Collo & Feigin, 2006; Lees et al., 2012). Functional Outcomes Functional outcome measures an individual’s abilities to interact with their environment in order to complete tasks of daily life. Tt focuses on an individual’s recovery in areas such as vocational and social functioning. Clinical outcome indicates levels of symptom resolution (i.e., a stroke patient with impaired motor abilities). The measurement of functional outcome speaks to a growing awareness that functional and clinical recovery do not necessarily parallel one another and highlights the importance of this construct to patient recoveries (Remington, 2010). Functional disability can stem from a variety of mental and physical health disorders. Findings have indicated that it could be the lead contributor to the daily burden of living with 6 mental illness (Lee et al., 2015). Functional recovery among patients with stroke is affected by the coexistence of psychosocial and cognitive impairments, as well as by a wide range of neurological and sociodemographic factors (Herrmann, Black, Lawrence, Szekely & Szalai, 1998; Fug, Liu, Wang Liu & Wang, 1997; Zhu et al., 1998; Kase, Wolf & Kelley-Hayes, 1998; Johnston et al., 2000; Appelros, Nydevik & Viitanen, 2003; Paciaroni, Arnold, Van Melle & Bugousslavsky, 2000; Paolucci et al., 1996; Alexander, 1994; Teasell, Mcrae & Finestone, 2000). Cognitive dysfunction contributes to functional impairment directly as well as through a psychosocial medium (Iosefescu, 2012). Significant psychosocial and cognitive impairments/disabilities occur amongst stroke survivors, which makes functional impairment common (Gresham, Alexander & Bishop, 1997). There is not a widely adopted framework for measuring functional outcome in the literature. However, the World Health Organization International Classification of Functioning, Disability and Health (ICF: WHO 2001, 2002) provides a multi-dimensional framework for health and disability as a basis for formulating outcome instruments. Three primary levels of functioning have been identified in the ICF framework – the body or body part, the whole person and the whole person in relation to their social context. Outcomes can be measured at any one of these levels. In the context of stroke, functional outcome typically refers to the level of functional independence that the individual regains. Functional independence is the ability to carry out activities of daily living safely and autonomously (Australian Institute of Health and Welfare. N.d. Paragraph 1). As mentioned, daily functioning issues are regularly subserved by cognitive deficits which limit functional recovery. Allanson, Pestell, Gignac, Yeo and Weinborn (2017) stated in a recent meta-analysis that understanding potential cognitive predictors of functional outcome is crucial. It helps to 7 estimate the usefulness of rehabilitation for patients and helps their families establish patient recovery expectations. Further, the identification of relationships between specific NP facets and functional outcomes can help tailor treatment plans and improve processes of care (Allanson, Pestell, Gignac, Yeo & Weinborn, 2017). Findings indicate that cognitive deficits could be more important determinants of post stroke functional outcomes than physical impairments and are associated with greater degrees of disability and mortality (Patel, Coshall, Rudd & Wolfe, 2002; Bays, 2001; Hochstenbach, Anderson, van Limbeek & Mulder, 2001). Therefore, the predictive utility of domain specific cognitive assessments is crucial to clinical decision making in a rehabilitation setting. A substantial body of literature has examined the relationship between cognition and daily functioning (Winstein et al., 2016). Previous Literature on Neuropsychological Assessment and Functional Outcomes Stroke can lead to a broad range of impairments including physical and cognitive issues and any problems which stem from these dysfunctions. Predictors that have displayed strong evidence of being precursors to inpatient rehabilitative care are cognitive deficits, age, post stroke levels of functioning and urinary incontinence (Hakkennes, Brock & Hill, 2011; Stein et al., 2015). Cognitive prognostics are independently predictive of patient outcomes from physical sequelae. Cognitive indicators have been shown to predict significantly more variance in functional outcome scores than demographic information, stroke characteristics and baseline levels of functioning alone (Bertolin, Patten, Greif, & Fucetola, 2017; Leitner, Miller, & Libben, 2018). A few recent NP assessments that have indicated utility in predicting functional outcomes include the Clock Drawing Task (Champond et al., 2019), an eye tracking adaptation of the Apples Test (Upshaw et al., 2019) and a battery of EF measures (Laakso et al., 2018). Further, the modified Rankin scale (functional outcome measure), Short Portable Mental Status 8 Questionnaire (SPMSQ; cognitive screener) and Barthel Index (functional outcome measure) predicted discharge location, among a large sample of patients with hemorrhagic stroke (N = 736; Stein et al., 2015). The only significant predictor was the Barthel Index, indicating that SPMSQ could be insensitive to cognitive functions that are related to daily functioning. It is important to note that cognitive screeners typically capture substantially fewer cognitive processes than a comprehensive NP evaluation. Consequently, they do not adequately inform clinical NP decision making and recommendations; this matter will be elaborated on further below. Research examining the relationship between NP assessments and downstream functional outcome has found significant effects. In 2011, Wagle et al., using the repeatable battery for the assessment of neuropsychological status (RBANS), found a significant association between the Coding (β = -0.484, p < 0.001) and Figure Copy (β = -0.233, p = 0.002) raw scores, at baseline, and modified ranking (mRS) scale scores 13 months later, with 42% of the variance being explained. However, the RBANS has no scale specific to the measurement of executive functions and does not properly address this construct (Randolph, Tierney, Mohr & Chase, 1998). Executive function (EF) has come to be an umbrella term used for a diversity of hypothesized cognitive processes, including planning, working memory, attention, inhibition, self-monitoring, self-regulation, and initiation carried out by prefrontal areas of the frontal lobes (Goldstein, Naglieri, Princiotta & Otero, 2014). At least three different executive function domains have been shown to differentially influence outcomes during rehabilitation (Miyake, Emerson, & Friedman, 2000) It should be noted that several elements of the RBANS, including measures of attention and the block design subtest tap elements of EF. Other research has examined the relationship between cognition and participation in life 3 months post-stroke. Participation in life is defined as involvement in all life situations, the 9 execution of a task or action (activity) and is also influenced by personal or environmental factors. Cognitive predictors in this study included the Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005) and two executive functioning tasks, the Executive Function Performance Test (EFPT; Baum et al., 2016) and the Dysexecutive Questionnaire (DEX; Wilson, Alderman, PW, Emslie & Evans, 1996). The MoCA and DEX were significantly related to functional outcome at three months post-stroke. The EFPT was non-significant indicating potential issues with the EFPT’s capacity to capture cognitions that are related to functional outcome amongst patients with stroke. In summary, research in this domain supports a relationship between cognition and functional outcomes. However, most studies employ measures that do not adequately assess the full range of cognitive domains that a comprehensive NP battery can capture, especially executive function skills. As a result, the full nature of this relationship may not have been adequately explored. This issue could undermine the predictive value of the factors in this type of research. Consequently, a cognitive evaluation’s clinical value and capacity to inform the brain-behavior relationship could be biased. Despite gaps in the literature, some research has indicated a multi-faceted relationship between cognitive sub-domains and functional outcome. For instance, Bertolin and colleagues (2017) examined the association between cognitive predictors and functional outcomes 6 months after stroke. Cognition was assessed using the Mesulam Cancellation Test (Mesulam, 1985), Short Blessed Test (Carpenter et al., 2011), Trails A and B (Reitan, 1955) and the Boston Naming Test (Strauss, Sherman & Spreen, 2006). These measures assess neglect, attentional abilities, orientation, registration, executive functioning and visual confrontation naming. The outcomes explored in this research comprised three domains including memory and thinking, communication and activities of daily living. Hierarchical regression indicated that introducing the cognitive predictors into the model significantly improved the amount of variance 10 accounted for, compared to a regression equation that consisted solely of demographic information, stroke characteristics and baseline levels of functioning (R2 change = 1.2%–2.4%). Jokinen and colleagues examined the relationship between a broad range of domain specific cognitions and functional outcome, as measured by the modified ranking scale (mRS), in patients with stroke 13 months post infarct (2015). Their sample consisted entirely of individuals with excellent clinical recovery (mRS = 0 – 1 at 3 months post stroke). They found significant relationships between functional outcome and memory functions (OR = 2.2), visuconstructional and spatial functions (OR = 5.1), executive functions and attention (OR = 3.2), aphasia (OR = 2.1), reading and writing (OR = 2.3) and abstract reasoning (OR = 2.3). These results are particularly important since they indicate a long-term effect of a broad range of cognitive functions on functional outcome, even amongst patients who are considered to have excellent clinical recovery. The focus of this study was to improve the clinical value of the NP assessment in the rehabilitation environment. Therefore, it is appropriate to examine cognition in relation to functional improvement during inpatient rehabilitation. One such inquisition examined how the mini mental state examination (MMSE; Folstein et al., 1975) related to improvement on various sub facets of the FIM. Significant relationships were observed between total scores on the MMSE and motor FIM improvement (r = .31, p = .04) as well as functional ambulation score improvement (r = .31, p = .03; Ozdemir, Birtane, Tabatabaei, Ekuklu & Kokino, 2001). Heruti and colleagues (2002) examined the effect of MMSE scores, using dyadic impairment classification, on FIM scores (2002). They found significant relationships between the MMSE and absolute functional gain motor (r = .193), absolute functional efficiency (r = .231), relative functional gain (r = .482) and relative functional efficiency (r = .323) on the motor component of the FIM. However, the shortcomings of cognitive screeners (i.e., MMSE) used in this 11 research may limit the detection of higher order cognitive functions which will be discussed below. Multiple studies indicate a significant relationship between cognition and functional outcomes amongst patients with stroke. However, there are several areas of need for further research in this domain. For instance, research on this association has not adequately captured the multifactorial nature of the relationship between cognition and functional outcome. Another area of need in the literature pertains to optimizing score interpretation to the rehabilitation environment. Optimizing Neuropsychological Assessment Current approaches to score interpretation in neuropsychology regularly underutilize the predictive capacity of the cognitive assessment for a variety of reasons. Firstly, clinical care units regularly employ brief screeners to detect post stroke cognitive issues. These brief mental status scales inadequately assess executive functioning skills (EF) and other higher-level cognitive functions (Winstein et al., 2016). The failure of cognitive screeners to adequately measure EF impairments limits the effectiveness of the assessment since EF deficits are among the most common cognitive dysfunctions among patients three months post stroke (Srikanth, Thrift & Saling, 2003). The prevalence of EF deficits after stroke range from 18.5% to 39% depending on the cognitive test used (Conti, Sterr, Brucki, & Conforto, 2015). EF skills are directly tied to daily functioning since they enable us to engage in productive and goal-oriented behaviors such as adjusting to unexpected changes in routine, scheduling and prioritizing tasks (Gazzaniga, Ivry, & Mangum, 2009). Current clinical NP practice typically involves reducing information across a variety of informative tools, including single measure cut off scores, to a coarse dichotomized classification of impairment (Cumming, Marshal, & Lazar, 2013). Typical NP assessment 12 interpretation practice largely depends on domain specific classification of disorders, based on single measure cut off scores. However, there is little agreement amongst neuropsychologists about the definition of abnormality (Binder, Iverson and Brooks, 2009). For instance, single test scores at or below 1 standard deviation below the mean (16th percentile) have been recommended as low by Heaton, Grant, and Matthews (1991) as well as Heaton, Miller, Taylor and Grant (2004). Meanwhile, Pearson Assessments one of the largest producers of norm-based NP assessments, classifies a score below the 10th percentile to be “borderline” and a score below the 2nd percentile to be “extremely low” (Wechsler, 1997a, 1997b). The variable nature of defined cut off levels between assessments can bias impairment classifications across NP assessment batteries. The Q-Simple approach to neuropsychological score interpretation practice has been proposed which addresses variance across score interpretation practices. The Q-Simple clinical score interpretation system includes seven classifications which are defined by standardized score levels (Schoenberg, Osborn, Mahone, Feigon, Roth & Pliskin, 2017). This framework has the potential to alleviate the interpretation practice concerns mentioned above. Another issue with interpreting NP assessments using single measure cut off scores is the prevalence of type I errors. A true score is a psychometric definition for a test score that is free from measurement error. Test score fluctuations due to intra-individual performance variability can lead to a patient with a true score in the disordered range to be classified as healthy, or vice versa. Ingraham and Aiken (1996) estimated that in a battery of only six tests, more than 20% of their normative sample would be classified as having an abnormal performance on at least two measures. Schretlen, Testa, Winicki, Pearlson, and Gordon (2008) confirmed the accuracy of Ingraham and Aiken’s model at one standard deviation below the mean. Further, Axelrod and Wall (2007) found that cut-off scores using the Reitan and Wolfson norms (Retain and Wolfson, 1993) from the Reitan and Wolfson battery led to false positives in a healthy population, among 13 an undergraduate sample. Surprisingly, a substantial number of participants were classified as impaired (29% of participants were deemed to be impaired on three or more of the tasks) using the Retain and Wolfson norms. It is noteworthy that normalcy in the sample was assessed using self-report and that no assessment of motivation was performed on these individuals. Potential issues with effort and health could have possibly contributed to these findings. Further, recent advancements in score interpretation practices, such as the Q-Simple system, have improved the consistency and accuracy of score interpretation practices. However, overall, this evidence raises the issue with relying too heavily on single assessment critical values. It is seemingly a mathematical certainty that levels of impairment will be overestimated, regardless of the cut off point’s magnitude. Current clinical interpretation practices could potentially be optimized by employing a full NP battery and augmenting, but not replacing, existing decision-making methods with group level score interpretation practices. Group level interpretation practices involve using multiple independent factors to predict a dependent factor, making multivariate statistical methods the ideal predictive medium. There are significant benefits to introducing this practice into existing clinical methods. The specificity of the assessment to sub facets of cognitive dysfunction would remain but the reliance on single measure cut off scores and the issues that go along with it could be alleviated. Secondly, group level score interpretation practices could be used to optimize the predictive utility of a NP battery for specific constructs. Thirdly, some multivariate techniques have a feature reduction component, meaning only the predictors with the highest degree of predictive utility would be retained. Feature reduction has implications for understanding the brain-behavior relationship and adapting assessment batteries to suit specific goals (i.e., predicting functional outcomes). 14 To date, a rich body of literature has examined the utility of statistical prediction compared to clinician decision making. Meehl was perhaps the strongest proponent of statistically derived clinical decision making. In his landmark book, Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence, he proposed that mechanical methods of prediction (i.e., formal, algorithmic, actuarial) would outperform clinical methods (Meehl, 1954). The debate between clinician and statistically based decision-making methods has endured since that time. An adequate summary of this debate can be found in a meta-analysis by Ægisdóttir et al., (2006) which examined 56 years of accumulated research on clinical versus statistical prediction. They concluded that clinician decision making should be complemented by statistical decision-making methods. Accurate predictive models that employ NP assessments in stroke care have been called for recently, in the literature. A comprehensive 2016 review article of stroke rehabilitation and recovery listed “Develop better predictor models to identify responders and non responders to different therapies” as one of their six main recommendations for future research directions in the field (Winstein et al., 2016, Conclusions section, para. 3). Further and Donders (2018), in a more recent review of the NP field, found the main value of the assessment, predicting patient outcomes over and above other diagnostic variables (demographic variables, medical or other), had not been regularly displayed in the literature. Accurate predictive modeling is a priority in stroke care research moving forward. A clinically informing predictive algorithm for functional outcome scores would enhance the clinical value of the NP assessment battery in a rehabilitation context. A comprehensive understanding of the relationship between domain specific cognitions and functional outcome is lacking in the literature. Further, clinical practice could be optimized by augmenting existing methods with group level score interpretation practices. 15 Therefore, the focus of this thesis will be on developing a clinically usable algorithm that informs the cognitive-behavioral relationship in stroke rehabilitation. This method has the added benefit of considering the necessary cost implications of stroke rehabilitation. A multivariate statistical method that may be appropriate for addressing these issues is ML. Machine Learning and Supervised Learning ML is an application of artificial intelligence that provides systems with the ability to learn and improve from experience without being explicitly programmed. ML algorithms, unlike most traditional statistics, presume that the underlying mechanism(s) of data generation are unknown or inconsequential (Breiman, 2001). The ML algorithms quickly learn the relationship between input variables (e.g., NP assessment scores) and response variables (e.g., functional outcome scores). After this relationship is learned, an algorithm is established, and the statistical tool can then be applied to an independent set of participants. These are a separate set of the sample’s participants from those who were employed in establishing the algorithm. ML can use the learned model to predict the response variable of a new independent subject (classification) or make point predictions (regression; Battista, Salvatore & Castiglioni, 2017). A variety of ML methods exist which differ in utility depending on a variety of sample characteristics, several of these methods will be described later in the proposal. As mentioned, ML is more adept at identifying multivariate patterns of data than conventional statistical models. Additionally, employing ML approaches typically leads to less false-positive and false-negative diagnoses than univariate statistical models (Hastie et al., 2009). ML is emerging in clinical neuropsychology and neurology to support the diagnostic process in a quantitative way (Battista, Salvatore & Castiglioni, 2017). ML models that employ NP assessment scores to produce classification parameters have been successful in differentiating populations with a high degree of accuracy. Examples include 16 classifying epilepsy localization and lateralization (Frank et al., 2018), differentiating MCI patients from controls (Battista, Salvatore & Castiglioni, 2017; Tunvirachaisakul et al., 2018), differentiating Alzheimer’s patients from non-Alzheimer’s patients (Gurevich, Stuke, Kastrup, Stuke & Hildebrandt, 2017), differentiating severe cognitive impairment patients from those with no cognitive impairment (Battista, Salvatore & Castiglioni, 2017) and predicting early onset Alzheimer’s disease (Ko, Ihm & Kim, 2019). Weakley et al. (2015) demonstrated the utility of ML in a sample of 272 subjects. Participants classified using clinical diagnosis criteria included 52 individuals with dementia, 97 with MCI, and 161 cognitively healthy older adults. They used a battery of 27 different NP assessments of cognitive, behavioural and functional abilities to classify patients across three dyadic combinations of levels from the Clinical Dementia Rating scale (CDR, 0, 0.5 and 1). Thousands of assessment battery configurations were compared, with a maximum of 6 measures per configuration, which would be nearly impossible to analyze manually. An accuracy of 98%, 82% and 94% was obtained when classifying CDR = 1+ versus CDR = 0, CDR = 0.5 versus CDR = 0, and CDR = 1 versus CDR = 0.5, respectively. The results showed that ML can produce a stable and parsimonious classification model using a relatively small sample size. To date, most ML applications in clinical neuropsychology have been classification based, that is they are used to predict membership in certain populations (i.e., healthy versus unhealthy) based on a set of predictors rather than predicting scores on an outcome measure (i.e., regression). However, regression-based learning models have been tested. As mentioned, ML methods do not rely on assumptions of conventional statistics. In terms of regression, most ML methods do not rely on the same assumptions as the conventional general linear model (GLM). These include assumptions of a linear relationship between variables, normal distribution of error, constant error across measurements, and that there is no correlation 17 between the errors (Cohen et al., 2019). In general, GLM analysis has limited viability in data sets when these assumptions are not met, making ML methods more flexible in regression applications. Additionally, ML methods are less sensitive to outliers and multicollinearity issues. Regression methods have significantly improved since the invention and subsequent popularization of GLM methods. Prior to ML analysis nonlinear features of a model needed to be mathematically transformed in order to fit the linearity assumption. This practice of feature extraction is not necessary with modern ML methods which have the capability of capturing more complex nonlinear relationships. It is noteworthy that when the nature of the relationship fits linear model assumptions GLM methods perform comparably or even outperforms nonlinear methods. Therefore, it is advantageous to run both linear and nonlinear models in order to assess the appropriateness of GLM modeling on a data set (Benjamin et al., 2017). Some ML regression methods are also regularization methods, meaning they can reduce overfitting. Overfitting occurs when a statistical model depends on noise that is specific to the data set and therefore does not generalize well to other data. Regularization techniques reduce overfitting by employing a penalty term. Ordinary least squares regression minimizes the residuals (distance between predicted value and actual value) and produces a linear function. Multiple ML methods exist to adapt ordinary least squares regression with a penalty term. The penalty term minimizes the effect of the coefficients with the least amount of predictive utility (i.e., Ridge regression) or eliminates low impact predictors entirely (i.e., Lasso regression). These methods reduce the number of predictors in a complex model which minimizes overfitting and can make the model more generalizable. The relatively parsimonious models that are produced by these methods have the added 18 benefit of identifying the variables with the highest degree of predictive utility, since those factors are retained or emphasized. This helps clarify the interpretation of the results by increasing the spread between strong and weak predictors, thus increasing the practicality of the model. Recent studies have employed regularization techniques using NP assessment batteries to produce highly predictive and interpretable models (Liu et al., 2016; Ko, Ihm & Kim, 2019; Goldman et al., 2014; Schmid et al., 2013). For instance, Ko, Ihm and Kim (2019) produced predictive models comprised of NP assessments with a battery of NP measures that are comparable with more expensive and invasive technologies, including positron emission tomography (PET) and cerebral spinal fluid analysis (CSF). NP assessment scores were used to predict levels of amyloid beta (Aβ) deposition, which is a hallmark of Alzheimer’s Disease (AD) that presents 10 to 20 years prior to symptom onset. The goal was to detect individuals who had Aβ accumulation, but no AD symptomatology, since early intervention is more effective when using anti amyloid therapies (Chételat et al., 2010, 2012). The penalized model (Least absolute shrinkage and selection operator; LASSO) predicted Aβ presence with close to 80% predictive accuracy. Further, the NP assessments that contributed most to the model were identified. Logical Memory (Wechsler, 1984) had the most utility in predicting disease presence. The results indicate the potential clinical value of the tools this method could inform, since NP assessment is less invasive, more widely available and more affordable than existing methods for diagnosing Alzheimer’s disease. Another important methodological process for increasing the generalizability of statistical models is cross validation. Cross validation involves assessing the performance of a model across the entire data set and reduces overfitting to specific clusters of data. Multiple methods of cross validation exist, including k-cross validation and repeated random sub-sampling. K-cross 19 validation involves separating the dataset into equally sized portions (i.e., the data set is split into 10 equal parts), then each portion is used as the test set with the remaining data used as the training set. The resulting models will be averaged to produce the final model. Repeated k-cross validation, available in R, allows for a set number of repeats for the cross-validation process. Repeated random sub-sampling involves the same process, except the testing sets are derived from random sampling, meaning certain data points can be used multiple times and others will not be used (Sief, 2019). Cross validation is increasingly common when ML is used including in Ko and colleagues Alzheimer’s research. The Present Study The present study employed ML with a battery of NP assessments to predict functional outcome, as measured by the functional independence measure. The sample consisted of patients with stroke from the rehabilitation department at Kelowna General Hospital. Broadly speaking, this work sought to address a need in the literature for a comprehensive evaluation of cognitive prognostics of functional outcome in a rehabilitation setting. Secondly, this thesis evaluated the utility of ML methods in relation to traditional general linear models within the framework of cognitive behavioral research on stroke rehabilitation. Thirdly, this inquiry will evaluate the effectiveness of utilizing ML techniques for producing clinical tools that support the neuropsychological decision-making process in a quantitative way. This method will ideally optimize clinical decision making in a rehabilitation context while retaining all the known benefits of the NP assessment. If successful, this approach could be employed amongst other aspects of NP decision making, perhaps related to the integration of multiple informative domains (i.e., neurological, medical, psychosocial etc.). This study had the added benefit of potentially identifying highly predictive cognitive domains and assessments. 20 Chapter 2: Body of Thesis Method Participants A certificate for harmonized review, which included Interior Health and the University of British Columbia Okanagan Behavioural Research Ethics Board was obtained prior to conducting any research (PAA #: H20-02254-A001). Data from 167 patients with stroke (64 females; mean age 69 ± 13) was used in this study, all of the analysis was performed on existing data that was collected from participants who had received care in the rehabilitation program at Kelowna General Hospital. For each patient, a complete medical history with neurological examination and clinical assessment was performed by physicians with experience in stroke disorders. The study was approved by the Research Ethics Board of the University of the British Columbia – Okanagan and Interior Health. The patients were considered eligible to participate if they met the following inclusion criteria: (1) capable of giving informed consent OR assent with consent from a primary caregiver, (2) no impairments in visual acuity (corrected vision ≥20/50), hearing, or motor functions that might interfere with testing, (3) fluent with English-language both expressive and receptive language, (4) minimum of sixth-grade education, (5) absence of serious psychiatric illness or psychosis, (6) absence of acute illness and/or neurological disease other than stroke, (7) between the ages of 19-100 years, (8) absence of aphasia. Each stroke participant was administered a battery of NP assessments. All the NP assessment batteries were administered by experienced psychometrists and were formulated and overseen by experienced neuropsychologists who were fluent in stroke rehabilitation. FIM data were collected by the occupational therapist during the patient’s stay in the rehabilitation department. The assessments that were employed in this study are described below and can be found in Table 2. 21 Procedure This research was conducted through a process of retrospective data collection. Informed consent was obtained through another ethically approved study (UBC REB Number: H16-02762). An amendment was made to this ethics application in order to use that data for this research. Data were taken from previous stroke assessments, obtained from inpatients in the rehabilitation department at Kelowna General Hospital (KGH). Measures The Functional Independence Measure. A crucial component to any predictive model, including ML models, is a valid and reliable outcome measure. Adequate functioning is typically measured in patients with stroke using various tests and functional scales (Keith, Granger, Hamilton & Sherwin, 1987). This information is employed to gauge rehabilitation progress, weigh the appropriateness of patient discharge and to determine discharge destination (Dobkin, 2005). The functional independence measure (FIM) was used as the outcome measure in this current research. The FIM is a standardized functional outcome measure in medical rehabilitation. It is recognized as one of the most widely utilized functional assessment tools to gauge progress in the rehabilitation of patients with stroke (Oztop, Ayas, Ustaomer, Cosar & Yemisci, 2013). Additionally, the FIM is the most widely used general measure of disability in North America and Australia (Murukami & Inuoye, 2002; Granger, Hamilton, Linacre, Heinemann & Wright, 1993; Ottenbacher et al., 2004). There is a large FIM database that is maintained by central coordinating centers, which allows for the comparison of functional outcomes by diagnoses as well as by facility (Murakami & Inouye, 2002). The FIM is subject to continuous review and refinement in order to maintain its standards and relevance. In addition, clinicians in subscribing centers undergo training courses 22 and accreditation examinations in order to maintain competency in FIM administration. The FIM consists of 13 motor and 5 cognitive items, with established content and construct validity, sensitivity and inter-rater reliability for the measurement of general functional ability across a wide range of rehabilitation conditions (Granger, Hamilton, Linacre, Heinemann & Wright, 1993). Motor items include disability assessment in feeding, grooming, dressing, toileting and mobility. Cognitive items assess communication, social interaction, problem-solving and memory. Scores range from 1 (totally dependent) to 7 (totally independent) for each of the 18 items, with a maximum score of 126 indicating total functional independence. The FIM gain is the difference between discharge FIM and admission FIM scores and measures functional improvement. The FIM efficiency is the FIM gain divided by the length of stay and measures the rate of functional improvement (Ottenbacher et al., 2004). Intervals between scores are not equal in terms of level of difficulty and cannot provide more than ordinal level information (Linacre et al., 1994). However, it has been suggested that summed scores be used on an interval scale while the individual items remain ordinal (Kidd et al., 1995). Functional outcomes were assessed near the beginning of the patient’s stay in the rehabilitation unit and proximal to discharge, it is recommended that the FIM be administered within 72 hours of arrival and within 72 hours of discharge, these guidelines were adhered to when possible. Model Predictors. The predictors in the model consist largely of NP assessments but also include demographic and medical information. In terms of demographic variables, age, education, biological sex, and handedness are included. Stroke characteristics including stroke type and location are also included. The NP battery employed in this research consists of assessments which are regularly administered to rehabilitation patients at Kelowna General Hospital (KGH). 23 The standard NP battery is comprised of the following measures: The Rey-O Complex Figure Test (Osterrieth, 1944) is a measure of visuospatial abilities, memory, attention, planning and working memory. Initially, participants copy a diagram onto a paper. The time to complete the drawing is recorded. The diagram is removed, and the patient is asked to recall the diagram from memory at an immediate (3 minutes) and long delay (30 minutes). The accuracy of both the copy and the delay are determined through scoring of the placement of 18 specific elements. The Wisconsin Card Sorting Test WCST; (Grant & Berg, 1980, Heaton, 1980) is a measure of executive function; requiring strategic planning, the utilization of environmental feedback to shift cognitive sets, goal-directing behavior, and the modulation of impulsive behavior. This test also assesses cognitive functions such as attention, working memory, and visual processing. The test consists of 128 response cards and 4 stimulus cards, and participants are asked to place the response cards onto the stimulus cards based on rules set by the examiner. The Buschke Selective Reminding Task (Buschke, 1973) is a list-learning test to evaluate verbal learning and memory. In this task participants are required to learn and recall a list of semantically unrelated words. The 12-word list is only presented on the first trial, and after this trial the subjects are selectively reminded of words not recalled. The task is discontinued after 3 trials of perfect recall, or after 12 trials, followed by a cued recall of both phonemic and multiple-choice cues. After a 30-minute-long delay, subjects are asked to repeat the list of words on free recall, as well as another delayed cued recall. The California Verbal Learning Test- Second Edition (CVLT-II; Delis, Kramer, Kaplan & Ober, 2000) is a frequently used test to assess verbal learning and memory. In this assessment the participant has a list of words read out loud to them and then asked to immediately recall the list. This same list is read and recalled over five consecutive learning trails. Subsequently, the 24 participant is presented with another list of words that they are asked to recall. After which he/she is asked to recall the first list. After this short delay free recall, the individual is asked to recall words from four specific categories. After a 20-minute delay, there is another free recall and categorical recall, followed by word recognition. The Color Trails Test (D'Elia, Satz, Uchiyama & White, 1996) is a task of visual attention and working memory and executive function. For Color Trails 1, the participant connects 25 color circles in sequential order. For Color Trails 2, each number is present in two colors, and the circles are again connected sequentially, but this time alternating colors; requiring visual attention and task switching. Executive function is required for task switching, as well as accuracy in response selection. The Wechsler Adult Intelligence Scale- Fourth Edition (WAIS-IV) (Wechsler, 2008) is an adult intelligence measure comprised of sixteen subtests to evaluate working memory, processing speed, perceptual reasoning and verbal comprehension indexes. The verbal comprehension index is measured through the vocabulary, similarities and information subtests. Perceptual reasoning index is comprised of block design, matrix reasoning and visual puzzles. Working Memory Index is assessed through arithmetic and digit span subtests. And finally, the processing speed index is evaluated through the coding and symbol search tasks. The Trail Making Test (Reitan, 1955) is a measure of processing speed and mental flexibility which a component of executive functioning. There are two parts, in part A the participant draws a line through a series of sequential circles numbered 1 through 25. In part B, the participant performs a similar task of 25 circles but alternates between numbers and letters. The Controlled Word Association Test – FAS and Animal Naming test (Benton and Hamsher, 1976) evaluates the spontaneous production of words under restricted search 25 conditions. First, participants will be given one minute to verbally produce as many words as they can think of given that begin with a certain letter (F, A, then S). This tests phonemic fluency. To test semantic fluency, participants will be asked to verbally produce as many different types of animals (Benton and Hamsher, 1976). The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Randolph, 1998) is a brief stand-alone NP battery that evaluates immediate and delayed memory, attention, language, and visuospatial skills. The Lafayette Grooved Pegboard (Lafayette Instrument, 2002) assesses fine motor speed and dexterity. It requires the participant to insert a series of identical pegs, as quickly as possible, into holes of varying angles one hand at a time. Missing Data The present sample was prone to missing data for several reasons; mainly variance in assessment battery compositions across patients and stroke related impairments which limited the completeness of certain assessments. In order to address this issue, data were over collected, information from 305 participants was inputted and a subset of 167 of those participants was retained for the analysis. Participants with greater than 50% of their data missing were excluded which led to the sample size reduction from 305 to 167. Some data still contained missingness after the removal of participants with high levels of missing data, so multiple imputation was employed with the missForest package (v1.4; Stekhoven, 2013). Data Analysis Statistics were conducted using R 4.0.2 (R Core Team, 2020), the missForest (v1.4; Stekhoven, 2013) and the caret package (v6.0 – 86; Kuhn, Wing, Weston, Williams, Keefer, Engelhardt, Cooper, etc., 2020). The full reproducible code is available in supplementary materials. This section will describe each step of the statistical analysis that was employed 26 for this work. This description will begin with how missing data were handled since it was the first step in the statistical procedure. There are two highly impactful factors related to missing data. The first is the nature of missingness. The most common model of addressing missing data involves three types of missingness. The nature of missing data will be assessed using Little's MCAR test which determines if a participant's data is missing completely at random (MCAR; Little, 1988). Missing completely at random data occurs when there is no relationship between the data and a construct that is extraneous to the dataset. Missing at random data occurs when the construct related to the missingness can be identified. Missing not at random exists when there is no identifiable pattern to missing data, meaning it is difficult to control for the factor that is influencing the missingness. Multiple imputations use by quantitative research has increased since being adopted by a number of statistical programs (e.g., R, SAS, S-plus, SPSS, and Stata). Despite this, MI has been underutilized since it was first introduced by Rubin (1987). We employed MI using the missForest package in R. MissForest is a nonparametric data imputation method which can accommodate both continuous and categorical variables. Further, the missForest method allows for model checking with an out of the (bag error estimate OOB). OOB error estimation involves using cases which are excluded from model building as a test set, since this makes them a good candidate for evaluating the model. Multiple imputation has been shown to be superior to a list-wise deletion approach and other traditional methods, except potentially in cases with very small amounts of data that are missing completely at random (Allison, 2002; Buhi et al., 2008; Cox et al. 2014). In 2011, Stekhoven & Buhlmann compared the missForest method, used in this current study, to commonly used multiple imputation algorithms including KNNimpute, MICE and MissPALasso. They used normalized RMSE and percentage falsely classified as comparison metrics and missForest outperformed all the other algorithms, including 27 reducing error by more than 50% in some cases. The second important factor when deciding on a missing data approach is the amount of missing data. Typical cut-off point recommendations for using multiple imputation range from 5% (Alice, 2015) to 10% (Dong, Peng, 2013). However, these levels have a limited evidence base to support them (Madley-Dowd, Hughes, Tilling & Heron, 2019). Fraction of missing information (FMI) has been found to be a more meaningful indicator of missingness than the number of missing cases. FMI uses efficiency gains as an evaluator, compared to using the proportion of missing data, when assessing the potential utility of multiple imputation methods in a dataset (Madley-Dowd, Hughes, Tilling & Heron, 2019). The FMI represents the fraction by which the standard error would shrink if the data set were complete. Conceptually, it would indicate the increase in model efficiency if missing values were not present. Values range from 0 to 1, with larger values indicating high variability between imputed sets, which could lead to unreliable estimates from MI. Interestingly, FMI results in the Madley-Dowd et al study, mentioned above, were similar when missingness depended on the auxiliary variable (Missing at Random) to when they did not (Madley-Dowd, Hughes, Tilling & Heron, 2019). A valid and reliable outcome measure is crucial to the validity of any regression model, the outcome in this research was the functional independence measure (FIM). At an item level the FIM can only be classified as an ordinal level variable, since intervals between scores are not equal (Linacre et al., 1994). Point prediction regression, which will be employed in this research, cannot accommodate ordinal level data. However, it has been recommended that scale and total scores on the FIM can be treated as interval level data (Kidd et al., 1995). Therefore, we will employ the total FIM score as the outcome variable in this research and treat it as an interval level variable. Analysis of the data will include multiple models using the total score from the FIM using every ML regression method. Further, traditional multiple regression will 28 be employed as a baseline of comparison for ML methods. ML regression analysis was performed with three different penalized regression methods and a random forest method. Penalized regression methods were selected based on appropriateness to the data and their level of predictive validity in previous studies. Penalized regression methods included Ridge, Lasso and Elastic Net regression. Penalized regression works by using a penalty term to reduce coefficient weights of low magnitude predictors to a minimal level (ridge regression) or all the way to zero (LASSO and elastic net). Penalized regression models are typically clearer and more interpretable since the relative distance between individual factor coefficients is greater. Random forest is an ensemble method that employs a cluster of regression trees, to predict a point value in the outcome variable. Random forest is a nonlinear method which is capable of identifying multiple difference clusters of related data which may not correlate in a linear manner. ML methods are described in more detail below. All four ML regression methods were used to assess the predictive utility of the NP battery. ML results were compared to ordinary least squares regression results using the root of the mean square error metric, r-squared coefficients are also reported. The recent paradigm shift across multiple disciplines from more traditional statistical models, such as OLS, towards ML has precipitated a transition from an emphasis on descriptive coefficients towards metrics which describe predictive accuracy (Plonsky, Erev, Hazan, & Tennenholtz, 2017; Walsh, Ribeiro, & Franklin, 2017; Alharthi, Guthier, & El Saddik, 2018; Kübler et al., 2020; Rosenbusch, Soldner, Evans & Zeelendber, 2021). RMSE has emerged as a popular method of comparing point prediction regression. The root of the mean squared error (RMSE) represents how much the predicted values of a regression model differ from the observed values, on average. The difference between a predicted value and an observed value is known as a residual in regression. RMSE is calculated 29 by squaring the residuals, finding the average of those residuals and then taking the square root of the mean score. This statistic represents the model’s accuracy in predicting continuous data points. Pragmatically, this means the model predicted multiple values, one for every participant, and on average it was inaccurate by the RMSE score. One key difference between OLS and ML is that some ML methods allow for hyperparameter tuning. Hyperparameter tuning involves specifying certain model parameters prior to fitting the model to the data. The parameters can determine complexity, the way the model incorporates new data and coefficient adjustment. General practice is to experiment with a wide range of values, this is often conducted through many iterations using modern statistical software (Bergstra & Bengio, 2012). When properly employed hyperparameter tuning has the potential to improve the model’s predictive validity, however without necessary precautions overturning can occur which limits the generalizability of the statistical model, this will be elaborated upon below. In the current study the penalized regression model’s hyperparameters were tuned using the glmnet package (Friedman, Hastie & Tibshirani, 2010). Glmnet is flexible and accommodates various regression methods including penalized linear regression. RMSE values were computed across different levels of the penalty term (lambda) for each of the three penalized methods, a sample of the results is provided (See Figure 1). Elastic net works by incorporating the functionality of ridge and lasso regression methodologies. The proportion of each method that is used in elastic net regression is tuned using the alpha parameter. For the elastic net model, a grid of 1000 random alpha levels was produced. For each of these levels of alpha 1000 different random lambda values were computed. The alpha level that coincided with the model that had the lowest RMSE was recorded and another 1000 iterations across different levels of lambda were processed. Our results provide the level of tune at which the model is the 30 most accurate (i.e., The lowest recorded level of RMSE). A commonly offered concern in the literature with ML is the possibility of overfitting. Models can be tuned with a greater level of dimensionality than conventional statistics in the General Linear Model umbrella. However, flexibility in applying the method can allow for artificially high results in a sample set. Consequently, such results cannot be generalized across samples or more importantly, to the underlying population of interest. This issue can potentially be exacerbated by overturning of hyperparameters, since this practice can lead to very tight fitting of the model to the sample data. However, re-sampling methods, which work by recreating multiple combinations of the same dataset and then combining the results, allow for more stable or realistic model estimates. This form of data aggregation is known as ensemble learning (Opitz and Maclin, 1999). Ensemble learning involves combining multiple models in order to increase model stability and therefore generalizability which can alleviate overfitting issues. We used cross validation provided by the glmnet package, which involves separating this sample into multiple sections, or “folds”. The model is fit on the total number of folds minus one, the unused fold is used to validate the model (Package = “glmnet”). The method attempts to alleviate the bias created by sampling variability by “sub sampling” within the data set. CV is designed to attempt to replicate multi-sample model validation. We used the repeated cross validation method provided by the Caret package in R (Kuhn et al., 2014) to increase the stability of our estimates. At each level of both lambda and alpha 10-fold cv with 3 repeats was employed in order to provide a more realistic measure of model performance. Ten-fold cv splits the data into 10 different sets, or folds, with replacement. The repeated element iterates this process the number of times the user specifies. Lastly, the best performing models of each method, in terms of RMSE, were retained and 30-fold cross-validation (cv) was performed again. These repeated cv 31 parameter levels were employed in producing the model coefficients and r-squared levels. One thousand random levels of the penalty term were computed for both ridge and lasso methods. Below is a brief description of each ML method that was used in this study. 32 Random Forest. Random forest is an ensemble method that employs a collection of regression trees. Predictions are made by calculating the mean or mode of regression tree point predictions. Regression trees work by finding clusters of data which are delineated with nodes, or roots, within which average values, or leaves, are used to predict scores on an outcome variable. The method works by iterating many different node configurations to optimize the model’s predictive utility. Random forest is available in R in the ‘Caret’ package, from the R CRAN repository. Ridge Regression. Ridge regression is a penalized linear regression method which employs a penalty term to shrink the model coefficients close to zero, but not exactly zero. The penalty term is called L2 norm, which is calculated by summing the squared coefficients. Ridge regression will produce the same results as ordinary least squares regression when the penalty term level is set to zero. Lasso Regression. The Least Absolute Shrinkage and Selection Operator technique is a penalized regression method which reduces the number of predictors into a more concise model. The penalty term used with this method is L1-norm, the sum of the absolute coefficients. The penalty has the effect of forcing coefficients with a minor contribution to the model to be exactly 0. The penalty term is dictated by lambda, a greater value equates to a greater penalty (Tibshirani, 1996). Lasso regression is available in R in the ‘caret’ package, from the R CRAN repository. Elastic Net Regression. Elastic net regression incorporates the L1-norm and L2-norm penalties from Lasso and Ridge regression methods. As a result, some coefficients can be shrunk to near zero levels, like ridge regression, and others to exactly zero, like lasso. Elastic net was created to offset overfitting issues from some lasso models (Oleszak, 2019). 33 Results As stated, we performed all of the statistical analysis using R statistical software. Prior to analyzing the data, we created several synthetic variables which combined measures of the same cognitive domain into a universal measure of that sub-facet. Firstly, the Trail Making Test (TMT) and Color Trails Test (CTT) were combined. We used the respective standardized score from each measure to combine into the new variable score which we labeled Standardized Trail Making 1 and Standardized Trail Making 2. Standardized Trail Making 1 included the standardized score for the total time for both the first part of the TMT and CTT measures. Standardized Trail Making 2 included the standardized score for the second component of CTT and TMT. The same process was repeated for the two verbal memory tests, including the Selective Reminding Test (SRT) and the California Verbal Learning Test (CVLT). We combined the verbal memory delay indexes for both the CVLT and SRT by standardizing the scores from their respective delay scores. Further, we combined the CVLT and the SRT total recall scores. The standardized score for total words recalled from both measures was combined into the Verbal Memory Total variable. The standardized delay scores from each measure were used to synthesize the Verbal Memory Delay Index indicator. Once the variables were synthesized, we examined the sample demographics and the descriptive statistics. As seen in our sample demographics table (Table 1) our sample was more predominantly male, elderly and on average had a high school level education. Table 1. Descriptive Statistics of the sample, mean and standard deviation for age and education as well as sex composition. Sample Demographics Variable Mean SD Age 69.45 13 Education 12.4 2.5 %Female 40 34 A variety of NP assessments were employed in this research, we endeavored to capture a comprehensive set of cognitive functions, these can be found in Table 2. Table 2. The battery of neuropsychological measures as well as the cognitive domain that each test assesses. Full Variable List Assessment Cognitive Domain(s) COWAT - FAS Phonemic fluency WAIS IV – Coding Processing speed, motor skill WAIS IV – Block Design Visual spatial abilities, motor skill, problem solving ability Rey-Osterrieth Complex Figure-Copy Visuospatial recall, visuospatial recognition, response bias, processing speed and visuospatial construction ability Color Trails Test A/Trail Making Test A Visual attention, working memory Color Trails Test B/Trail Making Test B Visual attention, working memory and executive functioning Wisconsin Card Sorting Test Executive functioning, attention, working memory, visual processing Selective Reminding Task Verbal learning and memory California Verbal Learning Test II Verbal learning and memory Grooved Peg Board Fine motor speed and dexterity Repeatable Battery for the Assessment of Neuropsychological Status Immediate memory, visuospatial/constructional, language, attention, and delayed memory. Repeatable Battery for the Assessment of Neuropsychological Status Immediate memory, visuospatial/constructional, language, attention, and delayed memory. Our battery assesses most of the main cognitive domains which stroke affects, according to the prior research mentioned above, with the exemption of abstract reasoning. The capacity of the battery to measure executive functioning (EF) was important given the potential connection between EF and stroke paired with the imbalance of research relative to other domains described 35 above. More specifically, the objective assessment of certain elements of executive functioning is research space which has been underserved. Missing Data Next, we cleaned the data, cases with greater than 50% of their scores missing were removed. Regression assumptions were then tested and variables with a high degree of multicollinearity (r > .75) were removed. Standardized Rey Delay, Standardized Trail Making 2, WCST Total Errors, WCST Perseverative Responses, WCST Conceptual Level Responses, Verbal Memory Delay and GPB Non Dominant Hand, all of which were standardized z scores, were removed due to high levels of interrelatedness with other assessments. Missingness was assessed using the “naniar” package in R. Below, in Figure 1, are the percentages of missing data across within each variable employed in the analysis. Figure 1. Percentage of missing data within each predictor. We then computed the fraction of missing information for each factor using the semTools package in R (v0.5-5, semTools, Jorgenson, 2021). As discussed above, FMI has been shown to be a better indicator of the impact of missing data on statistical results than absolute levels of 36 missing data (Madley Dawd, 2019). Results can be found in Table 3. Substantial levels of FMI included FAS total score (FMI =.250) and FAS category score (animals) (FMI = .197). Table 3. Mean scores and fraction of missing information scores for variances on each predictor. Variable Mean FMI Orientation 49.908 .068 FAS Total Score 40.083 .250 FAS Animals 41.738 .197 WAIS Coding 35.337 .060 WAIS Block Design 41.350 .059 Rey Copy Score 44.081 .055 Rey Immediate Score 39.099 .047 Standardized Trail Making 1 37.922 .015 Wisconsin Card Sort Test Perseverative Errors 41.626 .091 Wisconsin Card Sort Test Non-Perseverative Errors 44.486 .103 Wisconsin Card Sort Test Categories Completed 50.112 .068 Wisconsin Card Sort Test Failure to Maintain Set 50.066 .098 Verbal Memory Total Index 36.680 .091 Grooved Pegboard Dominant Hand 27.696 .090 We performed missing data imputation with the missForest package in R. The method accommodates continuous and categorical predictors in the same model and has outperformed other popular missing data methods in prior research (Stekhoven & Buhlman, 2011). MissForest model performance in this study was measured by out of bag (OOB) NRMSE which equalled .36. OOB refers to cases which are not selected for model building in the data partitioning process that occurs at each split of the imputation procedure. These cases were not used to 37 formulate the model and are therefore better candidates for model validation. OOB cases are a good indicator of potential overfitting issues, since cases used in model building would be biased towards fitting the model. Further, the bootstrapping procedure helps to stabilize the model by providing multiple versions of the results which are averaged to provide a more realistic estimation of the population statistic. Multiple Regression Penalized regression involves hyper-parameter tuning, meaning the penalty term requires manual adjustment to maximize model parsimony. Computing a range of hyper-parameter values and comparing their performance has been recommended (Bergstra & Bengio, 2012). Plot three shows the different levels of penalty term that were produced using the parameter tuning procedure for the ridge model with the difference score as an outcome. Figure 2. RMSE across levels of lambda, the penalty term for penalized ridge regression. Ridge, Lasso and Elastic Net regression models were all produced, for reasons mentioned prior, using tenfold cross validation with three repeats. We employed this methodology in order to identify the penalty term with the highest accuracy for each of the 38 penalized regression methods. Random forest was implemented using a 700-tree model, 18 predictors were used at each split. We retained the models with the highest r-squared across all levels of the tuning parameter(s). Table 4 shows the highest r-squared for each of the different methods using both the first and second FIM readings as outcomes. Table 4. Results of the different regression models, including Ridge, Lasso, Elastic Net, Random Forest and Ordinary Least Squares methods. Method Original FIM Reading R^2 Second FIM Reading R^2 OLS .16* .19* Ridge .22 .19 Lasso .14 .18 Elastic Net .17 .17 Random Forest .05 .11 All five methods were used with the original FIM and second FIM readings. These results indicate the highest r-squared from all iterations. Statistical significance at p < .05 is indicated by a “*”. OLS models were significant for both the first and second readings of the FIM. OLS regression is the only method in this research that significance testing can be applied to without producing biased results. Therefore, significance testing was not applied to any of the ML methods. Individual Predictor Analysis We produced individual model coefficients for every model, except the random forest (RF) models, as the RF method does not produce meaningful factor weights. The individual predictor weights are presented in Figures 3 and 4. The results are taken from the model for each method that indicated the lowest RMSE. The WAIS block design and orientation task had a consistently high relationship with the original FIM reading across all four methods for both outcomes. The coding task was substantially related to the second reading of the FIM as well. 39 Figure 3. Individual factor coefficients across linear methods with original FIM score outcome. Figure 4. Coefficients across linear methods with second FIM reading as outcome. -0.5-0.4-0.3-0.2-0.100.10.20.30.40.50.6 Sum of OLSSum of RIDGESum of LASSOSum of ELASTICNET-0.4-0.3-0.2-0.100.10.20.30.40.5Sum of GLMSum of RIDGESum of LASSOSum of ELASTICNET 40 Multivariate Predictive Utility We then calculated RMSE scores, using cross validation, across all four methods which are presented in Table 3. The process works by identifying the levels of the tuning parameter at which the RMSE is minimized. Cross validation, using 30 resamples, was performed on the model with the parameter levels fixed at the optimized level produced by the tuning process. We used the median as the metric for identifying the best predictive algorithm, since the mean can be affected by extreme scores. The ridge model produced the least amount of prediction error amongst the models which were regressed on the first FIM measurement. Ridge regression produced the lowest RMSE on the second FIM measurement. 41 Chapter 3: Conclusion Discussion Research Aim and Findings This work sought to evaluate four main research questions. Firstly, it examined the comprehensive evaluation of cognitive prognostics of functional outcome in a rehabilitation setting. Secondly, this thesis evaluated the utility of ML methods in relation to traditional general linear models within the framework of cognitive behavioral research on stroke rehabilitation. Thirdly, this inquiry will evaluate the effectiveness of utilizing ML techniques for producing clinical tools that support the neuropsychological decision-making process in a quantitative way. This method will ideally optimize clinical decision making in a rehabilitation context while retaining all the known benefits of the NP assessment. If successful, this approach could be employed amongst other aspects of NP decision making, perhaps related to the integration of multiple informative domains (i.e., neurological, medical, psychosocial etc.). This study had the added benefit of potentially identifying highly predictive cognitive domains and assessments. The adjunctive decision algorithm did not display sufficient utility to meaningfully contribute to current clinical care practices. The multiple regression findings confirmed the utility of employing a NP assessment battery to predict post stroke daily functioning. Penalized regression methods performed similarly to GLM regression, in terms of overall prognostic utility. The relative magnitude of individual factor coefficients between methods indicates that penalized regression could be useful in delineating high leverage NP predictors of functional outcome amongst patients with stroke. This discussion section will begin by discussing the results of diagnosing and handling missing data, next the results of each goal of this research will be broken down individually, lastly limitations and future research directions will be presented. 42 Missing Data We employed the fraction of missing information as the primary medium of diagnosing missing data issues. The fraction of missing information describes how much the standard error of measurement (SEM) is affected by missing data points (Madley-Dowd, Hughes, Tilling & Heron, 2019). The SEM indicates the size of the distribution of estimates for a model parameter. A larger distribution represents more variance across parameter estimates, which translates to a less stable and less accurate model. A higher FMI statistic indicates an imputation model that is less stable due to higher variance across imputations. Multiple imputation can mitigate the effect of missing data to the extent that the method produces stable results which accurately reflect the hypothetical values that they replace. A perfect imputation model would contain no bias from missing data. There were two variables with an FMI greater than .15, total FAS score (FMI = .197) and the animal component of the FAS (FMI = .250). Missing data were addressed by using the missForest nonparametric multiple imputation function that is provided by the missForest package in R. This method has been shown to outperform other common imputation procedures (Stekhoven & Bulman, 2011). Error rates of 1, measured with the NRMSE statistic, indicate that a model performs comparably to a random guess, whereas levels closer to 0 imply a minimal effect of imputed values on the data. The imputation model’s stability is an important factor when estimating missing values since a more stable model will produce consistent results across trials of running the MI method. Model stability typically increases with the number of imputations in the missForest package. When a variable presents a high FMI, it is important to increase the number of imputations in order to offset the instability present in the missing values. 1000 imputations were computed in order to increase the stability of the model. Interpreting RMSE in a multiple imputation environment is an unstandardized exercise. 43 Results can be examined in a relative sense (i.e., MI results versus another missing data method) or from a model bias standpoint (i.e., How much does the imputation error bias the model’s results). Interpreting MI results through a model bias lens involves calculating the degree to which the results are affected by multiple imputation errors. The level of bias resulting from imputation should be evaluated by examining FMI and the multiple imputation model’s accuracy together. This approach considers both the degree of bias contained in missing data and the degree of accuracy of the replacement estimates. The minimal level of influential FMI indicators combined with the missForest OOB error rate of .36 indicates that missing data had a relatively small impact on the results. Neuropsychological Battery Predicting Functional Outcome Overall, the implications of these results are multifaceted. Firstly, the results of this study indicate that penalized linear regression may not provide improvement in multivariate utility over OLS multiple regression in this research setting. However, these multivariate findings support the efficacy of GLM regression in this type of clinical NP research. The relative utility of GLM in this research space could be moderated by the suitability of the ratio of predictors to sample size. The NP battery was a statistically significant predictor of FIM scores across both measurements of post stroke functioning when OLS methods were employed. However, the multivariate effect sizes were lower in magnitude compared to some previous findings in the NP literature (Bertolin, Patten, Greif, & Fucetola, 2017; Leitner, Miller, & Libben, 2018). There are several potential reasons for the relatively low effect sizes in this research. The effect of the missing data could have distorted the factor weights of these predictors, which may have led to artificially low effect sizes. FAS total and animals were significant components of the language retrieval and verbal memory domains in our battery; the effect of the missing data may have impacted this significant cognitive predictor of functional outcome. Using the fraction of missing 44 information results, two of our factors produced coefficients that indicated a significant impact of missing data onto the standard error of our model (Total FAS score = .25, FAS Animals = .197). Resultantly, this may have diminished the multivariate effect sizes in the model. Another potential factor related to the low effect sizes in the model could be sampling variability. Effect sizes between the same constructs can vary across samples due to heterogeneity across samples, this sample may contain a lower effect size compared to other samples from the same population. Thirdly, cross validation was employed in this research which may have diminished the effect sizes from OLS methods. Publication bias occurs when studies with larger effect sizes are accepted into journal publications and research with smaller effect sizes are denied. CV leads to more realistic parameter estimates which diminishes the effect of publication bias, this may have led to lower effect sizes compared to other published data. Machine Learning One issue with OLS regression is that overfitting is common, which can diminish the generalizability of the results to other data (Lever, Krzywinski, & Altman, 2016). Overfitting issues can be exacerbated by a higher ratio of predictors to observations (Askin, 1982). We employed cross validation to try and alleviate potential OLS overfitting concerns. Cross validation (CV) facilitates more stable and accurate model estimates. The median RMSE from the CV results was the metric for comparing model performance. The median is not affected by outliers to the same extent as the mean and provides a more stable estimate. The top performing method differed slightly between outcome variables. Ridge regression produced the lowest median RMSE amongst models that predicted original FIM scores (RMSE = 20.26). The elastic net method displayed the best performance for the second FIM reading (RMSE = 12.81). The multiple regression models that used the second recording of the FIM as the outcome variable 45 were more accurate for every multiple regression method in terms of RMSE. These results confirm the previously established utility of a NP assessment when predicting daily functioning proximal to discharge from a rehabilitation program. This information could inform clinical recommendations in the rehabilitation environment which could help maximize the efficiency and effectiveness of stroke care programs. Machine Learning Versus OLS Regression The CV results indicate that the magnitude of the inter-method performance difference amongst multivariate multiple regression methods was less than the results of prior research. However, some ML methods outperformed OLS to a small degree in terms of RMSE levels. Overall, the multivariate performance of multiple regression methods was similar in our dataset. Some previous research has indicated superior multivariate predictive accuracy from ML regression compared to OLS when applied to NP data (e.g., Joel, Eastwick, & Finkel, 2017; Park et al., 2015; Plonsky et al., 2017;Wang & Kosinski, 2018). The relative performance of OLS versus ML aligns with findings from a 2020 study which compared the performance of regular logistic regression to ML logistic regression methods, including elastic net, LASSO and adaptive LASSO (Greenwood, Youssef, Letcher, Macdonald, Hagg, Sanson et al.). Their research produced a similar difference in results between OLS regression and ML methods to ours, meaning results clustered between multivariate results amongst the regression methods. There are several potential reasons for the weak effect of ML regression. Firstly, penalized regression (PLR) tends to outperform GLM regression in datasets that contain a high ratio of predictors to participants. This includes instances when the number of predictors is greater than the number of participants (Finch & Finch, 2016). A general rule of thumb for assessing sample size adequacy in linear regression is 50 + 8k (k = number of predictors) 46 which in our case would equal a sample size of 194. Another guideline is ten participants per factor which would equate to a sample size of 180 with the variables from our dataset (VanVoorhis, C. R. W., Morgan, B. L, 2007). In our dataset the participants (N = 167) greatly outnumbered the predictors (18) which could have diminished the advantage PLR can provide over OLS. Our dataset was slightly below multiple recommended thresholds for required sample size in an OLS regression model with 18 predictors. The relatively small divergence between our dataset and recommended cut-off levels for OLS methods could have minimized the incremental utility ML can provide in non-ideal datasets. In terms of a comparison between ML and OLS regression techniques, ridge regression coincided with OLS results the most when comparing coefficient weights. This could be because the mechanics of OLS are relatively similar to ridge versus the other penalized regression methods. Individual factor slopes are minimized during the ridge regression process but not eliminated entirely, which they are with Lasso and Elastic Net. Individual Cognitive Facets In the context of this research, it is important to analyze predictors individually in order to isolate the effect of cognitive sub-domains and NP assessments. This inquiry could educate the brain-behavior relationship by investigating the cognitive-functional network of stroke care factors. Further, these results can inform clinical practice by identifying strong cognitive predictors that may allow for assessment batteries to be fine tuned. Previous research has indicated that penalized linear regression is an effective tool for isolating highly predictive factors in a multiple regression environment. ML methods can detect impactful individual model coefficients despite weak overall effect sizes (Greenwood et al., 2020). Identifying the most predictive NP measures in our data can help inform the brain behavior relationship and contribute to NP psychometrics among patients with stroke. For instance, individual measures 47 that correlate highly with post-stroke daily functioning in our sample could be used to inform future research by establishing a baseline of assessments with utility in this application. A visual inspection of the coefficient magnitudes for both sets of models (First and second FIM reading) indicates that, generally, OLS methods produced coefficients with a slightly stronger relationship with the outcome than coefficients that were computed with ML methods. However, overall, there was stability between methods within coefficients. That is, the relative order of predictors, in terms of their relationship to the outcome variable, remained similar for each individual factor across different methods. As expected, the ML methods produced a more parsimonious model. Meaning, despite the weaker effect sizes, the model was cleaner to interpret in terms of identifying high leverage predictors. The factors that were the most predictive of the patient’s original FIM scores, using ML methods, were the block design (BD) and coding (CD) subtests of the Wechsler Adult Intelligence Scale (WAIS-IV), and both the categories completed and perseverative errors sub scales of the Wisconsin Card Sort Test (WCST). In terms of the second reading, BD, CD, orientation, perseverative errors and animals, measured by the FAS, were the strongest factors. As listed above, WAIS block design assesses visuospatial abilities; coding examines processing speed; WCST captures higher order executive functions, FAS measures phonemic word fluency and orientation taps an individual's awareness to space and time. The diversity in cognitive functions which were linked to functional outcome indicate the importance of employing a comprehensive NP assessment for post stroke care. A relatively narrow evaluation, such as a standalone cognitive screener, may have been insensitive to the rich grouping of cognitive facets that were identified in this research. The factors that were related to functional outcome in this research have been well documented in prior literature. The results lend evidence towards the importance of the relationship between EF and 48 functional outcome amongst stroke patients. As mentioned prior, the link between EF and functional outcome has been established but is underpinned with a relatively smaller body of evidence relative to other cognitive factors in this space. EF measures were retained from the iterative penalized regression process, including WCST perseverative errors and total categories completed of the WCST. Only the strongest factors are retained in penalized regression models which implies a substantive link between EF and functional outcome amongst patients with stroke. These results showcase the usefulness of penalized regression methods in this type of work, since their pruning effect helps to isolate stronger predictors. Cognitive sub facets, that were the most predictive of functioning both early and late in the post stroke rehabilitation process, were identified. Baseline levels of EF following stroke could be useful prognosticators of a patient's post-stroke level of functioning proximal to discharge from a rehabilitation program. The relationship between EF and functional outcome is noteworthy. There is a relatively smaller amount of evidence of a relationship between EF and functional outcomes found in previous literature compared to the other cognitive domains that were identified in the results of this thesis. The non-EF measures, listed prior, with a strong relationship with functional outcome capture cognitive domains that have been linked to post-stroke functioning in prior research (see above). Our results provide convergent evidence, through a novel statistical medium, of the predictive utility of these cognitive domains. Adjunctive Clinical Decision Algorithm The minimum level of predictive accuracy that is necessary to support the clinical decision-making process is important to establish. Therefore, we endeavored to operationalize the minimal level of accuracy that would be required from the decision algorithm for it to have clinical utility. The RMSE statistic was used to establish this benchmark, since it indicates the 49 average absolute difference between predicted values and actual values in the dataset. When interpreted through a clinical decision-making lens, this indicates the difference between the functional recovery level that a patient regains and the recovery level that the model predicted they would reach. If the error rate is too high then recommendations that the algorithm informs could diminish the quality of clinical recommendations and as a result patient treatment plans could be adversely affected. For instance, suboptimal recommendations by the clinician could affect the patient’s length of stay and quality of treatment. In turn, these biased decisions could have a negative impact on recovery outcomes. Biased clinical recommendations can also affect rehabilitation programs at a group level. For instance, a patient who participates in therapy for longer than necessary takes a space that another patient might require. Therefore, at minimum, the average magnitude of error produced by a clinically informing algorithm should not exceed the incremental difference between decision making benchmarks from the measure that the algorithm informs. The Uniform Data Systems for Medical Rehabilitation organization states that a FIM score of 60 can lead to approximately four hours of daily assistance, two hours of daily assistance for a score of 80 and individuals with scores over 100 require minimal assistance in daily life. The average error our best model produced for the first (RMSE = 20.25913) and second (RMSE = 13.21783) FIM readings could easily lead to mis-categorization of patients and adversely affect therapeutic trajectories. Therefore, we do not advocate for the clinical implementation of this algorithm in its current form. The results from the current research can be a useful first step for other studies that endeavor to establish clinically informing decision making algorithms. Results with significantly greater predictive accuracy than ours would indicate progress towards the ultimate goal which is AI assisted clinical decision making among neuropsychologists in a rehabilitation 50 setting. Or, perhaps these findings can be a building block towards a clinically impactful adjunctive decision-making algorithm. Limitations and Future Research The OLS effect sizes of .19 for both the first and second readings of the FIM indicate that 81% of the variance remains unexplained. Future work will be required to account for a larger proportion of scores in functional outcomes. One potential issue with our results could be related to the outcome measure. The FIM may not have adequately measured the functional outcome construct. The scale contains a limited number of questions that assess cognition, the three indicators are social interaction, problem solving and memory; all of which were on a 7-point Likert-type scale. These subscales could be inadequate in assessing the broad range of post-stroke daily functions that NP cognitive factors subserve. For instance, the FIM has produced a ceiling effect in previous research which examined rehabilitation from neurological damage (Davidoth, Roth, Haughton & Ardner, 1990; Riberto, Miyazek, Jucá, Sakamoto, Pinto & Battistella, 2004). A ceiling effect can truncate the range of the underlying construct that the assessment evaluates. Notably, Barbetta, Cassemiro & Assis’s research produced a ceiling effect in 90% of patients (2014). Further, our results indicate that the questions in the cognitive section of the FIM may not correlate to a high degree with the cognitive aspect of NP measures. The results from the block design task (BD) lend evidence to this possibility since this task was highly related to FIM scores across all four regression methods. BD requires a significant motor component, due to the number of precise manipulations of the blocks that are required to complete the task (Weschler, 2008). The substantive relationship between the coding task (CD) and the FIM further supports this hypothesis, since CD requires a high degree of fine motor coordination for completion as well (See Figures 1 and 2). The relatively small number of cognitive items could mean the items on the FIM measure load more highly with physical 51 sequelae than cognitive deficits. Some ML algorithms require sample sizes greater than several hundred, even for relatively simple models. Furthermore, methods that can model non-linear relationships typically require more participants than linear models (Brownlee, 2017). Future research should employ more non-linear ML methods in order to properly assess the utility of these statistical tools. This work will require a larger sample size, future studies should employ more participants to properly evaluate non-linear methods and to evaluate the relationship between penalized linear regression and ordinary least squares methods. Sample size requirements depend on model complexity and the method which is employed. For instance, a model with more features which employs non-linear modelling will require more participants than a penalized linear model with a narrow grouping of features. A linear ML model which performs at a satisfactory level with hundreds of participants could require thousands of participants for non-linear methods (Brownlee, 2019). Future research should also examine the improvement provided by adding NP predictors to more common prognostics (i.e., demographics and physical sequelae) when penalized regression or other ML regression techniques are employed. Hierarchical regression models, introducing later stage NP predictors, could perform differentially in OLS vs ML models. Future studies should examine the relationship between a NP assessment close to discharge and post-discharge patient functioning at a variety of temporal cross sections (i.e., 3 months, 6 months, etc.). The data were prone to missingness which could affect the generalizability of the results due to potential selection bias issues. Future studies should employ a more standardized battery of measures and take steps towards minimizing incomplete assessments. Assessment selection optimization should be employed to ensure psychometric issues do not affect model utility. For 52 instance, the ceiling effect of the FIM could bias results since the higher end of the underlying construct is not adequately represented. The outcome measure in future research should measure more non-physical elements of functional outcome including cognitive and emotional functions. One potential outcome measure which should be employed, as recommended by Leitner, Libben & Miller (2018), is the Mayo Portland Adaptability Inventory – Fourth Edition (MPAI-4; Malec et al., 2003). The MPAI-4 addresses a range of functional constructs including physical and cognitive abilities (Abilities subscale), participation restrictions and activity limitations (Participation subscale), and personal and environmental contextual factors(Adjustment subscale), with a total composite score calculated from the three subscales. The MPAI-4 could help to address limitations in research and clinical practice resulting from the use of unidimensional assessments (Malec 2004, Malex et al., 2003). The operationalization of the minimal acceptable error benchmarks was relatively crude in the current study. Future research should attempt to identify appropriate benchmarks with the feedback of NP clinicians who are experienced in a rehabilitation setting. Alternatively, the accuracy of the clinically informing algorithm could be evaluated in a relative sense. The performance of clinical decision making could be measured by comparing clinical recommendations with and without an adjunctive clinically informing algorithm. If performance improves with the information provided by the new algorithm, then it would indicate its utility. Clinically informing algorithms are becoming more common across a variety of disciplines. These tools should never be meant to replace the insights of a clinician, given the amount of subtle nuance that exists amongst stroke deficits that can only be captured by a proper clinical interview and neuropsychological assessment. However, they do have the potential to augment existing decision making which could increase the efficiency of stroke 53 care practices which would allow for greater levels of patient care with the same number of human resources. Multiple iterations of decision models which employ sophisticated mathematical tools should be composed with the feed back of experienced clinicians and then evaluated side by side. The measure of success or failure should be operationalized with feedback from clinicians such that the various decision models can be evaluated with a common outcome metric. This current proof of concept indicates that there is potential for clinical decision algorithms to be useful tools, however, more work is needed before they provide sufficient utility to impact current clinical care practices. 54 Bibliography Adamit, T., Maeir, A., Ben Assayag, E., Bornstein, N. M., Korczyn, A. D., & Katz, N. (2015). Impact of first-ever mild stroke on participation at 3 and 6 month post-event: The TABASCO study. Disability and Rehabilitation: An International, Multidisciplinary Journal, 37, 667–673. doi: 10.3109/09638288.2014.923523 Advanced Clinical Solutions for WAIS-IV and WMS-IV, Administration and scoring manual. (2009). San Antonio: NCS Pearson. Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., & Rush, J. D. (2006). The Meta-Analysis of Clinical Judgment Project: Fifty-Six Years of Accumulated Research on Clinical Versus Statistical Prediction Stefanía Ægisdóttir. The Counseling Psychologist, 34(3), 341–382. doi:10.1177/0011000005285875 Alexander MP. Stroke rehabilitation outcome. (1994). A poten-tial use of predictive variables to establish levels of care. Stroke, 25, 128-34. doi: 10.1161/01.str.25.1.128 R. Alharthi, B. Guthier and A. El Saddik, "Recognizing Human Needs During Critical Events Using Machine Learning Powered Psychology-Based Framework," in IEEE Access, vol. 6, pp. 58737-58753, 2018, doi: 10.1109/ACCESS.2018.2874032. Allanson, F., Pestell, C., Gignac, G. E., Yeo, Y. X., & Weinborn, M. (2017). Neuropsychological Predictors of Outcome Following Traumatic Brain Injury in Adults: a Meta-Analysis. Neuropsychology Review, 27, 187-201. doi: 10.1007/s11065-017-9353-5 Allison, P. D. (2012). Handling missing data by maximum likelihood. In Paper presented at the SAS Global Forum, Orlando, FL. http://www.statisticalhorizons.com/wp-content/uploads/MissingDataByML.pdf. Accessed 15 October 2021. American Psychiatric Association. (2013). Diagnostic and statistical manual for mental disorders, fifth 55 revision. Washington, DC: American Psychiatric Publishing Appelros P, Nydevik I and Viitanen M. (2003). Poor outcome after first-ever stroke. Stroke, 6, 34-122. Doi: 0.1161/01.STR.0000047852.05842.3C Askin, R. G. (1982). Multicollinearity in regression: Review and examples. Journal of Forecasting, 1, 281–292. doi: 10.1002/for.3980010307 Australian Institute of Health and Welfare. n.d. Metadata Online Registry. Level of functional independence. Retrieved from https://meteor.aihw.gov.au/content/index.phtml/itemId/ 449145 Axelrod, B. N., & Wall, J. R. (2007). Expectancy of impaired neuropsychological test scores in a non-clinical sample. International Journal of Neuroscience, 117, 1591–1602. doi:10.108 0/00207450600941189 Barbetta, D., Cassemiro, L. & Assis, M. The experience of using the scale of Functional Independence Measure in individuals undergoing spinal cord injury rehabilitation in Brazil. Spinal Cord 52, 276–281 (2014). doi: 10.1038/sc.2013.179 Barker-Collo, S., & Feigin, V. (2006). The impact of neuropsychological deficits on functional stroke outcomes. Neuropsychology Review, 16, 53–64. doi:10.1007/s11065-006-9007-5 Barnes C, Conner D, Legault L, Reznickova C, Harrison-Felix C. (2004). Rehabilitation outcomes in cognitively impaired patients admitted to skilled nursing facilities from the community. Archives of Physical Medicine and Rehabilitation, 85, 1602–1607. doi: 10.1016/j.apmr.2004.02.025 Battista, P., Salvatore, C., & Castiglioni, I. (2017). Optimizing neuropsychological assessments for cognitive, behavioral, and functional impairment classification: A machine learning study. Behavioural Neurology. Article ID 1850909. 19 Pages. doi:10.1155/2017/1850909 Bays, C. L. (2001). Quality of life of stroke survivors: a research synthesis. J Neurosci Nurs, 33, 310-6. doi:10.1097/01376517-200112000-00005. Beaglehole, R., Bonita, R., Horton R., et al. (2011) Priority actions for the non-communicable disease 56 crisis. Lancet, 377, 1438-1447. doi:10.1016/S0140-6736(11)60393-0 Bertolin, M., Van Patten, R., Greif, T., Fucetola, R. (2017). Predicting Cognitive Functioning, Activities of Daily Living, and Participation 6 Months after Mild to Moderate Stroke. Archives of Clinical Neuropsychology, 33, 562-576. doi: 10.1093/arclin/acx096 Battista, P., Salvatore, C., & Castiglioni, I. (2017). Optimizing neuropsychological assessments for cognitive, behavioral, and functional impairment classification: A machine learning study. Behavioural Neurology, 2017. doi:10.1155/2017/1850909 Baum, C. M., Wolf, T. J., Wong, A., Chen, C. H., Walker, K., Young, A. C., Carlozzi, N. E., Tulsky, D. S., Heaton, R. K., & Heinemann, A. W. (2017). Validation and clinical utility of the executive function performance test in persons with traumatic brain injury. Neuropsychological rehabilitation, 27, 603–617. doi: 1080/09602011.2016.1176934 Benjamin, A. S., Fernandes, H. L., Tomlinson, T., Ramkumar., Versteeg, C., Chowdhury, R., Miller, L., Kording, K. P. (2017). Modern machine learning far outperforms GLMS at predicting spikes.. Front. Comput. Neurosci, 12-56. doi: 10.3389/fncom.2018.00056 Benton, A. L., & Hamsher, K. (1976). Multilingual aphasia examination. Iowa City: University of Iowa Press. Berit, A., and Ove, D. (1998). The clock-drawing test. Age Aging, 27, 399–403. doi: 10.1093/ageing/afs149 Binder, L.M., Iverson, G. L., & Brooks, B. L. (2009). To erris human: "abnormal" neuropsychological scores and variability are common in healthy adults. Archives of Clinical Neuropsychology, 24, 31– 46. doi:10.1093/arclin/acn001 Boake, C. & Block, C. (2015, August 1). How is Neuropsychological Assessment Different from Cognitive Testing? The Society For Clinical Neuropsychology. Retrieved from https://www.scn40.org/neuroblog/how-is-neuropsychological-assessment-different-from 57 cognitive-testing#comments Bornstein, R. A., & Leason, M. (1985). Effects of localized lesions on the Verbal Concept Attainment Test. Journal of Clinical and Experimental Neuropsychology, 7, 421–429. doi: 10.1080/01688638508401274 Breiman, L. (2001). Statistical modeling: The two cultures. Statistical Science, 16, 199–231. doi: 10.1214/ss/1009213726 Brown J (1958) Some tests of the decay theory of immediate memory. Q J Exp Psychol 10: 12– 21. doi: 10.1080/17470215808416249 Brownlee, J. (2019, May 23). How Much Training Data is Required for Machine Learning? Machine Learning Mastery. https://machinelearningmastery.com/much-training-data-required-machine-learning/ Buhi, E. R., Goodson, P., & Neilands, T. B. (2008). Out of sight, not out of mind: Strategies for handling missing data. American Journal of Health Behavior, 32, 83–92. doi: 10.5555/ajhb.2008.32.1.83 Buschke H. 1973. Selective reminding for analysis of memory and learning. J Verbal Learn Verbal Behav. 12:543–550. doi:10.1016/S0022-5371(73) 80034-9 Buntin, M. B., Colla, C. H., Deb, P., Sood, N., Escarce, J. J. (2010) Medicare spending and outcomes after postacute care for stroke and hip fracture. Med Care, 48, 776–784. doi: 10.1097/MLR.0b013e3181e359df. Carpenter, C. R., Bassett, E. R., Fischer, G. M., Shirshekan, J., Galvin, J. E., & Morris, J. C. (2011). Four sensitive screening tools to detect cognitive dysfunction in geriatric emergency department patients: Brief Alzheimer’s Screen, Short Blessed Test, Ottawa 3DY, and the caregiver-completed AD8. Academic Emergency Medicine: Official Journal of the Society for Academic Emergency Medicine, 18, 374–384. doi: 10.1111/j.1553-2712.2011.01040.x 58 CDC. (2022, May 4). Stroke. Centers for Disease Control and Prevention. National Center for Chronic Disease Prevention and Health Promotion , Division for Heart Disease and Stroke Prevention. https://www.cdc.gov/stroke/about.htm Champod, A. S., Gubitz, G. J., Phillips, S. J., Christian, C., Reidy, Y., Radu, L. M., Darvesh, S., Reid, J. M., Kintzel, F., & Eskes, G. A. (2019). Clock Drawing Test in acute stroke and its relationship with long-term functional and cognitive outcomes. The Clinical Neuropsychologist, 33, 817-830. doi:10.1080/13854046.2018.1494307 Chan, R. C., Shum, D., Toulopoulou, T., & Chen, E. Y. (2008). Assessment of executive functions: Review of instruments and identification of critical issues. Archives of clinical neuropsychology, 23, 201-216. doi:10.1016/j.acn.2007.08.010 Chen L and Savalei V (2021) Three Sample Estimates of Fraction of Missing Information From Full Information Maximum Likelihood. Front. Psychol. 12:667802. doi: 10.3389/fpsyg.2021.667802 Chételat, G., Villemagne, V. L., Bourgeat, P., Pike, K. E., Jones, G., Ames, D., et al. (2010). Relationship between atrophy and beta-amyloid deposition in Alzheimer disease. Annals of Neurology, 67, 317–324. doi:10.1002/ana.21955 Chételat, G., Villemagne, V. L., Villain, N., Jones, G., Ellis, K. A., Ames, D., et al. (2012). Accelerated cortical atrophy in cognitively normal elderly with high β-amyloid deposition. Neurology, 78, 477–484. doi:10.1212/WNL.0b013e318246d6 Cohen, A. D., Chen, Z., Parker Jones, O., Niu, C., & Wang, Y. (2020). Regression‐based machine‐learning approaches to predict task activation using resting‐state fMRI. Human Brain Mapping, 41, 815–826. https://doiorg.ezproxy.library.ubc.ca/10.1002/hbm.24841 Conti, J., Sterr, A., Brucki, S. M. D., & Conforto, A. B. (2015). Diversity of approaches in assessment of executive functions in stroke: Limited evidence?. eNeurologicalSci, 1, 12-20. 59 doi:10.1016/j.ensci.2015.08.002 Cox, B. E., McIntosh, K., Reason, R. D., & Terenzini, P. T. (2014). Working with missing data in higher education research: A primer and real-world example. The Review of Higher Education, 37, 377–402. doi: 10.1353/rhe.2014.0026 Cumming, T. B., Marshal, R. S., & Lazar, R. M. (2013). Stroke, cognitive deficits, and rehabilitation: still an incomplete picture. Int J Stroke, 8. 38-45. doi:10.1111/j.1747-4949.2012.00972.x Dahl, T. H. (2002). International Classification of Functioning, Disability and Health: an introduction and discussion of its potential impact on rehabilitation services and research. J Rehabil Med, 34. 201–4. doi: 10.1080/165019702760279170 Davidoff, G, N., Roth E, J., Haughton, J, S., Ardner, M. S. (1990) Cognitive dysfunction in spinal cord injury patients: sensitivity of the Functional Independence Measure subscales vs neuropsychologic assessment. Arch Phys Med Rehabil, 71: 326–329. Delis, D. C., Kramer, J. H., Kaplan, E., & Ober, B. A. (2000). The California Verbal Learning Test-II. (2nd ed.). The Psychological Corporation. D’Elia, L. F., Satz, P., Uchiyama, C. L., & White, T. (1996). Color Trails Test: Professional manual. Lutz, FL: Psychological Assessment Resources. Dobkin, B. H. (2005). Clinical Practice. Rehabilitation after Stroke. N Engl J Med, 352, 1677-84. doi: 10.1056/NEJMcp043511 Dominic, A., Carone, E., Strauss, E., M. S. Sherman, & O. Spreen. (2006). A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary, Applied Neuropsychology, 14, 62-63, doi:10.1080/09084280701280502 Donders, J. (2019). The incremental value of neuropsychological assessment: A critical review. The Clinical Neuropsychologist, 34, 56-57. doi:10.1080/13854046.2019.1575471 Dong, Y. Peng, C.Y. (2013). From Principled missing data methods for researchers. Springerplus. Page 60 222. From: https://cran.r-project.org/web/packages/caret/index.html Finch, W. Holmes, & Finch, Maria E. Hernandez (2016). Regularization Methods for Fitting Linear Models with Small Sample Sizes: Fitting the Lasso Estimator using R. Practical Assessment, Research & Evaluation, 21. doi: 10.7275/jr3d-cq04 Folstein, M. F., Folstein, S. E., McHugh, P. R. (1975). “Mini-mental state”. A practical method for grading the cognitive state of patients for the clinician. J. Psychiatr. Res., 12, 189-198. doi: 10.1016/0022-3956(75)90026-6 Frank, B., Hurley, L., Scott, T. M., Olsen, P., Dugan, P., & Barr, W. B. (2018). Machine learning as a new paradigm for characterizing localization and lateralization of neuropsychological test data in temporal lobe epilepsy. Epilepsy & Behavior, 86, 58-65. doi:10.1016/j.yebeh.2018.07.006 Fraser, K. C., Meltzer, J. A., Graham, N. L., Leonard, C., Hirst, G., Black, S. E., & Rochon, E. (2014). Automated classification of primary progressive aphasia subtypes from narrative speech transcripts. Cortex, 55, 43. doi: 10.1016/j.cortex.2012.12.006 Friedman, J., Hastie, T., Tibshirani, R. (2010). Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software, 33, 1–22. doi:10.18637/jss.v033.i01 Freedman M. I., Leach. L., Kaplan. E., Winocur. G., Shulman K. J., Delis DC eds. Clock Drawing. Oxford: Oxford University Press, 1994. Friedman, J., Hastie, T., Tibshirani, R., Narasimhan, B., Tay, K., SImon, N., Qian, J., Yang, J. (2021). glmnet: Lasso and Elastic-Net Regularized Generalized Linear Models. R package version 4.1-3. Fuh, J. L., Liu, H. C., Wang, S, J., Liu, C. Y. & Wang, P, N. (1997). Poststroke depression among the Chinese elderly in a rural community. Stroke, 28, 1126-9. doi: 10.1161/01.str.28.6.1126 Goldman, J. G., Holden, S., Ouyang, B., Bernard, B., Goetz, C. G., & Stebbins, G. T. (2015). Diagnosing PD‐MCI by MDS task force criteria: How many and which neuropsychological tests?. Movement Disorders, 30, 402-406. doi:10.1002/mds.26084 61 Goldstein, S., Naglieri, J.A., Princiotta, D., Otero, T.M. (2014). Introduction: A History of Executive Functioning as a Theoretical and Clinical Construct. In: Goldstein, S., Naglieri, J. (eds) Handbook of Executive Functioning. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8106-5_1 Granger, C. V. Deutsch, A. Linn, R. T. (1998). Rasch analysis of the functional independence measure (FIMtm) mastery test. Archives of Physical Medicine and Rehabilitation, 79, 52-57. https://doi.org/10.1016/S0003-9993(98)90208-8 Granger, C. V., Hamilton, B. B., Linacre, J. M., Heinemann, A. W., Wright, B. D. (1993). Performance profiles of the functional independence measure. Am J Phys Med Rehabil, 72, 84-9. DOI: 10.1097/00002060-199304000-00005 Greenwood, C. J., Youssef, G. J., Letcher, P., Macdonald, J. A., Hagg, L. J., Sanson, A., Mcintosh, J., Hutchinson, D. M., Toumbourou, J. W., Fuller-Tyszkiewicz, M., & Olsson, C. A. (2020). A comparison of penalised regression methods for informing the selection of predictive markers. PloS one, e0242730. doi: 10.1371/journal.pone.0242730 Gresham, G. E., Alexander, D., Bishop, D. S. et al. (1997). American Heart Association Prevention Conference. IV. Preven-tion and Rehabilitation of Stroke. Rehabilitation. Stroke, 28, 1522-6. doi: 10.1161/01.str.28.7.1522 Gurevich, P., Stuke, H., Kastrup, A., Stuke, H., & Hildebrandt, H. (2017). Neuropsychological testing and machine learning distinguish Alzheimer’s disease from other causes for cognitive impairment. Frontiers in Aging Neuroscience. 9, 114. doi: 10.3389/fnagi.2017.00114 Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. New York, NY: Springer. Hatano, S. (1976). Experience from a multicentre stroke register: a preliminary report. Bulletin of the World Health Organization, 54, 541 - 553. https://apps.who.int/iris/handle/10665/260937 62 Hatano S. (1976). Experience from a multicentre stroke register: a preliminary report. Bulletin of the World Health Organization, 54(5), 541–553. Heaton, R. K., Grant, I., & Matthews, C. G. (1991). Comprehensive norms for an extended halstead-Reitan battery: Demographic corrections, research findings, and clinical applications. Odessa, FL: Psychological Assessment Resources. Heaton, R. K., Miller, S. W., Taylor, M. J., & Grant, I. (2004). Revised comprehensive norms for an extended Halstead-Reitan battery: Demographically adjusted neuropsychological norms for African American and Caucasian adults professional manual. Lutz, FL: Psychological Assessment Resources. Heaton, R.K. (1980). A manual for the Wisconsin Card Sorting Test. Odessa, FL: Psychological Assessment Resources, Inc. Herrmann, N., Black, S, E., Lawrence, J., Szekely, C. & Szalai, J, P. (1998). The Sunnybrook Stroke Study: a prospective study of depressive symptoms and functional outcome. Stroke, 29, 618-24. doi: 10.1161/01.STR.29.3.618 Heruti, Raphael & Lusky, Ayala & Dankner, Rachel & Ring, Haim & Dolgopiat, Mark & Barell, Vita & Levenkrohn, Shalom & Adunsky, Abraham. (2002). Rehabilitation outcome of elderly patients after a first stroke: Effect of cognitive status at admission on the functional outcome. Archives of physical medicine and rehabilitation, 83, 742-9. doi:10.1053/apmr.2002.32739. Hochstenbach, J. B., Anderson, P. G., van Limbeek, J., Mulder, T, T. (2001). Is there a relation between neuropsychological variables and quality of life after stroke? Arch Phys Med Rehabil, 82, 1360-6. doi:10.1053/apmr.2001.25970 Hornung, Roman. (2019). Ordinal Forests: Prediction and Variable Ranking with Ordinal Target Variables. R package version 2.3-1. Retrieved from: https://cran.rproject.org/web/packag es/ordinalFores t/ordinalForest.pdf 63 Ingraham, L. J., & Aiken, C. B. (1996). An empirical approach to determining criteria for abnormality in test batteries with multiple measures. Neuropsychology, 10, 120–124. doi: 10.1037/0894-4105.10.1.120 Iosifescu, D, V. (2012). The relation between mood, cognition and psychosocial functioning in psychiatric disorders. European Neuropsychopharmocology, 22, S499-S504. doi: 10.1016/j.euroneuro.2012.08.002 James, G., Witten, D., Hastie, T., & Tibshirani, R. (2014). An Introduction to Statistical Learning: With Applications in R. Springer Publishing Company, Incorporated. Joel, S., Eastwick, P. W., & Finkel, E. J. (2017). Is romantic desire predictable? Machine learning applied to initial romantic attraction. Psychological Science, 28, 1478–1489. doi: 10.1177/0956797617714580 Johnston, K. C., Connors, A. F. Jr, Wagner, D. P. et al. (2000). A predictive risk model for outcomes of ischemic stroke. Stroke, 31, 448-55. doi: 10.1161/01.str.31.2.448 Jokinen, H., Melkas, S., Ylikoski, R., Pohjasvaara, T., Kaste, M., Erkinjuntti, T., & Hietanen, M. (2015). Post‐stroke cognitive impairment is common even after successful clinical recovery. European Journal of Neurology, 22, 1288–1294. doi: 10.1111/ene.12743 Keith, R. A., Granger, C. V., Hamilton, B. B., & Sherwin, F. S. (1987). The functional independence measure. Adv Clin Rehabil, 1, 6-18. Kase, C. S., Wolf, P. A., Kelly-Hayes, M et al. (1998). Intellectual decline after stroke: the Framingham Study. Stroke, 29, 805-12. doi: 10.1161/01.str.29.4.805 Kidd, D. Stewart, G. Baldry, J. Johnson, J. Rossiter, D. Petruckevitch, D & Thompson, A. J. (1995) The Functional Independence Measure: A comparative validity and reliability study, Disability and Rehabilitation, 17, 10-14, doi: 10.3109/09638289509166622 Ko, H., Ihm, J. J., Kim, H. G., & Alzheimer’s Disease Neuroimaging Initiative (2019). Cognitive 64 Profiling Related to Cerebral Amyloid Beta Burden Using Machine Learning Approaches. Frontiers in aging neuroscience, 11, 95. https://doi.org/10.3389/fnagi.2019.00095 Kübler, R. V., Colicev, A., & Pauwels, K. H. (2020). Social media's impact on the consumer mindset: When to use which sentiment extraction tool?. Journal of Interactive Marketing, 50, 136– 155. doi: 10.1016/j.intmar.2019.08.001 Kroenke, K., Spitzer, R. L., Williams, J. B. W. (2001). The PHQ-9. Journal of General Internal Medicine, 16, 606-613. doi:10.1046/j.1525-1497.2001.016009606.x Lafayette Instrument. (2002). Grooved Pegboard user’s manual. Lafayette, IN. Lever, J., Krzywinski, M., & Altman, N. (2016). Points of Significance: Model selection and overfitting. Nature methods, 13, 703-704. doi: 10.1038/nmeth.3968 Little, R. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1198–1202. doi: 10.2307/2290157 Reitan, R. M. (1955). The relation of the trail making test to organic brain damage. Journal of Consulting Psychology, 19, 393–394. doi: 10.1037/h0044509Malec, J. (2004). Comparability of the mayo-portland adaptability inventory ratings by staff, significant others and people with acquired brain injury. Brain Injury, 18, 563–575. doi:10.1080/02699050310001646134 Malec, J. F., Kragness, M., Evans, R. W., Finlay, K. L., Kent, A., & Lezak, M. D. (2003). Further psychometric evaluation and revision of the Mayo-Portland adaptability inventory in a national sample. The Journal of Head Trauma Rehabilitation, 18(6), 479–492. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/1470787810.1097/00001199-200311000-00002 Mesulam, M. (1985). Principles of behavioral neurology: Verbal and Nonverbal Cancellation Test. Philadelphia: FA Davis Company. Oleszak, M. (2019, November 12). Regularization: Ridge, Lasso and Elastic Net. Data Camp. https://www.datacamp.com/community/tutorials/tutorial-ridge-lasso-elastic-net 65 Paciaroni M, Arnold P, Van Melle G and Bogousslavsky J. (2000). Severe disability at hospital discharge in ischaemic stroke survivors. European Neurology, 43, 30-4. doi: 10.1159/000008125 Paolucci, S., Antonucci, G., Gialloreti, L.E. et al. (1996). Predicting stroke inpatient rehabilitation outcome. The prominent role of neuro-psychological disorders. European Neurology, 36, 385-90. doi: 10.1159/000117298 Plonsky, O., Erev, I., Hazan, T., & Tennenholtz, M. (2017). Psychological forest: Predicting human behavior. In: The Pro-ceedings of the Thirty‐First AAAI Conference on Artificial Intelligence (AAAI‐17) (pp. 656–662). R Core Team 2020 for version of R used in the analysis. M. Alice. Imputing missing data with R; MICE package. (2015). Available at https://datascienceplus.com/imputing-missing-data-with-r-mice-package/, Accessed 29th Jan 2018 Riberto, M., Miyazak, M. H., Jucá, S.S., Sakamoto, H., Pinto, P.P.N., Battistella, L.N. (2004). Validation of brazilian version of Functional Independence Measure. Acta Fisiatr, 11, 72–76. doi: 10.1590/0004-282x20140066 Mike R Schoenberg, Katie E Osborn, E Mark Mahone, Maia Feigon, Robert M Roth, Neil H Pliskin. (2018). Physician Preferences to Communicate Neuropsychological Results: Comparison of Qualitative Descriptors and a Proposal to Reduce Communication Errors, Archives of Clinical Neuropsychology, 33, 631-643, doi: 10.1093/arclin/acx106 Schretlen, D. J., Munro, C. A., Anthony, J. C., & Pearlson, G. D. (2003). Examining the range of normal intraindividual variability in neuropsychological test performance. Journal of the International Neuropsychological Society, 9, 864– 870. doi: 10.1017/S1355617703960061 Health Care, Stanford. “Purpose of Neuropsychology Assessment.” Stanford Health Care (SHC) - Stanford Medical Center, 2019, stanfordhealthcare.org/medical-clinics/neuropsychology- 66 clinic/what-to-expect/purpose-of -assessment.html.l Hankey, G. J., Spiesser, J., Hakimi, Z., Bego, G., Carita, P., Gabriel, S. (2007). Rate, degree, and predictors of recovery from disability following ischemic stroke. Neurology, 68, 1583-1587. doi: 10.1212/01.wnl.0000260967.77422.97 Hakkennes, S. J., Brock, K., Hill, K. D. (2011). Selection for inpatient rehabilitation after acute stroke: a systematic review of the literature. Arch Phy Med Rehabil, 92, 2057-2070. doi: 10.1016/j.apmr.2011.07.189 Hilborn, J. V., Strauss, E., Hultsch, D. F., & Hunter, M. A. (2009). Intraindividual variability across cognitive domains: Investigation of dispersion levels and performance profiles in older adults. Journal of Clinical and Experimental Neuropsychology, 31, 412– 424. doi: 0.1080/13803390802232659 Hornung, R. (2019). Ordinal Forests. Journal of Classification, 37, 4-17. doi: 10.1007/s00357-018-9302-x Ingraham, L. J., & Aiken, C. B. (1996). An empirical approach to determining criteria for abnormality in test batteries with multiple measures. Neuropsychology, 10, 120–124. doi: 10.1037/0894-4105.10.1.120 Lee, R. S. C., Hermens, D. F., Naismith, S. L., Lagopoulos, J., Jones, A., Jones, A., Chitty, K. M., White, D., Robillard, R., Scott, E. M., Hickie, I. B. (2015). Neuropsychological and functional outcomes in recent-onset major depression, bipolar disorder and schizophrenia-spectrum disorders: a longitudinal cohort study. Translational Psychiatry, 5, PMC4462613 doi: 10.1038/tp.2015.50 Lees, R., Fearon, P., Harrison, J. K., Broomfield, N. M., & Quinn, T. J. (2012). Cognitive and mood assessment in stroke research. Stroke, 43, 1678-1680. doi: 10.1080/13854046.2019. 1575471 Leitner, D., Miller, H., Libben, M. (2018). Assessing the predictive value of a neuropsychological 67 model on concurrent function in acute stroke recovery and rehabilitation. The Clinical Neuropsychologist, 33, 831-853, doi: 10.1080/13854046.2018.1487586 Lezak M,D. Neuropsychological Assessment, 3rd Edition. New York, Oxford University Press, 1995. Lezak, M. D., Howieson, D. B., Loring, D. W., Hannay, H. J., & Fischer, J. S. (2004). Neuropsychological assessment (4th ed.). New York, NY, US: Oxford University Press. Linacre, J. M., Heinemann , A. W., Wright, B. D., Granger, C. V., Hamilton, B. B. (1994). The structure and stability of the functional independence measure. Archives of Physical Medicine and Rehabilitation, 75, 127 – 132. doi: 10.1016/0003-9993(94)90384-0 Liu, X., Cao, P., Zhao, D., & Banerjee, A. (2016). Multi-task spare group lasso for characterizing Alzheimer’s disease. In 5th Workshop on Data Mining for Medicine and Healthcare (Vol. 49). Madley-Dowd, P., Hughes, R., Tilling, K., Heron, J. (2019) The proportion of missing data should not be used to guide decisions on multiple imputation. Journal of Clinical Epidemiology, 110, 63-73. doi: 10.1016/j.jclinepi.2019.02.016 Madley-Dowd, P., Hughes, R., Tilling, K., Heron, J. (2019). The proportion of missing data should not be used to guide decisions on multiple imputation. Journal of Clinical Epidemiology, 110, 63-73. doi: 10.1016/j.jclinepi.2019.02.016 Maj, M., D’Elia, L., Satz, P., Janssen, R., Zaudig, M., Uchiyama, C., … Chervinsky, A. (1993). Evaluation of two new neuropsychological tests designed to minimize cultural bias in the assessment of HIV-1 seropositive persons: A WHO study. Archives of Clinical Neuropsychology, 8, 123–135. doi: 10.1016/0887-6177(93)90030-5 McClure, J.A., Salter, K., Foley, N., Mahon, H., Teasell, R. (2012). Adherence to Canadian Best Practice Recommendations for Stroke Care: vascular cognitive impairment screening and assessment practices in an Ontario inpatient stroke rehabilitation facility. Top Stroke Rehabil, 19.141–148. doi: 10.1310/tsr1902-141 68 Meehl P. Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis: University of Minnesota Press 1954. doi: 10.1037/11281-000 Mehool, D. P., Coshal, C., Rudd, A. G., Charles, D. A., & Wolfe, M. D. (2002). Cognitive Impairment after Stroke: Clinical Determinants and Its Associations with Long Term Stroke Outcomes. Journal of the American Geriatrics Society, 50, 700–706. https://doi.org/10.1046/j.1532-5415.2002.50165.x Miller, E.L., Murray, L., Richards, L., Zorowitz, R.D., Bakas, T., Clark, P., Billinger, S.A., & American Heart Association Council on Cardiovascular Nursing and the Stroke council (2010). Comprehensive overview of nursing and interdisciplinary rehabilitation care of the stroke patient: a scientific statement from the American Heart Association. Stoke, 41, 2402–2448. doi: 10.1161/STR.0b013e3181e7512b Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive psychology, 41, 49-100. doi: 10.1006/cogp.1999.0734 Miyake, A., Emerson, M. J., & Friedman, N. P. (2000). Assessment of executive functions in clinical settings: problems and recommendations. Seminars in speech and language, 21, 169–183. https://doi.org/10.1055/s-2000-7563 Mozaffarian, D., Benjamin, E. J., Go, A. S., Arnett, D. K., Blaha, M. J., Cushman, M., ... & Huffman, M. D. (2015). Executive Summary: Heart Disease and Stroke Statistics – 2015 Update. Circulation, 131, 434-441. doi:10.1161/CIR.0000000000000157 Murakami, M., Inouye, M. (2002). Stroke rehabilitation outcome study: a comparison of Japan with the United States. Am J Phys Med Rehabil, 81, 279-82. doi: 10.1097/00002060-200204000-00006 Murray, C. J. L., Barber, R. M., Foreman, K. J., Ozgoren, A. A., Abd-Allah, F., Abera, S. F., … Brugha et al. (2015). Global, regional, and national disability-adjusted life years (DALYs) for 306 69 diseases and injuries and healthy life expectancy (HALE) for 188 countries, 1990–2013: Quantifying the epidemiological transition. The Lancet, 386, 2145–2191. doi: 10.1016/S0140-6736(15)61340-X Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbonneau, S., Whitehead, V., Collin, I.,... Chertkow, H. (2005). The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society, 53, 695– 699. http://dx.doi.org/10.1111/j.1532-5415.2005.53221.x National Collaborating Centre for Chronic Conditions (UK). (2008). Stroke: National Clinical Guideline for Diagnosis and Initial Management of Acute Stroke and Transient Ischaemic Attack (TIA). Royal College of Physicians (UK). Nys, G. M., Zandvoort, M. J., de Kort, P. L., Jansen, B. P., de Haan, E. H., & Kappelle, L. J. (2007). Cognitive disorders in acute stroke: Prevalence and clinical determinants. Cerebrovascular Disease, 23, 408-416. doi:10.1159/000101464 D. Opitz, R. Maclin. (1999). Popular Ensemble Methods: an Empirical Study. J. Artif. Intell. Res., 11, 169-198. doi: 10.48550/arXiv.1106.0257 Ottenbacher K. J., Smith P. M., Illig S. B., Linn R. T., Ostir G.V., Granger C.V. (2004). Trends in length of stay, living setting, functional outcome, and mortality following medical rehabilitation. JAMA, 292, 1687-95. doi: 10.1001/jama.292.14.1687 Ozdemir, F., Birtane, M., Tabatabaei, R., Ekuklu, G., & Kokino, S. (2001). Cognitive Evaluation and Functional Outcome After Stroke. American journal of physical medicine & rehabilitation, 80, 410–415. doi: 10.1097/00002060-200106000-00003 Remington, G., Foussias, G. & Agid, O. (2010). Progress in Defining Optimal Treatment Outcome in Schizophrenia. CNS Drugs, 24, 9–20. doi: 10.2165/11530250-000000000-00000 Weakley, A., Williams, J. A., Schmitter-Edgecombe, M., & Cook, D. J. (2015). Neuropsychological test 70 selection for cognitive impairment classification: A machine learning approach. Journal of Clinical and Experimental Neuropsychology, 37, 899–916. doi:10.1080/13803395.2015.1067290
Thesis/Dissertation
eng
Psychology
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
O'Connor, Brian
Neuropsychology’s machine assistant : predicting functional outcomes with machine learning
Text