Arts and Social Sciences, Irving K. Barber Faculty of (Okanagan)
Psychology, Department of (Okanagan)
DSpace
UBCO
Leitner, Damian William
Doctor of Philosophy - PhD
University of British Columbia
Each year approximately 50,000 Canadians, one million Americans, and millions of people worldwide are hospitalized for stroke (Mozzaffarian, Benjamin, Go, et al., 2015). Cognitive impairment is common after experiencing a stroke and known to affect functioning on daily tasks (Nys et al., 2005; Barker-Collo & Feigin, 2006). While neuropsychological assessments are often employed to assess cognitive abilities and make inferences about functional capability there is growing interest in integrating contemporary technologies to augment the assessment (Burgess, Alderman, Forbes, et al., 2006; Bilder, 2011; Parsey & Schmitter-Edgecombe, 2013; Parsons, 2016). Eye tracking allows previously overlooked information, overt visual attention based on fixations and saccades, to be quantified to help elucidate how responses are made during testing. The current dissertation, comprised of three studies, investigated the validity of eye tracking during completion of the computerized Wisconsin Card Sorting Test (cWCST), a common test of higher-level cognition. Overall, current findings supported the validity of eye tracking with the cWCST across separate population samples. More specifically, results across the three studies provided initial evidence for the construct and criterion validity of fixations and saccades on cWCST, as well as the ecological validity for inpatients recovering from a stroke. Fixation counts, dwell times, and average time per fixation, as well as saccade count were shown to be influenced by cognitive load and response styles and also differentiated between healthy older adults and those who experienced a stroke. Last, findings also found that select fixation measures had concurrent and predictive validity for inpatients recovering from a stroke. This dissertation provided preliminary support for eye tracking during neuropsychological assessment of cognitive functions. Clinical implications and areas for future research are also discussed.
UPDATING OUR APPROACH TO NEUROPSYCHOLOGICAL ASSESSMENTS: EXAMINING THE VALIDITY OF EYE TRACKING WITH THE COMPUTERIZED WISCONSIN CARD SORTING TEST by Damian William Leitner B.Sc. (Hons.), University of British Columbia, 2009 M.A., The University of British Columbia, 2016 A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE COLLEGE OF GRADUATE STUDIES (Clinical Psychology) THE UNIVERSITY OF BRITISH COLUMBIA (Okanagan) April 2021 © Damian William Leitner, 2021 Page ii of 178 The following individuals certify that they have read, and recommend to the College of Graduate Studies for acceptance, a thesis/dissertation entitled: Updating our Approach to Neuropsychological Assessments: Examing the Validity of Eye Tracking with the Computerized Wisconsin Card Sorting Test submitted by Damian Leitner in partial fulfillment of the requirements of the degree of Doctor of Psychology. Dr. Maya Libben, Irving K. Barber Faculty of Arts and Social Sciences Supervisor Dr. Harry Miller, Irving K. Barber Faculty of Arts and Social Sciences Supervisory Committee Member Dr. Susan Holtzman, Irving K. Barber Faculty of Arts and Social Sciences Supervisory Committee Member Dr. Mary Jung, School of Health and Exercise Sciences University Examiner Dr. John Woodard, Wayne State University External Examiner Additional Committee Members include: Dr. Barbara Rutherford, Irving K. Barber Faculty of Arts and Social Sciences Supervisory Committee Member Page iii of 178 Abstract Each year approximately 50,000 Canadians, one million Americans, and millions of people worldwide are hospitalized for stroke (Mozzaffarian, Benjamin, Go, et al., 2015). Cognitive impairment is common after experiencing a stroke and known to affect functioning on daily tasks (Nys et al., 2005; Barker-Collo & Feigin, 2006). While neuropsychological assessments are often employed to assess cognitive abilities and make inferences about functional capability there is growing interest in integrating contemporary technologies to augment the assessment (Burgess, Alderman, Forbes, et al., 2006; Bilder, 2011; Parsey & Schmitter-Edgecombe, 2013; Parsons, 2016). Eye tracking allows previously overlooked information, overt visual attention based on fixations and saccades, to be quantified to help elucidate how responses are made during testing. The current dissertation, comprised of three studies, investigated the validity of eye tracking during completion of the computerized Wisconsin Card Sorting Test (cWCST), a common test of higher-level cognition. Overall, current findings supported the validity of eye tracking with the cWCST across separate population samples. More specifically, results across the three studies provided initial evidence for the construct and criterion validity of fixations and saccades on cWCST, as well as the ecological validity for inpatients recovering from a stroke. Fixation counts, dwell times, and average time per fixation, as well as saccade count were shown to be influenced by cognitive load and response styles and also differentiated between healthy older adults and those who experienced a stroke. Last, findings also found that select fixation measures had concurrent and predictive validity for inpatients recovering from a stroke. This dissertation provided preliminary support for eye tracking during neuropsychological assessment of cognitive functions. Clinical implications and areas for future research are also discussed. Page iv of 178 Lay Summary Neuropsychologists often asked to assess people who had a stroke and have problems with thinking and memory (i.e., cognition). Currently, the most common assessment method is paper-pencil testing with limited use of modern computerized/digital technology. For decades, eye tracking has helped researchers understand cognition and performance on daily tasks. The present dissertation investigated the use of eye tracking with traditional cognitive tests during assessment. Study 1 provided an initial investigation in a university lab setting and results suggested eye movement patterns are affected by test difficulty and predicted performance. Study 2 and 3 extended these results to hospital inpatients who experienced a stroke and healthy individuals from the community. Eye movements on testing differentiated between these two groups and helped to predict recovery of function after a stroke. Overall, results of this dissertation offer exciting new areas of research involving the use of eye tracking in clinical and experimental settings. Page v of 178 Preface The University of British Columbia’s Okanagan Behavioural Research Ethics Board granted ethics approval for this research on March 17th, 2017. The ethics approval certificate number for this research is H16-02762. As of the date of this submission, the data included in this dissertation has not been published. Page vi of 178 Table of Contents Abstract ........................................................................................................................................iii Lay Summary ...............................................................................................................................iv Preface ............................................................................................................................................v Table of Contents .........................................................................................................................vi List of Tables .................................................................................................................................x List of Figures..............................................................................................................................xii Acknowledgements ....................................................................................................................xiii Dedication ...................................................................................................................................xiv Chapter 1: Introduction………................................................................................................... 1 1.1 Neuropsychological Assessment............................................................................................1 1.1.2 Validity of Neuropsychological Testing…......................................................................2 1.1.3 Qualitative Analysis Approach to Neuropsychological Assessment…...........................4 1.2 Attention Underlies Higher Order Functioning..……………………………………...…7 1.2.1 Assessment of Executive Function…………….………………………………….……8 1.2.2 Wisconsin Card Sorting Test…………………………………………………………..10 1.3 Advancing Neuropsychological Assessment………………………………………………13 1.3.1 Computerized Neuropsychological Assessment………………….……..…………….16 1.4 Eye Tracking…………………………………………………….………………………….20 1.4.1 Eye Tracking and Cognitive Load………….…………………….…………………....23 1.4.2 Eye Tracking and Cognitive Function…………………………………………………24 1.4.3 Eye Tracking for Clinical Assessment………………………………………….….…..26 1.4.4 Eye Tracking and Daily Activities/Functioning…………………………………….…28 Page vii of 178 1.4.5 Eye Tracking Facilitating Function…………………………………………………...29 1.5 Stroke …………………………………..…………………………………….……...……30 1.5.1 Consequences of Stroke………………………….………………………………..…...31 1.5.2 Assessment of Cognitive Function After Stroke….…………………………………....33 1.5.3 Neuropsychological Assessment of Stroke…………………………………………….34 1.5.4 Eye Tracking Assessment Following Stroke…………………………………………..35 1.6 Summary of Literary Review……………………...……………………………………..37 Chapter 2: Study 1 - Pilot Study for Eye Tracking during Cognitive Testing...................... 40 2.1 Study 1 Purpose ……….......................................................................................................40 2.2 Study 1: Methodology………. ............................................................................................43 2.2.1 Participants...…………..………………………………………………………………43 2.2.2 Materials and Procedures………………………………………………………………44 2.3 Study 1: Results and Discussion..........................................................................................51 2.3.1 Results and Discussion of Hypothesis #1……..…..…………………………………...55 2.3.2 Results and Discussion of Hypothesis #2………..……..……….….………………….59 2.3.3 Results and Discussion of Hypothesis #3……….........……….……………………….61 2.4 Study 1: General Discussion….…………………………………….…………………….66 Chapter 3: Study 2 - Assessing Criterion and Construct Validity of Eye Tracking in Clinical and Community Sample ……………………………….…………………..................68 3.1 Study 2 Purpose…..…..........................................................................................................68 3.2 Methodology………………..................................................................................................72 3.2.1 Changes in Methodology From Study 1.....................................................................72 3.2.2 Participants…………..................................................................................................74 Page viii of 178 3.2.3 Materials and Procedures………………………………………………………….…77 3.3 Results and Discussion……………………………...........................................................81 3.3.1 Results and Discussion of Hypothesis #1a…………………………………..….......86 3.3.2 Results and Discussion of Hypothesis #1b………….………….………….………..88 3.3.3 Results and Discussion of Hypothesis #2…...……….……….……………………..89 3.3.4 Results and Discussion of Hypothesis #3………….…………………………….….91 3.3.5 Results and Discussion of Hypothesis #4...……….……….………………………..95 3.4 Study 2: General Discussion…………………………………………………..……….100 Chapter 4: Study 3 – The Ecological Validity of Eye Tracking During Recovery From Stroke.………………………………………………………………………………………….102 4.1 Study 3 Purpose.…….......................................................................................................102 4.2 Methodology……………..................................................................................................104 4.2.1 Participants................................................................................................................104 4.2.2 Materials and Procedures…………………………………………..………………104 4.3 Results and Discussion ....................................................................................................106 4.3.1 Results and Discussion of Hypothesis #1…..…….…………………..…..…..........110 4.4 Study 3: General Discussion..…………………………………………………………..117 Chapter 5: Conclusion ……......................................................................................................118 5.1 Overall Discussion and Clinical Implications…..…….………………………………118 5.1.1 Construct Validity of Eye Tracking …………….……………..…………………..120 5.1.2 Criterion Validity of Eye Tracking………………………………………………...121 5.1.3 Ecological Validity of Eye Tracking………………………………………………122 5.1.4 Clinical Implications……………..………………………………………….……..123 Page ix of 178 5.2 Limitations………….……………….…………………………………………………126 5.3 Future Research…………………….…………………………………………………128 References ..................................................................................................................................131 Appendices……………..............................................................................................................173 Appendix A: Preliminary Analyses, Assumption Tests, Power Analyses.........................173 Appendix B: Study 1 Instructions.........................................................................................178 Page x of 178 List of Tables Table 1 Mean Scores on the Computerized Wisconsin Card Sorting Test Across Cognitive Load Groups……………………..……………………………………54 Table 2 Spearman Rho Correlation Matrix of Eye Tracking Measures with Neuropsychological Test Scores…………………………………………………62 Table 3 Hierarchical Regression (Three-level) of Demographic, Cognitive Test Scores, and Eye Tracking Data on Categories Completed from the Computerized Wisconsin Card Sorting Test………………………………………………….…63 Table 4 Hierarchical Regression (Three-level) of Demographic, Cognitive Test Scores, and Eye Tracking Data on Perseverative Errors from the Computerized Wisconsin Card Sorting Test……………………………………………………...…………64 Table 5 Cognitive Tests Comprising Each Cognitive Domain and Composite Score……76 Table 6 Demographics of Healthy Control and Inpatients Recovering From a Stroke.….83 Table 7 Raw cWCST Scores of Healthy Controls and Inpatients Recovering From a Stroke…………………………………………………………………………….85 Table 8 Logistic Regression Predicting Likelihood of Stroke Based on Fixation and Saccade Counts…………………………………………………………………..87 Table 9 Spearman Rho Correlations Between Select Cognitive Test Scores and Eye Tracking Data From the Areas of Interest on the Computerized Wisconsin Card Sorting Test for the Control Group………………………………………………96 Table 10 Spearman Rho Correlations Between Major Cognitive Domain Composite Scores and Fixations on the Areas of Interest on the Computerized Wisconsin Card Sorting Test for Inpatients Recovering From a Stroke………………………......98 Table 11 Baseline, Discharge, and Change Scores on the Functional Independence Measure for Inpatients Recovering From a Stroke…………………..…………………...107 Page xi of 178 Table 12 Correlations Between Scores on the Functional Independence Measure, Computerized Wisconsin Card Sorting Test, and Eye Tracking Data……….…109 Table 13a-c Multiple Hierarchical Regression Predicting Admission, Discharge, and Change Scores on the Functional Independence Measure ……….………………...112-114 Page xii of 178 List of Figures Figure 1 Dell Precision M4800 Laptop Equipped With the SMI Infrared Eye Tracker RED-m 250………………………………………………………………………47 Figure 2 Four Areas of Interest (AOIs) Superimposed Onto the Video Recording of the Computerized Wisconsin Card Sorting Test……………………………………..49 Figure 3 Average Total Dwell Time on Each Area of Interest on the Computerized Wisconsin Card Sorting Test……………………………………………….……57 Figure 4 Average Fixation Counts on Each Area of Interest by Conceptual or Non-Conceptual/Trial-and-Error Responses…………………………………….…….93 Page xiii of 178 Acknowledgements This dissertation would not have been possible without the circle of support from mentors, peers, and family. My sincere gratitude to my supervisors, Drs. Maya Libben and Harry Miller, for their unwavering and incredible support and guidance. The wisdom, kindness, and commitment I received from you both was unparalleled, inspiring, and instrumental in my academic and clinical training. I could not have asked for better mentors through this academic journey. I am grateful for the insights and feedback from my committee members who helped me to initially bring this project together. I truly appreciate your time and effort that was provided on this dissertation. I want to thank the members of the PLAN Lab and staff of the Department of Psychology at Kelowna General Hospital who assisted with the data collection and logistics that made this project possible. A special heartfelt thanks to my wife who has supported me throughout my years of education and gone above-and-beyond to help me reach my goals. Page xiv of 178 Dedication To my family who have inspired me and kept me motivated to see this journey through to the end. Specifically, I dedicate this to my wife, Naomi - words cannot describe how much I appreciate your understanding and support – we made it! To my daughter, Victoria, I cannot wait to see the adventures that your curiosity will take us on. “Kid, you’ll move mountains!” Page 1 of 178 CHAPTER 1 Introduction 1.1 Neuropsychological Assessment Neuropsychological assessments are conducted to determine cognitive function in patients following brain damage, brain disease, or severe mental illness (Harvey, 2012; Lezak, Howieson, Bigler, & Tranel, 2012). Referrals are typically made regarding a preliminary or initial diagnosis, differential diagnosis, treatment response, and/or prediction of function and recovery. The assessment involves a chart review for relevant history, clinical interviews, behavioural observations, and cognitive testing. The major cognitive domains assessed during testing typically include attention, memory, visuospatial skills, processing speed, language, reasoning, judgment, and problem-solving. Administration of tests is conducted under standardized conditions; stimulus presentation is controlled by the examiner (e.g., neuropsychologist or psychometrist). Test performance is typically based on accuracy and/or response time(s). Scores are compared to a healthy normative group that accounts for demographic characteristics of the examinee, such as age and education. Results from testing are particularly important for identifying cognitive strengths and weaknesses relevant to daily function (Bennett, 2001; Lezak et al., 2012). Test scores are consolidated and used, in addition to clinical interviews and chart review, to assist in case conceptualization and provide a summary of the results, including recommendations/opinions, based on the obtained data. Neuropsychological tests are designed to minimize extraneous factors while assessing an ability or cognitive function (Barr, 2001). While there is no single, “perfect” test or measure, neuropsychology has always strived for methods and tools that will help improve clinical decision-Page 2 of 178 making and reduce errors in judgment (Strauss et al., 2006, p. 28). Tests are not intended to be used in isolation for diagnostic or prognostic purposes (Barr, 2001); rather, test batteries are used to assess multiple cognitive domains to provide an overall depiction of cognitive abilities. Neuropsychological assessments are considered the ‘gold standard’ for profiling cognitive function (Kessels, 2019), and therefore, rigorous investigation of tests and methodology are frequently conducted to ensure reliable and accurate results. 1.1.2 Validity of Neuropsychological Testing The validity of neuropsychological tests across populations is critical to the assessment process as not all tests are created equal, or for similar purposes (Strauss et al., 2006, p. 3). Assessing the validity of scores from neuropsychological tests includes evidence that is: content-related, criterion-related, construct-related, and response process-related (Anastasi & Urbina, 1997). Content-related evidence pertains to the theory the test is based upon and the definition of the domain of interest, including the format, tasks, and administration. For example, a test that requires learning a word list or story details over multiple trials and then recalling this information after a timed delay would represent good content validity. Criterion-related evidence pertains to the identification of subgroups, the ability to discriminate between groups, correlation with previously established tests, and classification statistics. For example, high positive and negative predictive power of scores from a test that differentiates individuals with brain injury from healthy controls would represent good criterion validity. Construct-related evidence pertains to the development of hypotheses, evaluating psychometric properties, convergent-discriminant validity, and sensitivity to experimental manipulation. For example, a memory test with a strong correlation with other memory tests and weaker correlation with visuospatial tests would represent good construct validity. Response process evidence pertains to the analysis of test-approach, including Page 3 of 178 variables that may not be formally assessed by a test, to understand how performance is achieved. For example, identifying strategies such as ‘chunking’ items from a memory list to assist in recall would represent evidence of response process. Guidelines suggest updating of psychometric properties of tests no longer than every 10 years (Nunally & Bernstein, 1994). Important to the aim of this dissertation, it is also recommended technological methods be considered that help improve the validity of testing and relevance to daily functioning (Parsons, 2016). Neuropsychology often aims to infer daily function based on data obtained during the assessment. Therefore, ecological validity is an important aspect of test development. Ecological validity is described by Franzen and Wilhelm (1996) as the test’s verisimilitude (how well a test reflects daily tasks) and veridicality (the degree to which a test predicts behaviour in daily living). There has been concern with making inferences about daily function based on tests designed through construct-driven theories (Zimiles, 1996; Goldstein, 1996; Burgess, Alderman, Forbes, et al., 2006). Indeed, cognitive tests do not always adequately predict behaviour related to cognitive impairment in novel, unstructured, real-world settings (Chaytor, Schmitter-Edgecombe, & Burr 2006; Sbordone, 2010). Moreover, many have argued for more function-based tests that reflect real-world behaviour (Goldstein, 1996; Burgess et al., 2006). However, function-based tests, such as the Multiple Errands Tests, have also been criticized for being time-consuming, costly, difficult to replicate or standardize across settings, and for being vulnerable to the influence of extraneous variables (Parsons, 2016, p. 8; Nalder, Clark, Anderson, & Dawson, 2017). A survey of neuropsychologists in Canada and the United States suggests construct-driven tests continue to be the primary approach to assessment (Rabin, Barr & Burton, 2005; Rabin, Paolillo, & Barr, 2016). In a review of neuropsychological tests pertaining to everyday functioning, Chaytor and Schmitter-Edgecombe (2003) found that scores from many neuropsychological tests only have a moderate Page 4 of 178 level of ecological validity and moderating factors required further exploration. As well, Kessels (2019) stated, “This lack of process-purity… calls for novel paradigms that will facilitate ‘precision neuropsychology’ i.e. unraveling specific impairments in detailed cognitive processes rather than broad cognitive domains.” (p. 359). The questions asked, and investigations conducted, in neuropsychology are influenced by the tools we use to examine and understand the brain (Casaletto & Heaton, 2017). Thus, investigation is warranted into methods that improve the validity of results and the inferences made based on testing while (ideally) minimizing demands on additional required resources in implementing new methods. 1.1.3 Qualitative Analysis Approach to Neuropsychological Assessment Response process evidence (Anastasi & Urbina, 1997) is important to help highlight how test scores are achieved. While test scores provide valuable information for evaluation, qualitative information (e.g., behavioural observations) is also important in the assessment process. Observations made during the clinical interview and cognitive testing can provide evidence of how cognitive deficits manifest in daily life. This qualitative information can often put results, and the nature of the overall assessment, into context and allow for results to be individualized to clients. However, inferences about general behaviour based on presentation during assessment may be difficult given the relatively short interaction (Chaytor & Schmitter-Edgecombe, 2003). Therefore, it is important to gather as much relevant information as possible during this short window of time. Qualitative information (e.g., observations of client behaviour and approach to testing) during assessments has been purported to assist in clinical decision-making, above and beyond scores on traditional paper-pencil tests (Kaplan, 1988; Milberg, Hebben, & Kaplan, 2009; Lamar et al., 2010). Page 5 of 178 The Boston Process Approach (BPA; Kaplan, 1988) is a prominent example of assessing the qualitative nature of behaviour during a neuropsychological assessment (Milberg et al., 2009). The BPA includes a qualitative analysis of test approach, formulated strategies, and errors to help identify cognitive dysfunction. Kaplan (1988) argued that focus on total scores omits the nature and efficacy of strategies employed by the patient in arriving at a correct or incorrect solution. By monitoring the process of problem-solving and decision-making, information can be gleaned about: (mal)adaptive strategies and behaviours which may inform diagnoses; change in function over time; approach to intervention and treatment; or recommendations. The BPA has been carried out with commonly used neuropsychological tests such as the WAIS Block Design, Rey-Osterrieth Complex Figure Test, Object Assembly, Digit Symbol Test, and Clock Drawing Test (Kaplan, 1988). The BPA has been shown to help discriminate subtypes of patient populations, despite similar test scores, based on behavioural and error analysis (Milberg et al., 2009). For example, Lamar and colleagues (2010) found that the BPA was effective at differentiating patients with dysexecutive syndrome secondary to either vascular dementia or Alzheimer’s disease by examining the process of decision-making and problem-solving during testing. Specifically, distinct differences between these two groups were noted on their ability to switch mental sets, speed of processing information, and disinhibition/perseveration of responses. As well, analysis of errors and strategies during testing has been quantified into the computerized scoring systems such as the California Verbal Learning Test – Second Edition (Delis, Kramer, Kaplan, & Ober, 2000). The summary printout provides a comprehensive analysis of scores and errors for greater appreciation of the various components of memory (e.g., immediate and delayed recall, learning, recognition, and interference). Thus, relying solely on total scores, and omitting error patterns or Page 6 of 178 strategies, may result in overlooking underlying or subtle cognitive deficits that have important implications for behaviours and function in daily life. The approach of analysis of error patterns or strategies was integrated into the Framingham Health Study in the 1990s, a longitudinal multi-generational examination of risk factors and progression of dementia, which began in the 1940s (Au, Piers, & Devine, 2017). In the study, the BPA was applied to neuropsychological testing to improve diagnostic accuracy and delineate cognitive deficits that progress with the disease. The BPA has also been applied to the development or improvement of neuropsychological tests by modifying and extending existing metrics so they could capture qualitative components (Miller & Barr, 2017). However, the BPA is difficult to replicate and compare across studies because of limited standardization procedures (Au et al., 2017). It is therefore reasonable to consider methods and tools to quantify response processes for analysis to shed further light on cognitive functions. Quantifying clinical observations for analysis has been proposed as a method to further elucidate test scores and understand performance. Known as the “Quantified Process Approach” (Poreh, 2000; Poreh, 2006), observational data, previously overlooked, is compiled, analyzed, and indexed to provide a more thorough understanding of results from neuropsychological assessments. In the past, qualitative data and variables outside test scores, such as what a person is focused on, were difficult to quantify for analysis (Parsons & Duffield, 2019). In addition to providing ease of administration, scoring, and analysis, computers offer faster and more powerful methods for recording and analyzing qualitative data (Poreh, 2006; Parsons, McMahan, & Kane, 2016; Miller & Barr, 2017). Moreover, many new technologies designed and implemented in neuroscience have allowed major advances to be made in our understanding of brain-behaviour relationships (Parsons, 2016; Miller & Barr, 2017; Kessels, 2019). Au and colleagues (2017) state, Page 7 of 178 “…digital capture of [computer assessments] can be described as BPA on steroids” (p. 15), referring to their capacity to provide new insights into the brain as it engages in cognitive tasks. However, relatively few studies have investigated the use of computers for capturing qualitative information in real-time to provide information on the patient’s approach to cognitive testing. As noted above, integration of qualitative information during neuropsychological assessments is important for clarifying test results, including providing context to test scores as well as the overall assessment. 1.2 Attention Underlies Higher Order Functioning Many of our most complex activities and functions (e.g., problem solving) are dependent on basic cognitive processes, such as attention (Lezak et al., 2012). Impaired attention can result in downstream impairment of problem-solving, task-maintenance or switching, and recall (Bennett, 2001). In this regard, test batteries allow for the consideration of performance across cognitive domains to better understand the overall clinical picture. For example, if attention is impaired, encoding of new information, maintaining information in a temporary store, and being able to recall information from memory may all be adversely affected. Moreover, these difficulties may then result in poor problem-solving, because not all the information is available to make an informed response. Indeed, overt attention has been shown to predict anticipated actions and goal-oriented behaviour (Atkinson, Simpson, & Cole, 2018). Coull (1998) proposed that subdomains of attentional processes, including attentional orientation, selective attention, sustained attention, and alternating attention are purported to be the basis for higher level cognition, known as executive functions (EF). However, correlations between tests of sequential and alternating visual attention (e.g., Trail Making Test) and problem solving (e.g., Wisconsin Card Sorting Test) have Page 8 of 178 been shown to be moderate at best, perhaps due to the heterogeneous nature of the EF system (Testa, Bennett, & Ponsford, 2012). 1.2.1 Assessment of Executive Function Executive functions (EFs) monitor, control, and organize complex thoughts and behaviour for goal attainment (Anderson, 2008; Gazzaniga et al., 2014). EFs can be broadly divided into two subdomains: the “cold” logic-based processes (discussed below) and the “hot” emotion- driven processes that include social cognition, emotion regulation, affect, and theory of mind (Zimmerman, Ownsworth, O’Donovan, Roberts, & Gullo, 2016; Chan, Shum, Toulopoulou, & Chen, 2008). Considered to be complex, higher-order, cognitive functioning, the cold processes of EF can be broken down further into planning, working memory, inhibition, and mental flexibility, and are often assessed during neuropsychological testing (Chan et al., 2008; Diamond, 2013; Faria, Alves, & Charchat-Fichman, 2015). As well, some also include processing speed and verbal fluency as components of EF (Faria et al., 2015). Planning involves the identification of sequential steps required to achieve a goal, as well as considering the most efficient approach (Chan et al., 2008; Faria et al., 2015). Working memory is the ability to temporarily hold and manipulate information in mind and is typically active during learning, reasoning, and problem-solving while necessary information is kept in mind (Diamond, 2013; Baddeley & Hitch, 1994). Inhibition is being able to control and/or override our attentional allocation, behaviour, thoughts, or emotions in a manner that is more appropriate or required in a given situation (Chan et al., 2008). Mental flexibility pertains to changing perspective on our way of thinking, especially when problem-solving, and often involves inhibiting previous patterns of thinking or responses (Diamond, 2013). EFs allow us to: (1) initiate and alternate between different tasks, (2) monitor and organize information based on objectives, and (3) inhibit stimuli or distractions to maintain focus on a Page 9 of 178 task/problem (Miyake, Friedman, Emerson, Witzki, Howerter et al., 2000). In daily living, executive processes give us the ability to engage in goal-oriented and productive behaviours such as prioritizing tasks, scheduling the day, and adjusting to unexpected changes in routine (Gazzaniga et al., 2014). Impairment in EFs have been shown to significantly affect the ability to carry out everyday activities at work, school, and home, as well as the ability to drive, or develop and maintain social relationships (Bell-McGinty, Podell, Franzen, Baird, & Williams, 2002; Boyle, Paul, Moser, & Cohen, 2004; Cahn-Weiner, Boyle, & Malloy, 2002; Cahn-Weiner, Ready, & Malloy, 2003; Jefferson, Paul, Ozonoff, & Cohen, 2006). While many frontal regions of the brain are involved in EFs (Stuss, 2011), contemporary technology (e.g., fMRI) has revealed that EFs are more than just a product of frontal lobe activity (see Alvarez & Emory, 2006 for meta-analytic review). Activities requiring EF engage a wide distribution of neural networks that primarily involve the prefrontal cortex, parietal cortex, basal ganglia, thalamus, and cerebellum (Collette, Van der Linden, Laureys, et al., 2005; Rabinovici, Stephens, & Possin, 2015). Moreover, this neural network is also predicated on the effective functioning of dopaminergic, cholinergic, noradrenergic, and serotonergic systems (Barnes, Dean, Nandam, et al., 2011; Robbins & Roberts, 2007) as well as white matter tracts (Kerchner, Racine, Hale, et al., 2012). As a complex network, the EF system is susceptible to conditions (e.g., neurological, psychiatric, medical) that affect its functioning, and contribute to impaired performance with daily activities (Conti, Sterr, Brucki, & Conforto, 2015). Given the susceptibility to impairment and association with success on daily tasks, it is important that neuropsychological test batteries include multiple measures of EF (Hachinski, Iadecola, Petersen, et al., 2006). To capture the multiple facets of EF, various neuropsychological tests proposed to target higher-order cognitive functions are available (Rabin et al., 2006; Rabin et Page 10 of 178 al., 2016). As noted above, the validity of scores from individual neuropsychological tests is not ‘perfect’, with a large amount of variance unaccounted for in predicting functional outcomes (see Chaytor, Schmitter-Edgecombe, & Burr, 2006). Findings for tests of EF have been mixed, with some studies demonstrating significant correlation between function and EF test scores (Burgess, Alderman, Evans, Emslie, & Wilson, 1998; Nys et al., 2005) and others showing weak or no correlation (Chaytor, Schmitter-Edgecombe, & Burr, 2006; Amieva, Phillips, & Della Sala, 2003; Bogod, Mateer, & MacDonald, 2003). The discrepencies across studies between test scores and real-world functioning may be, in part, related to behaviour during testing, such as use of strategies and approach to testing (Goldstein, 1996; Chaytor & Schmitter-Edgecombe, 2003). Qualitative data (e.g., patient behaviour) can have important implications on interpretation of scores as well as predicting function (Silver, 2000). For neuropsychology, a limitation of assessing EFs is that testing occurs in an isolated/controlled environment and is often structured by the examiner, which may mask underlying executive impairment (Bennett, 2001). Thus, methods that can help neuropsychologists quantify behaviour such as overt attention (eye movements) during tests of EF within a structured test environment may help to understand performance and improve inferences of daily functioning. 1.2.2 The Wisconsin Card Sorting Test (WCST) The WCST (Grant & Berg, 1948; Heaton, Chelune, Talley, Kay, & Curtiss, 1993) is the most commonly used test of EF by neuropsychologists (Rabin, Barr, & Burton, 2016), and the cWCST (Heaton & Psychological Assessment Resources, 2003) is the second most commonly used computerized neuropsychological test (Rabin, Spadaccini, Brodale, Grant, Elbulok-Charcape, & Barr, 2014). The WCST assesses abstract reasoning, mental flexibility, directing behaviour towards goal achievement, and disinhibition (Heaton et al., 1993). Four key cards are Page 11 of 178 placed consisting of one red triangle, two green stars, three yellow crosses, and four blue circles. A sorting deck of 128 cards is provided that depict figures of varying color (red, green, yellow, blue), shape (triangle, star, cross, and circle), and number (one to four). Each card is placed below the key card and feedback is provided after placement (i.e., “right” or “wrong”) with no further instruction throughout testing. After 10 consecutively correct responses the category is switched without indication or warning. Testing is discontinued once six categories are achieved or all 128 sorting cards have been placed. Multiple scores are obtained on the WCST to help elucidate performance, including total trials administered, number of correct and incorrect responses and number of categories achieved. Further detailed analysis of performance is obtained through scores on type of errors committed (perseverative versus non-perseverative), failure to maintain set, conceptual level responses, number of trials to achieve first category, and learning to learn. The number of categories completed is based on matching to one of three specific patterns (color, form, or number) that switches after 10 consecutively correct responses, to a maximum of six correct categories (each pattern completed twice). Perseverative errors are based on the continued use of an incorrect response despite feedback that the response is incorrect. Non-perseverative errors are based on random error commission, which may be contributed to trial-and-error or breaking from a correct response pattern. Loss of set is based on switching response pattern to an incorrect response after at least five consecutively correct responses are made. Conceptual level responses are based on making three or more consecutively correct responses (including ambiguous correct responses). The WCST has discrete areas of interest related to task completion (i.e., the key cards, the previously placed cards, and the trial stimulus card). Testing involves the exploration of the information (e.g., option of four key cards and stimulus card), manipulation of the information to Page 12 of 178 form hypotheses (e.g., matching category), integration of feedback after placement (i.e., right or wrong), integrating previously placed cards (i.e., sorting piles), and maintaining hypothesized solution while inhibiting others (e.g., adhering to matching category). Successful completion of the WCST requires multiple facets of EF: hypothesis generation and testing to determine how cards match, working memory to maintain responses to target category, inhibition to refrain from making responses that are no longer correct, and mental flexibility to adapt to change of target category. The WCST is purported to be a measure of EF and frontal lobe (dys)function (Milner, 1963; Demakis, 2003). However, performance is also susceptible to lesions (e.g., stroke) in other, non-frontal lobe, brain regions (Buchsbaum, Greer, Chang, & Berman, 2005; Axelrod, Goldman, Heaton, et al., 1996; Su, Wuang, Chang, & Kwan, 2006). Moreover, Jodzio and Biechowska (2010) reported that scores from the WCST had poor accuracy for detecting executive dysfunction in stroke survivors and that a multicomponent approach to evaluating EF is needed during assessment. Further, results of studies investigating the ecological validity have also been mixed. Some have shown that WCST test scores are significant predictors of functional status at time of discharge from hospital after stroke (Greve, Bianchini, Hartley, & Adams, 1999), level of independence in the community following hospitalization in psychiatric populations (Heinrichs, 1990), occupational status after recovery from brain injury (Kibby, Schmitter-Edgecombe, & Jong, 1998), and real-world goal-directed behaviour (Lawrence, Houghton, Douglas, et al., 2004). In contrast, others have shown test scores to be poor predictors of daily function (Mccue & Aitken, 1998; Manchester, Priestley, & Jackson, 2004) and suggest poor verisimilitude (Burgess et al., 1998). Mixed results may stem from focusing on outcome scores, and not on how test scores are obtained (e.g., response processes such as strategies employed, information attended to, etc.). As Page 13 of 178 noted in section 1.1.3, observation and analysis of test approach can provide rich information to help identify difficulties on testing, improve diagnostic accuracy, and help delineate cognitive impairment (Au et al., 2017; Lamar et al., 2010; Kaplan, 1988). For example, Somsen (2007) found that a trial-by-trial analysis of sorting during the computerized WCST (cWCST), that included inspection- and reflection-time before card placement, predicted number of categories achieved. Along these lines, computerized versions of tests can provide quantitative details about test behaviour unavailable on paper-pencil versions. Further, how and where attention is directed has been shown to predict performance during problem solving (Rouinfar, Agra, Larson, Rebello, & Loschky, 2014) and is able to differentiate between low and high performers (Hu, Wu, & Gu, 2017). However, little is known about overt attention during completion of the WCST and further empirical investigation may help inform performance on testing and provide a potential area for further examination and intervention in clinical populations. 1.3 Advancing Neuropsychological Testing Previous advancements of neuropsychological assessment and methodology have identified factors which contribute to variability in results. Casaletto and Heaton (2017) highlight previous advances in neuropsychological assessments that have focused on the quantitative methodology such as expanding normative databases, identifying influential factors related to test performance (e.g., demographics), standardizing testing for comparison across settings, comparison of fixed and flexible test batteries, and integration of the BPA (see section 1.1.3) to elucidate errors/responses in testing. For example, Heaton and colleagues (1991) found a combination of demographic factors (age, sex, education) was a large contributor to variance in test scores. Moreover, the Mayo Older Americans Normative Studies (MOANS) aimed to improve the clinical utility of common neuropsychological tests for older adults by establishing and Page 14 of 178 continuing to update a normative database on neurologically healthy elderly populations (Ivnik, Malec, Smith, et al., 1992). Together, advances in traditional testing methods have been critical in continuing to provide valid and reliable assessments of brain-behaviour function across diverse populations (Parsons & Duffield, 2019). Despite previous advances, over the past decade, it has been highlighted that the integration of contemporary technology into neuropsychology has been slow compared to other fields (Jagaroo, 2009; Parsey & Schmitter-Edgecombe, 2013; Parsons, 2016; Miller & Barr, 2017). This hesitation may limit advances to the theoretical understanding of brain-behaviour relationships as well as clinical practice in neuropsychology (Dodrill, 1997; Jagaroo, 2009). Parsons and Duffield (2019) argue that neuropsychology has begun falling behind current initiatives set by the National Institute of Health on advancing scientific developments in behavioural and social sciences (see Collins & Riley, 2016). Indeed, some of the most commonly used neuropsychological tests are slightly revised versions of the traditional paper-and-pencil tests developed nearly 80 years ago (Parsons, 2016). While prior quantitative advances have been critical in strengthening existing assessment approaches (Casaletto & Heaton, 2017), neuropsychology may benefit from integrating current technology from neuroscience and cognitive psychology to continue leading the translation into clinical practice (Bilder, 2011; Parsons, 2016; Miller & Barr, 2017; Casaletto & Heaton, 2017; Kessels, 2019). However, implementing new technology into neuropsychological assessments raises several potential concerns (Miller & Barr, 2017). These include: the perceived limitations of psychometric developments in specific clinical populations, lack of robust evidence to demonstrate equivalence with traditional tests, unfamiliarity with procedures and administration, reduction of behavioural observations with automated tests not requiring an examiner (e.g., psychometrist), large volume of data generated, lack of training with computerized Page 15 of 178 testing, and convincing clinicians to transition away from traditional paper-pencil testing methods (Gualtieri & Johnson, 2006; Wild et al., 2008; Lezak et al., 2012; Gates & Kochan, 2015). Nonetheless, there has been growing interest in integrating computer and digital technology to help evolve current neuropsychological assessments (Jagaroo, 2009; Bilder, 2011; Miller & Barr, 2017; Casaletto & Heaton, 2017; Parsons & Duffield, 2019; Kessels, 2019). Advances in neuroscientific technology have benefitted other health disciplines in the assessment and treatment of various clinical populations. For example, diffusion tensor imaging (DTI) has provided further understanding of neural network function, dysfunction, and communication (Bartzokis, Lu, Heydari, et al., 2012) by providing a noninvasive approach to measuring white matter pathways in the brain (Mueller, Lim, Hemmy, & Camchong, 2015). Moreover, DTI has been applied in the research of clinical populations such as schizophrenia, traumatic brain injury, multiple sclerosis, autism, and aging, as well as neurosurgical planning (O’Donnell & Westin, 2011). Bilder (2011) highlights multiple factors that are driving the efforts to revolutionize approaches to neuropsychological testing and investigation of brain-behaviour relationships. These influential factors include: developments in neuroimaging for visualizing brain structure and function; genotyping (Human Genome Project) for identifying individual differences and variation in behaviour; information science and growth of the internet for rapid and wide dissemination of research and knowledge, collaboration about brain and behaviour, and providing educational and clinical services not previously available; and increased emphasis on providing evidence-based approaches in healthcare services. Bilder (2011) suggests that assessment innovation is a workable option; for example, computer-based testing coincides with the rapid deployment and sharing of knowledge and information and allows for more adaptive forms of Page 16 of 178 testing and improved precision to highlight individual differences. As well, Miller and Barr (2017) draw parallels between our engagement with technology in our daily lives (e.g., computer screens and smart phones) and proposed technological methods for neuropsychological testing. Taken together, there has been increasing interest of implementing technology into neuropsychological assessments, however, as a discipline, neuropsychology has been relatively slow in implementing such changes compared to other disciplines, in favor of traditional paper-pencil tests. Rather than supplanting traditional paper-pencil tests, integrating technology into existing approaches to extract new data points may be (1) more palatable to clinicians and (2) helpful to enrich obtained data from traditional cognitive tests to improve ecological validity (Casaletto & Heaton, 2017; Miller & Barr, 2017; Marcotte, Scott, Kamat, & Heaton, 2010). 1.3.1 Computerized Neuropsychological Assessment. While not currently the standard approach, it is predicted that utilization of computerized neuropsychological assessment devices (CNADs) for scoring, administration, data storage, and analysis will continue to gain momentum in research and clinical assessments (Lezak et al, 2012). The American Academy of Clinical Neuropsychology and National Academy of Neuropsychology has highlighted standards and guidelines for the development and use of CNADs in practice (see Bauer, Iverson, Cernich, Binder, Ruff et al., 2012). Numerous benefits have been reported, regarding the use of CNADS. The power of computer processing for detailed analysis of test performance, via passive data collection on behaviour during testing, may help elucidate theories and concepts about brain-behaviour relationships and help infer function (Parsons, 2016). Used appropriately, computers “can outperform a human examiner in precision and rapid implementation of adaptive algorithms” (Bilder, 2011, p. 6) and provide detailed trial analysis and task manipulation for increased Page 17 of 178 sensitivity and specificity of neural function and dysfunction. Moreover, use of computers for administration and analysis could improve test selection and interpretation, and ultimately supplement and enhance clinical decision-making (Bilder, 2011, p. 7; Au et al., 2017). Other reported benefits of incorporating computers into neuropsychological assessments include: standardized administration and quick scoring against prior testing and age-related normative data, reduction of human error, recording accuracy and response speed, assessing multiple cognitive functions with algorithms to enhance information gathering, and reducing cost of materials (Bilder, 2011). In contrast, the use of computers for cognitive testing also have limitations and cautions that should be addressed, or users should be cognizant of, before the above-mentioned benefits can be realized. Indeed, the American Academy of Clinical Neuropsychology and National Academy of Neuropsychology have also identified barriers and limitations to the use of computerized tests (Bauer et al., 2012), which may readily apply to contemporary neuroscience technology (e.g., devices) used in the test setting: First, consideration must be given to the intended population, such as older adults who have less experience and need more time or assistance before feeling comfortable with the computer test modality. Not doing so may result in uncertainty of whether errors are a result of inexperience or true cognitive difficulties. Second, there is limited empirical evidence available regarding the potential interaction of physical, psychiatric, and/or neurological conditions with computerized testing that may limit the interpretation of results (Bauer et al., 2012). Similar to use of all cognitive tests, the clinician must be competent and skilled with the test/device as well as the data for effective and accurate use. Third, the use of computerized tests may not include the opportunity to provide thorough observation of behaviour compared to examiner-administered tests, therefore, missing clincically useful information (e.g., task Page 18 of 178 engagement, reaction to testing). Fourth, additional consideration must be given to the software/hardware to ensure that the device operates with the same specifications that were used during development of the normative database for accurate comparison. Moreover, an understanding of how electronic data is collected, stored, and transmitted are essential to ensure security and protection from threats to data integrity and privacy. Taken together, CNADs offer many potential advantages and unique opportunities for novel data collection. However, the limitations and challenges outlined above must also be considered, and it is important that CNADs be held to the same standard in their development. In considering transition to computerized testing from paper-pencil tests, it is important to point out that individual and cohort differences may require attention in implementing new methods of administration. For example, performance differences between manual and computerized versions of testing have been shown to be affected by computer-related anxiety, experience, and attitude (McDonald, 2002), as well as individual differences related to social interaction or verbal comprehension (Ozonoff, 1995). Further, Arrieux and colleagues (2017) report that while commercially available CNADs have demonstrated potential benefit in some populations (e.g., concussed athletes and military personnel), further research is required to investigate the psychometric properties. However, the authors suggest development of psychometric properties by comparing CNADs to traditional neuropsychological tests may be a flawed process, comparing ‘apples to oranges’, and CNADs should be viewed as their own unique assessment tools. In cases where traditional tests have been adapted to computer administration/scoring, some have shown to have at least partial psychometric equivalence to their traditional counterparts, fewer scoring errors, and fewer false-positives and negatives in diagnosing (Tien, Spevack, Jones, et al., 1996; Steinmetz, Brunner, Loarer, & Houssemand, 2010; Page 19 of 178 Rabin et al., 2014). The numerous benefits and challenges of transitioning to more computer and digital assessment have been outlined previously, as well as methods to address the proposed challenges (see Parsons et al., 2018). Relevant to the current dissertation, the WCST has been adapted to a computerized version (cWCST) (Heaton & PAR staff, 1993), and is one of the top employed computerized tests used by neuropsychologists (Rabin et al., 2006; Rabin et al., 2016). The transition to computerized administration has advantages, such as reduction of human error/bias in administration and scoring via automatic recording and analyzing of responses (Feldstein, Keller, Portman, Durham, Klebe, & Davis, 1999). However, studies are mixed in regard to its psychometric equivalence to the traditional paper-pencil version. Some have suggested equivalence between the two versions (Fortuny & Heaton, 1996; Tien et al., 1996) while others report varying levels of discrepancy in the distribution of scores between the two versions (Feldstein et al., 1999; Steinmetz et al., 2010). Mixed findings suggest that the psychometrics between the two versions are questionable and not reliably interchangeable. Many studies have examined the validity of scores from the cWCST using structural and functional neuroimaging during test completion (see Barcelo, 2001 for review). While some studies have argued that the scores from the traditional WCST have poor to adequate ecological validity (Chiu, Wu, Hung, & Tseng, 2017; Chaytor et al., 2006), relatively little investigation has been conducted with the cWCST. Nevertheless, the tests continue to be frequently employed in clinical test settings (Rabin et al., 2014; Rabin et al., 2016). Thus, it is prudent that scores from the cWCST, in conjunction with novel methods, be investigated to ascertain the validity of the data. Regarding prevalence of use, according to Rabin et al. (2014), survey results of Canadian and American clinical neuropsychologists showed nearly half (45.5%) of the 495 respondents Page 20 of 178 reported never using CNADs, 14.1% reported rarely, 19.6% reported sometimes, and 18.2% reported often, and 2.6% reported always. The authors found that younger age and fewer years of practice increased likelihood of utilizing CNADs in neuropsychology (see also Parsons, 2016). Therefore, while most assessments are still conducted using traditional paper-pencil neuropsychological tests (Rabin et al., 2016), there is greater potential for future generations of neuropsychologists to be more open to adopting CNADs into practice. Given the increasing interest and push for neuropsychology to embrace more contemporary methods of assessment, careful examination during the integration of new technology into practice is warranted (Parsey & Schmitter-Edgecombe, 2013). In the Framingham Heart Study (noted above), advanced technological instruments have been a major focus in expanding the BPA during neuropsychological assessments to provide incremental value, monitor incidence and process of neurodegenerative disease, and inform potential interventions to preserve independence/daily functioning (Au et al., 2017). 1.4 Eye Tracking What the eye is focused on is a direct indicator of overt attention and what is being processed by the mind (Just & Carpenter, 1980; Armstrong & Olatunji, 2009). Eye tracking is used to continuously measure the point of gaze (fixation) or movement (saccade) of the eyes (Poole & Ball, 2006). Fixations and saccades are quantified using eye tracking equipment and software to obtain information about visual attention and insights to cognitive processes (Duchowski, 2002; Lai et al., 2013). Computer software and eye tracking technology has become increasingly sophisticated and readily available to consumers for providing more options to record and analyze eye movements across settings. Page 21 of 178 Contemporary eye tracking software provides algorithms for extracting numerous data points for analysis (Salvucci & Goldberg, 2000; Reingold, 2014). Both fixations and saccades can be broken down into more specific functional units for investigation (Schall & Bergstrom, 2014). Fixation data can be subdivided and include total and average dwell times as well as total count and provide an indicator of information processing, task complexity, and interest/engagement. Likewise, saccadic data can include the sequence, count, amplitude, velocity, and duration of eye movements. Notably, saccades also include smooth pursuit, vestibulo-ocular, and vergence movements (Purves, Augustine, Fitzpatrick et al., 2001). Compared to saccades, smooth pursuit movements are relatively slow and often intentional movements of the eyes to keep a target/stimulus fixated on the fovea. Vergence movements help to align the fovea with targets located at varying distances. Vestibulo-ocular movements help to maintain fixation on a target by compensating for head movements. Eye tracking software often provides a variety of characteristics for fixations and saccades, as noted above, for use in experimental paradigms (Purves et al., 2001). Voluntary saccades and fixations are under cognitive control and are responsive to ongoing processing of visual stimuli (Rayner & Reingold, 2015). Saccades are rapid eye movements that orient the fovea to visual information for detailed processing. Visual attention and processing new information require inhibition of saccades to maintain focus (fixations) on a desired object and process information (Kulke, Atkinson, & Braddick, 2016). During fixations, gaze remains relatively stable and saccades must be inhibited to acquire information related to task demands or goals, and determine subsequent action (Nuthmann & Malcolm, 2016; Kulke et al., 2016; Just & Carpenter, 1980). Moreover, this fixation time is an indicator of top-down processing (Lou, Cai, Liu, & Li, 2019). Fixations on, and voluntary saccades towards, information relevant for problem Page 22 of 178 solving have been shown to predict correct responses (Grant & Spivey, 2003) as well as distinguish between stages of problem solving (Hodgson et al., 2000). Salvucci and Goldberg (2000) identified two reasons for utilizing fixations, in addition to saccades, for understanding cognitive processing: (1) little to no processing of information occurs during saccades, and (2) the small microsaccades or ‘flicks’ that occur during fixation often have little bearing on higher cognitive processing, such as analyzing information to solve novel problems. Moreover, the authors highlight that areas of interest (AOIs) are one of the most common and precise methods of quantifying fixations. AOIs represent important visual targets on a presentation screen and facilitate data interpretation (Hessels, Kemner, van den Boomen, & Hooge, 2016). AOIs are used the same way a region of interest (ROI) is used during noninvasive MRI scans in studying brain connectivity and function (Mueller et al., 2015). An ROI is applied over a tract or structure of interest in the brain, prior to diffusion data being processed by the scanner and summary data are computed across subjects and available for statistical analysis. In a similar manner, AOIs are applied to a video or across test trials, and after task completion, eye tracking data can be compiled and analyzed. Most commercially available systems measure eye movements using a corneal-reflection/pupil-centre method (Goldberg & Wichansky, 2003). Using this method, an infrared camera mounted to a computer with image processing software locates and identifies the eye for tracking. An LED light is directed into the eye to create a strong reflection to facilitate tracking. Light enters the retina and is reflected back to create a “bright pupil” effect (p. 11). Computer software identifies and compares the centre of the pupil with the corneal reflection to create a “point-of-regard” (i.e. where the eye is looking on the screen; p. 17). A calibration process is conducted to tailor the software calculations of eye movements to each individual. Calibration involves following a dot along a grid-pattern (9 to 13 points) on the screen and holding the gaze Page 23 of 178 until the system obtains an accurate point-of-regard for each location. The calibration process ensures the accuracy of capturing eye movements (saccades and fixations) at all points on the screen across individuals. Remote eye tracking models, using infrared light, have become increasingly popular, partly because they do not require a head restraint and allow for some natural head movement during recording (Niehorster, Cornelissen, Holmqvist, Hooge, & Hessels, 2018). The advancement of eye tracking hardware and software has been associated with an exponential increase in number of published peer-reviewed research studies over time (Reingold, 2014). An appealing aspect of eye tracking during cognitive testing is that it can examine eye movements with minimal intrusion and interruption of standardized administration (Mele & Federici, 2012; Richardson & Johnson, 2008). Given the standardized nature of test administration during neuropsychological assessments, it is important that the integration of new tools is seamless and does not interfere with the integrity of test procedures. 1.4.1 Eye Tracking and Cognitive Load Cognitive load is the amount of mental effort required by working memory during a cognitive task such as problem solving (Sweller, 1988; Sweller & Van Merrienboer, 1998). Cognitive Load Theory delineates cognitive load into three types: intrinsic load which is the task at hand and based upon expertise with the material and how well it fits within a previous schema (e.g., determining a matching category with cards); extraneous load which is imposed by the format and how information is presented, often not contributing to learning and acquisition (e.g., presented on a computer screen); and germane load which is effort required to solve the problem (Zagermann, Pfeil, & Reiterer, 2016). When cognitive load increases beyond the capacity of attention and working memory, performance declines (Sweller, 1988). Cognitive load increases Page 24 of 178 with greater demands placed on attention and working memory, such as dual-tasks and initial complexity of the task (Sweller, 1988). Tasks that are automatic or mastered and require little effort/attention have little impact on cognitive load. Matsumoto and colleagues (2011) demonstrated that greater mastery of a problem set results in more goal-oriented “top-down” processing of information compared to “bottom-up processing” by those who are less skilled (p. 6); that is, eye movements are being directed to information more relevant at producing a correct solution. One of the most common approaches to manipulating cognitive load is dual-tasks that require engagement in two activities simultaneously or having to switch back and forth between tasks (Lavie, 2010). Speed of completing daily activities has also been shown to decrease for those engaged in dual-tasks compared to single-tasks (Kizony, Levin, Hughey, Perez, & Fung, 2010). As cognitive load increases, distinct eye movement patterns become apparent during problem-solving (Lu Pinke, 2014). Increasing cognitive load has been shown to increase variability of fixation and saccades between relevant and irrelevant stimuli related to task performance (Lu Pinke, 2014; Heuer & Hallowell, 2015; Zagermann et al., 2016). With greater cognitive load (e.g., increasing number of distractors) saccades tend to increase toward irrelevant stimuli during problem solving (Tokuda, 2008), for the number of fixations to increase specifically on previously viewed stimuli (Rayner & Pollatsek, 1981), and for the duration of fixations to increase (Vlaskamp & Hooge, 2006). The observed increase in number of fixations on target AOIs during problem solving is believed to reduce the cognitive load on working memory to help free up mental resources and facilitate problem solving (Ballard, Hayhoe, & Pelz, 1995). 1.4.2 Eye tracking and Cognitive Function Eye tracking has become a popular method for observing visual and cognitive processes in the study of behaviour (Salvucci & Goldberg, 2000; Richardson & Spivey, 2004; Thomas & Page 25 of 178 Lleras, 2007). Eye tracking studies using saccades and fixations have been a basis of major advancements in our understanding and development of cognitive functions such as language comprehension, learning and memory, mental imagery, problem-solving, and decision making (Rayner, 1998; Richardson & Spivey, 2004; Thomas & Lleras, 2007; Karatekin, 2007). During problem-solving, correct responses are made when fixations and saccades are directed towards information relevant to deductive reasoning (Grant & Spivey, 2003). Fixation frequency is an indicator of a target’s perceived importance; and fixation duration can reflect a target’s perceived difficulty of processing (Jacob & Karn, 2003). For example, Hodgson et al (2000) used a modified version of the Tower of London (a neuropsychological test of EF/planning) with a sample of healthy adults to examine eye movements on pictures of trials that displayed a “Goalspace” (target configuration of balls) and a “Workspace” (picture of balls to be sorted; p. 1-2). They found that fixations on task irrelevant information or previous trials predicted increased error frequency. Longer fixation times were observed on the Goalspace at the start and end of each trial, and on the Workspace during the middle of each trial. Task difficulty was associated with longer fixations on the Workspace. Moreover, patterns of fixations and saccades have been shown to distinguish between high- and low-skill level during problem-solving (Hu, Wu, & Gu, 2017). Similar studies have also shown sequence of fixations predict performance on tasks of visual memory (Bochynska & Laeng, 2015). Others have found fixations and saccades estimate skill level on reading tests (Bax, 2013), differentiate cognitive styles during processing and decision-making (Nisiforou & Laghos, 2013), and learning across different modalities (e.g., text, graphic, video, etc.; Van Gog & Scheiter, 2009; Van Gog & Schieter, 2010). Across studies, eye tracking has been shown to predict many factors involved and needed for successful performance on neuropsychological tests of EF, like the cWCST (e.g., working memory, decision-making, Page 26 of 178 learning, and problem solving). In keeping with a quantitative process approach (Poreh, 2006) and the Boston Process Approach (Kaplan, 1988), eye tracking provides a method to capture and analyze eye movements that is not possible by an examiner during traditional test administration. Chaytor and colleagues (2006) found that tests of EF alone account for only 18-20% of the variance in complex abilities of daily life. The authors found that inclusion of “extra-test” variables to the regression model, such as strategy use and cognitive demand during testing, accounted for 51% of the variance (p. 225). This qualitative information is important because similar outcome scores can be obtained despite differing cognitive impairments (Kaplan, 1988; Lamar et al., 2010). For example, low scores on the cWCST may be due to difficulty focusing, set maintenance, response perseveration, or concept attainment. Previous studies suggest total number of correct trials and number of categories completed are the strongest predictors of independent function after stroke (Chiu et al., 2017); fewer trial errors predict improvements in functional mobility in the elderly (Gothe, Fanning, Awick, et al., 2014); also, perseverative errors predict functional dependence in patients with traumatic brain injury (TBI), 6-months after discharge from rehabilitation (Hanks, Rapport, Millis, & Deshpande, 1999). Thus, scores tapping differing facets of EF on the cWCST appear to predict activities in daily living. With a process approach in mind, the integration of eye tracking may augment cWCST test taking by quantifying how/what information is being attended to. Recording and analysis of fixations and saccades offers a way to study how visual attention is associated with test approach, learning across trials, and overall performance (e.g., total categories completed, error types). 1.4.3 Eye Tracking for Clinical Assessment Eye movements have become increasingly popular in clinical neuroscience for studying normal and abnormal brain function (Leigh & Kennard, 2004). Diagnostically, eye tracking data Page 27 of 178 has been shown to provide powerful discriminate validity between controls and clinical populations. For example, eye tracking data had nearly perfect (98.3%) accuracy differentiating schizophrenia from healthy controls, with differences in fixation being the single best discriminator (Benson, Beedie, Shephard, Giegling, Rujescu, & St. Clair, 2012). Initiating and inhibiting eye movements have been shown to differentiate patients with Parkinson’s Disease from healthy controls, as well as provide an indicator for severity (Amador, Hood, Schiess, Izor, & Sereno, 2006). Eye tracking has previously demonstrated utility in detecting visual deficits during reading, visual search, and scene perception in patients with TBI (Rayner, 1998). One study demonstrated that eye tracking measures of patients with mild TBI, 1-week post-injury, were able to predict those who developed post-concussive syndrome (PCS) from those who did not with 100% sensitivity and specificity, surpassing neuropsychological test scores (Heitger, Jones, & Anderson, 2008). Overall, results suggest eye tracking may serve as a valuable marker for diagnosing, but less is known about its clinical utility in predicting function and how eye tracking may be beneficial in rehabilitation and assessment of daily function. Despite the growing interest in eye tracking technology, validity studies investigating eye tracking with neuropsychological tests are sparse. Some studies have demonstrated a correlation between abnormal eye movements and impaired performance on the Digit Symbol Test and Controlled Oral Word Association Test, as well as scores on the MMSE (Friedman, John, Jesberger, Choy, & Meltzer, 1995; Yang, Wang, Su, Xiao, & Kapoula, 2013). However, results were based on separate tasks; eye tracking was not used in real-time on the cognitive test. Preliminary studies have shown promising integration of eye tracking for patients with limited physical mobility (e.g., patients with amyotrophic lateral sclerosis) with the Trail Making Test (Hicks, Sharma, Khan et al., 2013) and a small cognitive test battery (Poletti, Carelli, Solca et al., Page 28 of 178 2017) by using fixations to allow patients to select stimuli during testing. Otherwise, and to the best of our knowledge, there is limited empirical evidence demonstrating the validity of eye tracking data during cognitive testing of EF in real-time. 1.4.4 Eye Tracking and Daily Activities/Functioning How gaze and fixation relate to a variety of daily tasks has received increasing interest over the years (Hayhoe & Ballard, 2005; Land, 2009). Eye tracking has been shown to identify subtle differences between searching for an item and making a decision among items at a grocery store (Gidlof, Wallin, Dewhurst, & Holmqvist, 2013), problems during reading tasks (Smith, Glen, Monter, & Crabb, 2014), and sequence of attention to visual cues while walking and interacting with objects and people (Foulsham, 2015). Three systems are thought to allow action of daily tasks: the gaze system locates and fixates on task-relevant objects; the motor system controls the limbs to engage; and the visual system supplies information. These systems provide the sequential actions to complete the task (Land, 2009). During daily tasks that require multi-steps (e.g., tea making), gaze will move onto the next relevant object just prior (approximately 1-sec) to manipulation of the object (Land, 2009). This suggests that nearing the end of the previous step, the gaze system will initiate to track the next piece of information (e.g., location of object) relevant for task completion. Given the tight and complex neural networks in these systems, it is easy to see why brain injury often compromises functional abilities. Damage may include systems that involve saccades and fixations required for learning and mastering tasks or skills. Greater familiarity and mastery of a task is associated with gaze and fixation toward a target location rather than on the object being manipulated (Sailer, Flanagan, & Johansson, 2005). Previous eye tracking research comparing amateur and professional athletes has consistently shown discrepancies in their fixations and gaze paths (Richardson & Spivey, 2004). For example, skilled golfers are less likely Page 29 of 178 to look at their club and more likely to look at the ball and hole (Vickers, 1992), more advanced cricket players are quicker at estimating location of the ball after a pitch by fixating on release of the ball and anticipated location the ball will bounce (Land & McLeod, 2000). Further, findings by Sailer et al. (2005) suggest that eye movements provide information related to task engagement, feedback related to adequate completion of a task, transition into subsequent steps, and indicate level of competency with the task or skill. In clinical settings, Seligman and Giovannetti (2015) suggest that eye tracking holds promise in detecting subtle functional decline in patients with mild cognitive impairment and differentiating this subgroup from more severe impairment (e.g., Alzheimer’s disease) and healthy controls. Given that neuropsychologists are routinely asked to comment on daily function or rehabilitation planning for patients, eye tracking can provide valuable information that is otherwise unavailable through observation or self-report alone. Moreover, this may provide a powerful method to inform intervention or detect subtle causes of functional impairment in daily living. 1.4.5 Eye Tracking Facilitating Function Some studies have also shown that performance on cognitive tests may be influenced by directing fixations and saccades. Johansson and Johansson (2014) demonstrated that directing eye movements affected performance during recall on a visual memory task. Participants viewed a four-quadrant panel of 24 objects and were later asked to recall details regarding spatial relationships between objects or details of an object. Gaze was manipulated to free viewing, central fixation on the screen, or fixation to the quadrant containing, or not containing, the target object. Fixation on the quadrant that was congruent with the previously viewed object resulted in more accurate recall of its spatial location and details. During problem solving, directing saccades and fixations to relevant details while examining a picture-problem has been shown to improve Page 30 of 178 reasoning and successful problem resolution (see Grant & Spivey, 2003). Further, directing eye movements in patterns related to the solution improved visual problem solving compared to directing eye movements in patterns unrelated to the solution (Thomas & Lleras, 2007). In higher-stakes settings, eye tracking is also being used as a tool to augment training of medical professionals in diagnostic accuracy and treatment decision-making (Brunye, Drew, Weaver, & Elmore, 2019). Specifically, the authors highlight how eye tracking can be used to facilitate training from novice to expert, monitor and maintain competencies, detect error commission and classification, and inform remediation strategies in training. Together, these studies provide some evidence for the use of eye tracking as a guide for behaviour and task performance. However, despite growing popularity, little investigation has been conducted in the assessment and rehabilitation of inpatients recovering from a stroke. Thus, it is not clear if similar approaches can be used during neuropsychological testing to augment results, infer function, and predict outcomes in this population and setting. 1.5 Stroke Stroke, also known as a cerebrovascular accident (CVA), is an interruption of blood flow to the brain. According to the World Health Organization, stroke is defined as, “rapidly developing signs of focal (at times global) disturbance of cerebral function lasting more than 24 hours or leading to death with no apparent cause other than that of vascular origin” (Aho et al., 1980; p. 114). Approximately 15 million people around the world are hospitalized for stroke-related symptoms annually (Mackay, Mensah, Mendis, & Greenlund, 2004). In North America, approximately 46,500 Canadians are hospitalized and diagnosed with stroke-related symptoms each year, and approximately 315,000 Canadians are living with the detrimental effects of stroke (Heart and Stroke Foundation, 2014). In the United States, more than 795,000 Americans are Page 31 of 178 hospitalized and diagnosed with stroke-related symptoms each year (Center for Disease Control and Prevention, 2015). Approximately 610,000 are first-time stroke patients, and 185,000 have previously suffered a stroke. It is estimated that each year approximately $3.6 billion (CDN) and $34 billion (US) is spent on stroke-related physician services, hospital costs, lost wages and decreased work productivity (Heart and Stroke Foundation, 2014; Mozzaffarian et al., 2015). Multiple factors increase the risk and prevalence of stroke, such as hypertension, diabetes, hyperlipidemia, smoking, coronary heart disease, and obesity (Schneider, Pancioli, Khoury, et al., 2003; Kissela, Khoury, Alwell, et al., 2012; George, Tong, Kuklina, & Labarthe, 2011). Moreover, some of these conditions (e.g., hypertension, hyperlipidemia, and coronary heart disease) become more prevalent with age and therefore increase the risk of stroke as well (Milionis, Liberopoulos, Goudevenos, et al., 2005). The American Heart Association advises that the incidence of stroke is predicted to rise with the aging “Baby-Boomer” population, and international studies predict that stroke rates among younger adults (24-64 years) will double in the next 15 years (Heart and Stroke Foundation, 2014; p. 11). With the increasing number of strokes occurring each year, the long-term forecasted rise in numbers, and the increased burden on healthcare systems, effort to integrate contemporary methods (e.g., computers and eye tracking) and refine our understanding of cognitive deficits, dysfunction, and what can be done to minimize their impact on independence and optimize recovery, is warranted. 1.5.1 Consequences of Stroke The consequences of stroke can be multifaceted and are often influenced by the brain region affected (Teasell, Bayona, & Heitzner, 2015; Tatemichi, Desmond, Stern, Paik, Sano et al., 1994; Barker-Collo, Starkey, Carlene, Lawes, Feigin et al., 2012). Thanks to advances in modern technology and public education, timely interventions have resulted in more people surviving Page 32 of 178 strokes. Consequently, up to 66% of stroke survivors are living with long-term deficits (National Institute of Health [NIH], 2014). In Canada, stroke is among the leading causes of disability requiring rehabilitation (Canadian Institute for Health, 2009). Deficits from stroke include paralysis or movement disorders, sensory disturbances such as pain, problems with receptive or expressive language, problems with thinking and memory, and emotional disturbance (NIH, 2014; American Heart Association [AHA], 2015). The prevalence cognitive impairment has been found in up to 80% of poststroke survivors. (Sun, Tan, & Yu, 2014). The American Heart and Stroke Associations report that approximately 2/3 of patients present with cognitive impairment after a stroke (Gorelick et al., 2011). Stroke can affect various cognitive domains such as attention, memory, visuospatial ability, language and comprehension, self-awareness, planning and organizing, and problem-solving (Barker-Collo, Feigin, 2006). While physical impairments are common and often the target of most rehabilitation, it is well established that cognitive function plays a significant role in functional outcomes (Hochstenbach, Anderson, Limbeek, & Mulder, 2001; Chlopan, Hagen, & Russell, 1990; Paolucci, Antonucci, Pratesi, Traballesi, Lubich, & Grasso, 1998; Nys et al., 2005a; Barker-Collo & Feigin, 2006; Feigin, Parag, Lawes, & Senior, 2010). Cognitive impairment may even limit rehabilitation of motor recovery, further reducing functional outcomes (Whyte, Skidmore, Aizenstein, Ricker, & Butters, 2011). Moreover, cognitive impairment has been shown to negatively affect functional outcome independent of physical function (Tatemichi et al., 1994; Heruti, Lusky, Dankner, Ring, Dolgopiat, et al., 2002). Together, cognitive deficits are detrimental to basic activities of daily living (e.g., self-care) and instrumental activities of daily living (e.g., employment), making it difficult for patients to reintegrate back into the home and community (Wagle, Farner, Flekkoy, Wyller, Sandvik et al., 2011). Cognition appears to have a strong influence on recovery and outcome of stroke patients; Page 33 of 178 however, there is still much to be gleaned on the contribution of specific cognitive domains and function (Kessels, 2019; Parsons, 2016; Miller & Barr, 2017). 1.5.2 Assessment of Cognitive Function After Stroke World Health Organization guidelines for management after stroke recommend thorough assessment of cognitive function, because of the heterogeneity of cognitive sequelae and recovery (Miller, Murray, Richards et al., 2010). Understanding cognitive impairment is a priority in addressing life after stroke, according to surveyed stroke survivors, caregivers, and healthcare professionals (Pollock, George, Fenton, & Firkins, 2012). However, research pertaining to cognitive function after stroke is limited (Lees et al., 2012) and may translate into deficits not being assessed or treated, and therefore, adversely affecting functional outcomes (Morris, 2009). The frequent employment of cognitive screeners, while useful, may be too insensitive for more complex cognitive deficits, like executive dysfunction (Pendlebury & Rothwell, 2009) and potentially delayed the advancement of understanding the relationship between EF and functional outcomes (Lees et al., 2012; Chan, Khan, Oliver, Gill, Werring et al., 2014; Pendlebury & Rothwell, 2009). Common screening tools include the Mini Mental State Exam (MMSE; Folstein, Folstein, & McHugh, 1975) and the Montreal Cognitive Assessment (MoCA; Nasreddine, Phillips, Bedirian, Charbonneau, Whitehead et al., 2005), that are often employed in research and clinical settings as cost- and time-effective methods of assessment (Abdul-sattar, Godab, 2013; Patel, Coshall, Rudd, & Wolfe, 2002; Scott & Mayo, 2018; Lees et al., 2012). However, their efficacy has been challenged, especially in comparison to domain-specific neuropsychological tests, viewed as the gold standard (Nys et al., 2005b; Godefroy, Fickl, Roussel, Auribault, Bugnicourt et al., 2011; Chan, et al., 2014). One study demonstrated that the prevalence of post-stroke cognitive impairment ranged from 24% to 39% based on MMSE scores and increased to 96% Page 34 of 178 according to comprehensive neuropsychological testing (Gutierrez, Savborg, Pahlman, et al., 2011). Cullen and colleagues (2007) warn about the over reliance on screeners, which provide limited assessment of cognitive domains, and for certain areas (i.e., EF), fail to examine at all. Ultimately, Cullen et al. (2007) suggest the use of more specialized assessments, such as a neuropsychological evaluation. Taken a step further, neuropsychology may also benefit from incorporating eye tracking (and other contemporary technologies) to further elucidate cognitive deficits and the association with functional outcomes (Bilder, 2011; Parsons, 2016; Miller & Barr, 2017; Casaletto & Heaton, 2017; Kessels, 2019). 1.5.3 Neuropsychological Assessment Following Stroke As previously mentioned, neuropsychological assessments are considered the gold standard in delineating cognitive deficits in clinical settings, including stroke rehabilitation (Lezak et al., 2012). However, this service remains relatively underutilized in clinical and research settings, compared to more function- and physically-based assessments, despite many stroke patients experiencing long-term cognitive deficits (Lees et al., 2012; Barker-Collo et al., 2010; Han, Anderson, Jones, Harmann, Sattin, 2014). The use of neuropsychological tests to evaluate specific cognitive domains is warranted for a more thorough understanding of the brain-behaviour relationship and to infer function after stroke (Barker-Collo & Feigin, 2006; Lees et al., 2012). Assessments often include tests of various cognitive domains, including EFs, that are important for tasks of daily living. Impairment of EF is often seen in patients after a stroke and is identified as a significant predictor of functional outcome after brain injury (Srikanth, Thrift, Saling, Anderson, Dewey et al., 2003; Poulin, Korner-Bitensky, Dawson, & Bherer, 2012; Sun, Tan, & Yu, 2014; Barker-Collo et al., 2006; Perna, Loughan, & Talka, 2012; Leininger, Strong, & Donders, 2014). For example, Nys et al (2005) found that patients who demonstrated executive Page 35 of 178 dysfunction within the first three weeks after stroke were seven times more likely to display an array of cognitive impairments 6-months later, compared to patients without executive dysfunction. The Neuropsychological Working Group of the National Institute of Stroke Disorders and Canadian Stroke Network (NINDS-CSN) has recommended measures of EF be included in all assessments of patient after a stroke (Hachinski et al., 2006). 1.5.4 Eye Tracking Assessment Following Stroke Assessment of stroke recovery is often weighted toward motor and sensory function, underestimating the role of cognitive deficits (Lees et al., 2012). Examination of eye movements has been shown to provide a sensitive measure of cognitive deficits in stroke populations, even after recovery of motor and sensory functions (Dong, Yan, Johnson, Millist, Davis, et al., 2013). As well, Lodha and colleagues (2019) highlight the importance of assessing cognitive functions, like visual attention and decision-making, above and beyond motor and sensory outcomes, in determining higher level functioning (e.g., driving) after stroke. Following a stroke, specifically to the right hemisphere, patients may experience a phenomenon known as unilateral spatial neglect (Gammeri, Iacono, Ricci, & Salatino, 2020). Unilateral neglect (UN) is the inability to report, respond, or orient to stimuli presented to the contralateral side of the lesion (Parton, Malhotra, & Husain, 2004) and is associated with poor functional outcomes (Kerkhoff & Schenk, 2012). Previous testing methods in identifying UN after stroke, such as tasks of line bisection, cancellation, drawing/copying, and reading show inconsistent sensitivity and are vulnerable to compensatory strategies by patients, resulting in this phenomenon being overlooked by clinicians (Bonato, 2012; Delazer, Sojer, Ellmerer, Boehme, & Benke, 2018). There is a growing body of research using eye tracking to elucidate the effect of UN on overt visual attention (Walle, Nordvik, Becker, et al., 2019; Upshaw, Leitner, Rutherford, Page 36 of 178 Libben, & Miller, 2019; Delazer et al., 2018; Doricchi, Guariglia, Figliozzi, Silvetti, Bruno, & Gasparini, 2005; Husain, Mannan, Hodgson, et al., 2000; Karnath, Niemeier, Dichgans, 1998). Notably, visual task paradigms to analyze saccades and fixations have been shown to have superior sensitivity to detecting less severe cases of UN than traditional paper-and-pencil methods (Delazer et al., 2018). As well, analysis of fixations using eye tracking during computerized tasks were better at differentiating subtypes of UN (allocentric versus egocentric) after stroke, compared to paper-and-pencil tests, and demonstrated that a combined subtype (allocentric + egocentric) was associated with worse functional independence compared to either subtype alone (Upshaw et al., 2019). While eye tracking has been a fruitful avenue in exploring spatial exploration in patients with UN after a stroke, there is a paucity of research examining the use of eye tracking to elucidate performance on tasks of EF. More generally, lesioning in various areas of the brain has been known to affect saccadic eye movements (Leigh & Kennard, 2004). Patients with stroke in the left or right hemisphere show increased eye movements and fixations compared to controls, suggesting less efficient visual processing (Walle et al., 2018). Patients with stroke in the right hemisphere (who did not present with hemispatial neglect) were more accurate at detecting targets in the ipsilesional visual field (same side as lesion) than in the contralesional visual field. Moreover, longer and more frequent saccades are made in the contralesional visual field in contrast to the ipsilesional visual field (Mapstone, Weintraub, Nowinski, Kaptanoglu, & Gitelman, et al., 2003). These studies have also demonstrated that salient properties of stimuli are processed top-down, opposed to bottom-up, to determine how they match to the properties of a target. While these studies suggest improved detection and diagnostic accuracy compared to paper-pencil methods, less is known about how Page 37 of 178 well eye tracking can predict recovery after a stroke above and beyond neuropsychological test scores. While previous studies have demonstrated distinct eye movement patterns after a stroke (e.g., patients with UN) on experimental designs, less is known about how fixations and saccades during traditional neuropsychological tests of EF (e.g., cWCST) differ between populations and predict test performance, concurrent function, or recovery after stroke. After a stroke, patients often struggle with problem-solving due to perseveration, inflexible thinking, and holding information in mind. In keeping with a process approach, such as the BPA, that highlights how errors and scores are achieved on testing, eye tracking provides a novel set of data for consideration when examining performance on cognitive testing and may provide new avenues for inferring function after a stroke. 1.6 Summary of Literary Review Many factors have coalesced in producing this dissertation and are briefly summarized here. First, updates to neuropsychological testing have primarily been concerned with quantitative revisions to established normative data and reconfiguring traditional test materials. Some have argued that this focus has limited advancement of precision and accuracy of neuropsychological assessments as well as our understanding brain-behaviour relationships. Other fields have benefitted from integration of contemporary technologies. Further, governing groups in the health and social sciences (e.g., NIH, AACN) have suggested incorporating technology into practice would be beneficial, and specifically for neuropsychology. However, cautions and concerns have been raised about the implications of this transition and reliance on computerized testing. Integrating technology into traditional approaches to testing, rather than replacing them, may be more acceptable to clinicians, and provide new types of data to analyze in understanding test Page 38 of 178 performance, sequelae of injury, and recovery. Further, advances in computer technology and software have made eye tracking a tool that is less invasive, cost- and time-efficient, and able to provide rich and detailed data on behaviour during testing, compared to previous versions of eye tracking hardware. As Kessels (2019) states, “there is a need for novel paradigms that are theory-driven on the one hand, but on the other hand can be related to clinical outcomes” (p. 365). Second, existing tests of EF (e.g., WCST) have been challenged for not reflecting daily tasks. Some suggest new types of tools and data, obtained through passive observation during neuropsychological testing, would provide a better reflection of daily behaviour and function. In the past, examination of factors outside test scores using methods, such as the BPA, has been shown to discriminate problems in test performance and identify subtypes of common clinical presentations seen in neuropsychology. However, the BPA is limited to what the examiner can note during the testing process and is difficult to standardize across research. Advances in the power and sophistication of computers now allow for large amounts of specific qualitative data to be recorded for review, processing, and analysis in a non-intrusive manner, to help predict performance and function. How test scores are achieved, and performance is altered, help guide evaluations and intervention recommendations. Further, this integration would ease transition of technology into traditional tests, by not replacing but augmenting the data that is obtained. In accordance with the BPA and qualitative analysis of data, eye tracking data may provide valuable insights during cognitive testing, specifically with EF, such as what details are being attended to over the course of problem solving. Third, eye tracking has shown promise in elucidating various cognitive functions in experimental contexts with healthy individuals and for diagnostic clarity in some clinical populations. Moreover, eye tracking has been purported as an effective tool to assess competency, Page 39 of 178 predict outcomes, and facilitate training of professionals in various fields (e.g., medicine and aviation). Cognitive domains that are assessed by neuropsychology, such as problem-solving, memory, and attention, have been explored using experimental eye tracking paradigms with promising results. Together, findings suggest a strong potential for eye tracking as a tool for neuropsychology. However, despite the surge of eye tracking studies examining the process of cognitive functions, relatively little is known about the validity of eye tracking with neuropsychological testing for inpatients recovering from a stroke. Some neuropsychological tests are visually-based, and have been translated for computer use (e.g., cWCST), providing a platform for eye tracking to be implemented for further evaluation without significant disruption of traditional procedures. Page 40 of 178 CHAPTER 2 Study 1: Eye Tracking During Assessment of Executive Function: A Pilot Study 2.1 Study 1 Purpose The aims of Study 1 were two-fold: (1) To investigate whether fixation dwell time and saccade count predict standardized cognitive test scores. (2) To investigate how manipulating task difficulty (i.e., cognitive load) affects fixation dwell time and saccade count. These two points guided the investigation of validity for eye tracking with cognitive testing and cognitive load (aims #1 and #2). We incorporated a qualitative approach, similar to the Boston Process Approach (BPA), to identify how scores are achieved on testing using video-recordings of eye movements – overt visual attention – that may otherwise be overlooked by an examiner (Poreh, 2000). The quantification of eye movements provided the opportunity to analyze and explore patterns and associations with test scores and the influence of cognitive load during testing. Investigating factors associated with test performance is important to provide incremental value to neuropsychological testing and contribute to our understanding of brain-behaviour relationships (Donders, 2019). Examining the approach to testing (e.g., Boston Process Approach) provides a rich source of information that can augment test results and help us understand how scores and errors occur (Kaplan, 1988; Chaytor & Schmitter-Edgecombe, 2006). For example, in the Framingham Heart Study, various digital technologies were shown to enhance the BPA in Page 41 of 178 assessing and detecting the insidious onset of cognitive decline (Au et al., 2017). Incorporating eye tracking to capture overt visual attention during testing allows novel data for analysis. Dwell time and saccade count have been shown to predict performance on daily activities and standardized cognitive tests in clinical and non-clinical populations (see section 1.4.3). Providing eye tracking in a seamless and unobtrusive manner is important because saccades and fixations are not traditionally (or possible to be) documented in behavioural observations from an examiner and are often unreliable via self-report by the patient (Poreh, 2000). In relation to aim #1 (prediction of performance), higher-level EFs have previously been purported to depend upon basic cognitive functions such as attention (Lezak et al., 2012; Atkinson et al., 2018; Coull, 1998) and difficulties with attention are associated with challenges on tasks of EF (Bennett, 2001). Indeed, fixation time is purported to be an indicator of top-down processing related to goal-oriented behaviour (Lou et al., 2019). Eye tracking research has demonstrated that overt visual attention, based on dwell time and saccades, has been shown to change over the course of problem solving (Hodgson et al., 2000) and correlate with successful problem resolution (Grant & Spivey, 2003). Moreover, as task-mastery is achieved, more attention is paid to relevant, compared to irrelevant, information required for correct responses (Lu Pinke, 2014; Hodgson et al., 2000). While significant correlations between eye movements during novel eye tracking paradigms and performance on the WCST have been reported (Litman, Hommer, Clem, Ornsteen, Ollo, & Picker, 1991) the current study sought to investigate the association between dwell time and saccade count, in real-time, with performance on the cWCST and manualized cognitive tests. Thus, Study 1, aim #1 sought to investigate the concurrent validity of fixations and saccades (eye tracking) by comparing them with scores on the cWCST and other visual- as well as verbal-based cognitive tests. Page 42 of 178 Important to aim #2 (influence of cognitive load), eye movement data provides an indicator of the influence of experimental manipulation during cognitive tasks (Lou et al., 2019). Indeed, cognitive load has been associated with longer dwell times and more saccades during problem solving (Vlaskamp & Hooge, 2006; Hodgson et al., 2000). A stepwise increase to cognitive load in Study 1 was intended as a reasonable approximation to mimic cognitive difficulties after stroke and relatively greater difficulty completing the cWCST (investigated more directly in Studies 2 and 3). Specifically, the dual-task paradigm, where a participant is required to engage in a secondary activity to increase cognitive load and reduce the capacity to fully engage in a primary task, can be an effective method of impeding task performance (Kachouri, Laatar, Borji, Rebai, & Sahli, 2020; Mirelman, Shema, Maidan, & Hausdorff, 2018). For Study 1, low, moderate, and high cognitive loads were used to simulate the healthy control and inpatient stroke groups, respectively, that were recruited for Study 2 and 3. Cognitive load was chosen as a manipulation because cognitive capacity is often reduced after stroke (Cicerone, et al., 2000), that includes executive functions (Nys et al., 2007), and an associated decline with functional independence (Nys et al., 2003). Study 1, aim #2 sought to further investigate the construct validity by examining how fixations and saccades were affected by manipulating cognitive load during the cWCST. In the context of the current dissertation, Study 1 provided an initial investigation on the validity of fixations and saccades during neuropsychological testing. Specifically, Study 1 investigated: (1) the correlation and prediction of cWCST test scores from dwell time and saccades on cWCST AOIs; (2) the correlations of the same eye tracking measures with additional manualized cognitive test scores; and (3) differences on the same eye tracking measures between three cognitive load groups. Moreover, the results from Study 1 helped to inform the design and approach to Studies 2 and 3. The following hypotheses were predicted for Study 1: Page 43 of 178 1. Dwell time and saccade count on AOIs placed on the cWCST cards will differ between cognitive load groups, with the lowest dwell time and saccade count in the low cognitive load group and highest dwell time and saccade count in the high cognitive load group. 2. In accordance with previous findings that performance on problem solving tasks is associated with dwell time on solution relevant and irrelevant information (Hodgson et al., 2000; Hu et al., 2017; Bednarik & Tukiainen, 2007; Cooper et al., 2009), dwell time and saccade count on the cWCST AOIs will predict the number of categories achieved and perseverative errors made during the cWCST; and 3. Dwell time and saccade count on the cWCST AOIs will be significantly correlated with visual-based, but not verbal-based, cognitive tests that are manually administered. 2.2 Methodology 2.2.1 Participants Participants in Study 1 were undergraduate students enrolled in psychology courses at the University of British Columbia Okanagan Campus. Recruitment was conducted through an online recruitment system (SONA). Participants received 1-credit toward their course or $10 as reimbursement for their time. Exclusion criteria included those who: (1) had previously received a neuropsychological evaluation for any reason, (2) had a history of brain injury, neurodegenerative disease, or mental illness, or had (3) uncorrected visual impairment. The total sample size after recruitment was 128 students. Following data collection, examination revealed 5 participants with incomplete/missing eye tracking data and 11 participants with incomplete/missing paper-pencil data. To maximize the sample size, participants with missing/incomplete paper-pencil data but with complete eye tracking and cWCST data were Page 44 of 178 retained for respective analyses (hypotheses #1 and #2). After exclusion, 123 undergraduate students (66% female) were included in analysis. Age of participants ranged from 18 to 42 years of age (M = 20.6 SD = 3.1). 2.2.2 Materials and Procedures Computerized Wisconsin Card Sorting Test (cWCST). The cWCST (Heaton & PAR, 2003) is adapted from the original manualized WCST (Grant & Berg, 1948) and, as described in section 1.2.2 above, assesses abstract reasoning, cognitive set shifting, directing behaviour towards goal achievement, and disinhibition. Participants are presented with a stimulus card and asked to match it to one of four key cards using a computer mouse or keyboard. Successful performance requires matching to one of three sorting principles. The measure consists of a maximum of 128 trials (fewer trials if six categories achieved in less than 128 trials). Trail making test (TMT). The TMT (Reitan, 1955) measures attention, speed, and mental flexibility, as well as visual search and visual-spatial sequencing/scanning abilities (Strauss, et al., 2006, p. 668). The test consists of two parts, each with 25 circles. Part A includes numbers 1 to 25 that are matched in sequential order. Part B, considered a test of EF, includes numbers 1 to 13 and letters A to L that are matched by alternating letter-number-letter in ascending order. Score is recorded in seconds to completion. Digit span (DS). The WAIS-IV DS subtest (Weschler, 2008) is part of the working memory index and measures attention, encoding, and auditory processing. It is comprised of three tasks: recall a series of numbers (1) in order, (2) backwards, and (3) in ascending sequence. Subsequent trials in each task involve an increasing number string. Scores for each task are summed to provide a total score. Page 45 of 178 Coding. The WAIS-IV Coding subtest (Weschler, 2008) is part of the information processing index and assesses short-term visual memory, psychomotor speed, visual perception and scanning ability, visual-motor coordination, and attention. The score is the number of correct marks made in two minutes. Consonant trigrams (CCC). The CCC (Brown, 1958; as cited in Melton, 1963) is a test of auditory working memory and divided attention. The test consists of two parts. The first part requires the immediate repetition of three letters after they are read aloud by the tester. The second part consists of the tester saying three letters, followed by a number. The examinee counts backward by threes from that number out loud for a set amount of time (i.e., 3- 9- or 18-seconds) before the tester signals for the letters to be recalled. Raw scores are summed for each delay and provide a total score. Computer proficiency questionnaire-12 (CPQ-12). The CPQ-12 (Boot, Charness, Czaja, Sharit, Rogers, et al., 2013) contains items related to the participant’s ability to perform tasks on a computer. Items are rated on a 5-point Likert-type scale from 1 (Never tried) to 5 (Very easily). The CPQ consists of 6 topics, with two questions per topic: Computer basics, Printer, Communication, Internet, Calendar, and Entertainment. A total score is obtained by summing responses from each topic. Eye Tracker Setup and Analysis. The study was conducted in an isolated testing room. Manual tests were administered as per the publisher’s standardized instructions. Administration order for the computer and manual tests was counterbalanced. For the cWCST, participants were seated in front of a Dell Precision M4800 laptop equipped with the SMI Infrared Eye Tracker, RED-m 250 (Figure 1). The display and eye tracker were centered to the midline of the participant, 60cm from the participant’s eyes, as per the SMI manual. A chin rest was used to maintain this Page 46 of 178 distance. Stimuli were presented on a 343mm (horizontal) x 190.5mm screen (vertical). Screen resolution was set at 1920x1080 pixels with a 60 Hz refresh rate. Page 47 of 178 Figure 1 Dell Precision M4800 laptop equipped with the SMI Infrared Eye Tracker, RED-m 250 Page 48 of 178 A 5-point calibration and 4-point validation routine was employed for each participant. Fixations within 0.4 degrees on both the x- and y-axes were deemed acceptable. Fixations were defined with a maximum dispersion of 2° visual angle and a minimum duration of 80msec. The SMI software, Experiment CenterTM, was used to create a screen recording of participant eye movements and performance on the cWCST. The SMI BeGazeTM software was used to analyze the video recording of either the number of trials required to complete all 6 categories or after 128 trials were administered, whichever came first. Trial 1 was always omitted from analysis because the participant was free to view the screen, presenting trial 1, while test instructions were read aloud by the examiner. Inclusion of this trial would have artificially inflated fixations and saccades by including approximately 60-seconds of viewing before the participant started the test. Video recordings were clipped at the moment a new screen was displayed indicating the end of testing, again to avoid inflation of eye measures. Four AOIs were created on the cWCST: target card (correct response), distractors (incorrect responses), workspace (previously placed cards), and stimulus card (Figure 2). Notably, the AOIs were not visible to participants during task completion (see Figure 1). Total saccades, total dwell time, and average dwell time on each AOI were included in analyses. The target and distractor AOIs were adjusted to accommodate the new correct and incorrect response on each subsequent trial. Frame-by-frame analysis at 1/10-second was used during this transition to make the adjustment. For standardization across trials, the AOIs were switched at the moment a visual display of “CORRECT” or “WRONG” was on the screen for each trial. The computerized administration ensured that the display provided a consistent, reliable timeframe and method for adjusting the AOIs on each trial. Page 49 of 178 Figure 2 Four areas of interest overlapped onto the video recording of the Computerized Wisconsin Card Sorting Test Note. Areas of interest (AOI): target/correct response (green), distractor/incorrect response (3 red), previously placed cards (yellow), and stimulus card (blue). The green and red AOIs were adjusted at the start of each 128 trials using frame-by-frame review of the recording to stay consistent with correct and incorrect response placements throughout the test. Note that AOIs were not visible to participants during task completion. Page 50 of 178 T-scores for perseverative errors and raw number of categories completed on the cWCST were retained as dependent variables. Standardized total scores were used from the manualized tests (i.e., CCC, DS, Coding, and TMT) that were administered before or after the cWCST. Fixation dwell time was recorded in milliseconds (ms) while fixation and saccade counts were recorded as absolute values. Cognitive Load Manipulation. In the current study, each participant was randomly assigned to one of three cognitive load groups using publicly available online software (randomizer.org). Notably, initial randomization and recruitment resulted in group sizes of 44, 44, and 50. The exclusion in analysis due to missing/incomplete paper-pencil data resulted in further differences between the high cognitive load group and the low and moderate cognitive load groups. The Low Cognitive Load (LCL) group (n = 36) were told once at the start of testing exactly how to match to each of the categories on the cWCST. Participants in the Moderate Cognitive Load (MCL) group (n = 38) were administered the cWCST as per the standardized administration instructions found in the cWCST manual (see Heaton & PAR, 2003). Participants in the High Cognitive Load (HCL) group (n = 49) were administered the cWCST similar to the MCL group. In addition, participants also engaged in a motor task that required rhythmic finger-tapping. Dual tasks that include a motor and a cognitive component often elicit decreased performance in one or both tasks (Leone, Feys, Moumdjian, D’Amico, Zappia, & Patti, 2017) and are associated with more errors during problem solving (de Jong, 2010; Sweller, 1998), specifically on the WCST (Lange, Kroger, Steinke, Seer, Dengler et al. 2016). As well, more saccades and longer dwell times on AOIs are also associated with increasing cognitive load on problem solving tasks (Hodgson et al., 2000; Lu Pinke, 2014). Here, instructions were given to simultaneously tap the index finger of the non-dominant hand (hand not operating the computer mouse) to the beat of a digital Page 51 of 178 metronome, set at 1-beat/second. A research assistant observed the participant, out of the participant’s field of view, to ensure the tapping task was maintained across testing and provided a prompt to continue tapping to the rhythm, as needed. The inclusion of the tapping task to create a dual-task paradigm (i.e., tapping and complete the cWCST) was conducted to direct attentional resources away from the cWCST. The cognitive load manipulation was only applied during completion of the cWCST; manualized tests were administered as per standardized administration guidelines. As well, the manipulation of the LCL group was included to maximize the potential effects of manipulating cognitive load by creating a ceiling effect and capturing dwell time and saccade counts associated with this group. Further, this manipulation was intended to reflect the potential differences that would be seen in cWCST performance between a healthy control group and inpatients admitted to hospital due to a stroke. Statistical Analyses Statistical analyses were conducted in SPSS Version 20. Analyses were conducted using standardized scores of cognitive tests as well as saccades and fixations on AOIs. Preliminary analyses were run to investigate group characteristics and to test assumptions (see Appendix A). On select analyses, and where appropriate, extreme outliers or cases with missing/corrupted data were omitted. Multivariate and univariate analyses as well as correlation and hierarchical regression were conducted to investigate differences between cognitive load groups and the associatin of eye tracking data with cognitive tests scores, respectively. 2.3 Results cWCST Scores Between Groups Performance on the cWCST was analyzed between each cognitive load group, based on categories achieved and number of perseverative errors. Results of Kruskal-Wallis tests revealed Page 52 of 178 a significant difference between cognitive load groups on cWCST categories completed, χ2 (2) = 8.21, p = .017, with a mean-rank category completed score of 66.50 for LCL, 53.71 for MCL, and 54.21 for HCL. Follow-up Mann-Whitney U tests revealed a significant difference in categories completed between LCL (Mdn = 6.00) and MCL (Mdn = 6.00; U = 448, p = .005), as well as between LCL and HCL (Mdn = 6.00; U = 576, p = .005), but no significant difference between MCL and HCL, U = 819, p = .913. Ceiling effects were noted for categories achieved; all groups completed 6 categories. Therefore, the cumulative percentage for each group is provided: in the LCL group, 89% of participants achieved 6 categories, 11% achieved 5 categories; in the MCL group, 77.8% achieved 6 categories, 22.2% achieved 5 or less; and, in the HCL group, 78.3% achieved 6 categories, 21.7% achieved 5 or less. Results of the second Kruskal-Wallis analysis revealed a significant difference between groups and perseverative error T scores, χ2 (2) = 39.28, p < .001. Follow-up Mann-Whitney U tests revealed a significant difference in perseverative errors between LCL (Mdn = 80.0) and MCL (Mdn = 51.0; U = 180.0, p < .001), between LCL and HCL (Mdn = 50.0; U = 146.5, p < .001, but not between MCL and HCL, U = 779.5, p = .65). These results suggest the LCL group performed better on the cWCST than the MCL and HCL groups, but the MCL and HCL groups performed similarly. While the manipulation to reduce cognitive load appeared to be effective, the manipulation to increase cognitive load on participants in the HCL group was not strong enough; shown by similar performance on the cWCST between the MCL and HCL groups on scores of categories completed and perseverative errors. As well, this result may have been due to a general ceiling effect for number of categories achieved in this sample, that was unaffected by the HCL manipulation. Additional exploratory analyses were conducted on number of trials administered, number of correct responses, and total errors. Assumptions were met to conduct parametric analyses using Page 53 of 178 analysis of variance. Results of an ANOVA revealed a significant group difference for trials administered, F(2, 115) = 9.51, p < .001; as well as total error count, F(2, 115) = 11.91, p < .001. No significant difference was found for number of correct responses, F(2, 115) = 0.126, p = .88. Post-hoc analyses, with a Bonferroni-correction, found significantly fewer trials were administered to the LCL group (M = 82, SD = 20) compared to the MCL group (M = 99, SD = 21; p < .001) and the HCL group (M = 99, SD = 18; p < .001). Further, significantly fewer errors were committed by the LCL group (M = 10, SD = 11) compared to the MCL group (M = 28, SD = 22; p < .001) and to the HCL group (M = 28, SD = 21; p < .001). Results are displayed in Table 1. Page 54 of 178 Table 1 Mean Scores on the Computerized Wisconsin Card Sorting Test Across Cognitive Load Groups (N = 123) cWCST Mean Scores Cognitive Load Group LCL (n = 36) MCL (n = 38) HCL (n = 49) Trials Administered (raw) 80 (17.8) 99 (21.5) 99 (18.9) Correct Responses (raw) 71 (9.8) 70 (11.4) 70 (12.2) Perseverative Errors (T-score) 68 (13.7) 50 (15.2) 48 (11.9) Total Errors (raw) 9 (9.1) 28 (21.2) 28 (21.7) Categories Completed (raw) 5.9 (0.23) 5.4 (1.3) 5.3 (1.5) Note. cWCST = Computerized Wisconsin Card Sorting Test. LCL = Low Cognitive Load, MCL = Moderate Cognitive Load, HCL = High Cognitive Load. T-scores = standardized scores adjusted for age and education. Page 55 of 178 2.3.1 Hypothesis #1: Dwell time on AOIs during the cWCST will differ between cognitive load groups, increasing from the lowest to highest cognitive load A multivariate analysis of variance (MANOVA) was conducted to determine the effect of cognitive load on total dwell time (milliseconds) for each of the four AOIs. A significant omnibus effect emerged, V = .22, F(8, 198) = 3.02, p = .003, η2 = .11. Follow-up univariate analyses revealed significant differences for dwell time on workspace, target, and distractor AOIs (p values < .05). Planned contrasts were conducted using LCL as the reference group. Dwell time on workspace was significantly less for the LCL group (M = 90,960ms, SD = 36,350ms) than for the MCL group (M = 123,888ms, SD = 47,224ms, p = .006), and dwell time on the stimulus card was less for the LCL group (M = 79,054ms, SD = 30,848ms) than for the MCL group (M = 97,783ms, SD = 32,326ms, p = .02). As well, dwell time on the target (correct response) was less for the LCL group (M = 34,382ms, SD = 18,350ms) than the HCL group (M = 44,420ms SD = 22,093ms, p = .03), and dwell time on distractors (incorrect responses) was less for the LCL (M = 59,319ms SD = 30,464ms) than the and HCL group (M = 76,649ms SD = 36,322ms, p = .02). Taken together, total dwell times were significantly shorter for the LCL group than the MCL and HCL groups on all AOIs. A mixed measures two-way ANOVA was run to determine differences in total dwell time across each AOI. An interaction emerged between AOI dwell time and cognitive load, V = .13, F(6, 200) = 2.22, p = .04, η2 = .06. Mauchley’s Test indicated violation of sphericity, χ2(5) = 123.54, p < .001; thus, the Greenhouse-Geisser statistic was interpreted for within-subject effects and the interaction was not significant F(3.51, 177.38) = 2.12, p = .08, η2 = .04. However, within-subjects effects revealed a main effect for AOI mean dwell time, F(1.76, 177.38) = 84.94, p < .001, η2 = .46. Collapsing cognitive load groups, significant differences were found between all AOIs Page 56 of 178 (all p values < .001). Overall, AOI dwell time was greatest for workspace (M = 108,499ms, SD = 47,591ms), followed by stimulus card (M = 86,467ms, SD = 32,324ms), distractors (M = 67,578ms, SD = 36,781ms), then target (M = 38,985ms, SD = 19,699ms). As well, there was a main effect for mean dwell time across cognitive load F(2, 101) = 5.98, p . 004, η2 = .11. Contrasts revealed a significant difference of overall dwell time during completion of the cWCST between LCL (M = 65,929ms SE = 3478ms) and MCL (M = 81,647ms, SE = 3368ms, p = .002) groups, as well as LCL with HCL (M = 78,571ms, SD = 2939ms, p = .007) groups. Results are displayed in Figure 3. Page 57 of 178 Figure 3 Average Total Dwell Time for Each Area of Interest on the Computerized Wisconsin Card Sorting Test Note. Areas of Interest (AOI) = Target [correct] response, Workspace [previously placed cards], stimulus card, or distractions [incorrect response options]). A main effect for AOI dwell time was found. Most time was spent on the workspace, followed by the stimulus card, distraction responses, then target response. Error bars represent 95% CI. Page 58 of 178 Discussion Study 1, Hypothesis 1 In summary, there was partial support for the first hypothesis; cognitive load influenced eye tracking measures. In keeping with a process approach, despite similar test performance (e.g., all groups achieving 6 categories), the LCL group had shorter AOI dwell times associated with fewer trials administered and fewer errors than the MCL and HCL groups (see Table 1 for cWCST scores). Shorter AOI dwell times from the LCL group were also associated with fewer errors being made and suggest fixations in the LCL group were predominantly directed toward correct responses. Findings are in keeping with previous studies showing that successful problem solving is associated with fixations toward solution-relevant information (Grant & Spivey, 2003; Hu et al., 2017), that increasing cognitive load increases number of fixations during task completion (Heuer & Hallowell, 2015; Lu Pink, 2014), and that fixations distinguish between high- and low-performers during trial-and-error problem solving tasks (Hu et al., 2017). Moreover, a main effect was found for AOI dwell time with significantly more time on the workspace (previously placed cards) than the stimulus or key cards and this difference was most pronounced in the MCL and HCL groups. Taken together, results are in keeping with findings by Hodgson et al (2000) showing that more fixation on workspace compared to a target goal and longer fixations with increasing task difficulty. Our manipulation of rhythmic tapping (1-beat/sec) did not elicit the predicted effect of cognitive load burden. These results contrast earlier findings that suggest cognitive-motor interference during dual task engagement (Leone et al., 2017; Papegaaij et al., 2017), including rhythmic tapping (Park & Brunken, 2014). Two explanations are offered for this finding. First, the monotony of the motor-task; the pace of tapping was consistent, likely reducing the burden effect on working memory and attentional resources once the beat was established. Therefore, cognitive Page 59 of 178 resources could be re-allocated to the cWCST. Another explanation for this finding is that the cWCST was a relatively simple task for this population sample (ceiling effect). Indeed, normative data from the publisher’s manual suggest young adults with more than high school education achieve all 6 categories with relatively few perseverative errors. The manipulation of increasing cognitive load was not powerful enough to overcome the ceiling effect. 2.3.2 Hypothesis # 2: Total and average dwell time and saccades will predict scores for the number of categories achieved and perseverative errors on the cWCST Spearman’s Rho correlation analyses were conducted to examine the relationship of total dwell time and saccades with perseverative errors and categories completed on the cWCST. These two scores are most often utilized in research and in clinical settings to (Strauss et al., 2006). We found the cWCST scores had moderate correlations with total dwell times on each AOI (r = .-20 to -.40), weak to moderate correlations with average dwell times per fixation on each AOI (r = .00 to .23), and moderate to strong correlations with saccade count (r = -.45 to -.53). Results are displayed in Table 2. Next, we examined how eye tracking measures were able to predict select scores on the cWCST. Results of preliminary assumption tests are summarized in Appendix A. The first hierarchical linear regression was run using number of categories completed. Anxiety and alertness were also included in the models to account for their potential influence on test performance (McDonald, 2002; see section 1.3.1). A three-level model was created: (1) demographic variables, as well as subjective anxiety and alertness at time of testing; (2) scores on the Coding, Digit Span, TMT A & B, and CCC; and (3) total and average dwell time AOIs and total saccade count. The third-level model of demographics, paper-and-pencil test scores, and eye tracking measures was statistically significant, R2 = .45, F(18, 58) = 2.63, p = .003, adjusted R2 = .28. The addition of eye Page 60 of 178 tracking measures for the prediction of number of categories completed for the third-level model was a significant increase in R2 of .37, F(9, 58) = 4.34, p < .001. Total dwell time on workspace (β = -0.45, p = .02) was a statistically significant predictor for number of categories on the cWCST. Results are displayed in Table 3. Our second hierarchical linear regression was run using the same model to predict T-scores for perseverative errors (higher score equals fewer errors). The third-level model was statistically significant, R2 = .53, F(18, 56) = 3.52, p < .001, adjusted R2 = .38. The addition of eye tracking measures to the prediction of T-scores for perseverative errors was a significant increase in R2 of .43, F(9, 56) = 5.74, p < .001. Alertness (β = 0.23, p = .03), Digit Span score (β = 0.22, p = .01), as well as dwell time on Workspace (β = -0.60, p = .002) and Distractors (β = -0.42, p = .03) were significant predictors in the final model. Results are displayed in Table 4. Discussion Study 1, Hypothesis 2 In summary, we found the final hierarchical model that included total and average dwell time and total saccade count on AOIs from the cWCST was statistically significant at predicting perseverative errors and number of categories completed on the cWCST, above and beyond demographics and other cognitive test scores. In the final models, increasing workspace dwell time was associated with fewer number of categories completed (i.e., lower performance). For perseverative error T-scores, increasing workspace dwell time and distractors predicted more perseverative errors while increasing alertness and Digit Span scores predicted fewer perseverative errors. These findings are in keeping with previous studies showing longer fixations on a workspace were associated with more difficulty determining the solution to a visual problem (Hodgson et al., 2000) and with increasing cognitive load on working memory (Rouinfar et al., 2014; Lu Pinke, 2014). Digit Span score and alertness predicted fewer perseverative errors Page 61 of 178 suggesting attention and working memory contributed to adapting responses as necessary (cognitive flexibility) and maintaining a new correct response pattern. More focus toward the workspace and poorer performance may suggest that correct conceptualization on the cWCST category has not been achieved, there is more uncertainty about card placement, and therefore, a greater chance of perseverative responding. Taken together, and in line with past research (Chaytor et al., 2006; Kaplan, 1988) eye tracking provided some insight and association to cWCST scores with findings offering some initial support for the integration of contemporary technology to further elucidate performance via a process approach. 2.3.3 Hypothesis #3: Dwell times and saccade counts during the cWCST will be significantly correlated with other visual-based, but not verbal-based, cognitive tests Correlations between eye tracking measures and scores on the manualized tests (Coding, Digit Span, CCC, and TMT) were weakly to moderately correlated (r = .01 to .24). The strongest correlations were observed between total dwell time on the distractor key cards with TMT Part A time (r = -.24, p = .01), stimulus card dwell time with CCC scores (r = -.22, p = .03), and total saccade count and the Digit Span WAIS-IV subtest (r = -.21, p = .04) .In this case, lower scores on the TMT indicated longer, rather than shorter, time to complete the test (worsening performance). Results are displayed in Table 2. Page 62 of 178 Table 2 Spearman Rho Correlation Matrix of Eye Tracking Measures with Neuropsychological Test Scores (N = 117) Target Distract Workspace Stimulus TMT A TMT B cWCST Cat cWCST PE DS CCC CD TOTAL FIXATION DWELL TIME Target 1 .78* -.38* .28* -.16 -.09 -.20* -.28* -.03 -.04 -.01 Distract -- 1 -.17 .30* -.24* -.19 -.33* -.40* .00 -.15 -.03 Workspace -- -- 1 .19 -.02 .01 -.32* .34* -.05 .01 -.01 Stimulus -- -- 1 .05 -.04 -.24* .26* -.16 -.22* .05 AVG FIXATION DWELL TIME Target 1 .87* .19* .41* -.01 -.10 -.14 -.10 .10 .05 -.05 Distract 1 .28* .50* -.10 -.13 -.23* .18 .13 -.01 -.05 Workspace 1 .43* .14 .17 -.11 .00 -.02 -.10 -.02 Stimulus 1 .14 .00 -.03 .07 .10 -.05 -.02 TOTAL SACCADES -.10 -.13 -.45* -.53* -.24* -.17 -.02 Note. cWCST = Computerized Wisconsin Card Sort Test; TMT = Trail Making Test; DS = Digit Span (WAIS-IV); CCC = Consonant Trigrams; CD = Coding (WAIS-IV). Total Fixation Dwell Time = Total time fixating on AOI during cWCST. Average Fixation Dwell Time = Average time per fixation on AOI during cWCST. Total Saccades = Total saccade count during completion of the cWCST. * = p < .05; significant correlations are in bold text. Page 63 of 178 Table 3 Hierarchical Regression (Three-level) of Demographic, Cognitive Test Scores, and Eye Tracking Data on Categories Completed From the Computerized Wisconsin Card Sorting Test cWCST Categories Completed Model 1 Model 2 Model 3 Variable B β B β B β Constant 5.55 -- 5.94 -- 9.33 -- Age (yrs) 0.02 0.04 .02 .04 -0.08 -0.19 Education (yrs) -0.06 -0.05 -0.03 -0.03 0.16 0.15 Sex -0.07 -0.03 -0.12 -0.04 -0.32 -0.12 Alertness 0.13 0.11 0.20 0.18 0.14 0.12 Anxiety 0.05 0.06 0.05 0.06 -0.00 -0.00 TMT A -- -- -0.03 -0.03 -0.19 -0.15 TMT B -- -- 0.20 0.25 0.10 0.13 Digit Span -- -- 0.03 0.05 0.03 0.05 Coding -- -- -0.01 -0.01 -0.02 -0.04 CCC -- -- -0.02 -0.01 -0.03 -0.17 Total Saccades -- -- -- -- 0.00 0.01 Target Dwell Time -- -- -- -- 0.00 0.34 Target Avg Fixation -- -- -- -- -0.01 -0.25 Work Space Dwell Time -- -- -- -- -0.00 -0.45* Work Space Avg Fix -- -- -- -- 0.03 0.11 Stimulus Card Dwell Time -- -- -- -- -0.00 -0.30 Stimulus Card Avg Fix -- -- -- -- 0.00 0.20 Distractor Dwell Time -- -- -- -- -0.00 -0.33 Distractor Avg Fix -- -- -0.01 -0.26 R2 (Adjusted) / F .02 (.00) / 0.31 .08 (.00) / 0.56 .45 (.28) / 2.63** ΔR2 / F for ΔR2 -- .06 / 0.31 .37 / 4.34*** * = p < .05, ** = p < .01, *** = p < .001 ; significant correlations are in bold text. Page 64 of 178 Table 4 Hierarchical Regression (Three-level) of Demographic, Cognitive Test Scores, and Eye Tracking Data on Perseverative Errors From the Computerized Wisconsin Card Sorting Test cWCST Perseverative Errors Model 1 Model 2 Model 3 Variable B β B β B β Constant 42.22 -- 23.81 -- 53.91 -- Age (yrs) 0.11 0.02 0.16 0.03 -0.89 -0.17 Education (yrs) 0.22 0.02 -0.30 -0.02 0.64 0.05 Sex -0.17 -0.01 1.85 0.05 4.04 0.12 Alertness 2.43 0.16 3.21 0.22 3.25 0.23* Anxiety 0.86 0.08 0.32 0.03 -0.30 -0.03 TMT A -- -- 0.01 0.00 -.3.00 -0.19 TMT B -- -- 0.21 0.02 -0.35 -0.04 Digit Span -- -- 1.35 0.20 1.41 0.22* Coding -- -- 0.39 0.07 0.11 0.02 CCC -- -- 0.14 0.06 0.10 0.04 Total Saccades -- -- -- -- 0.00 0.01 Target Dwell Time -- -- -- -- 0.00 0.02 Target Avg Fixation -- -- -- -- -0.11 -0.50 Work Space Dwell Time -- -- -- -- 0.00 -0.60* Work Space Avg Fix -- -- -- -- 0.04 0.12 Stimulus Card Dwell Time -- -- -- -- 0.00 -0.23 Stimulus Card Avg Fix -- -- -- -- 0.09 0.32 Distractor Dwell Time -- -- -- -- 0.07 -0.42* Distractor Avg Fix -- -- 0.00 -0.40 R2 (Adjusted) / F .04 (.00) / 0.65 .10 (.00) / 0.76 .53 (.38) / 3.52*** ΔR2 / F for ΔR2 -- .06 / 0.88 .43 / 5.74*** * = p < .05, ** = p < .01, *** = p < .001 ; significant correlations are identified in bold text. Page 65 of 178 Discussion Study 1, Hypothesis #3 While it was anticipated that dwell time and saccades on the cWCST would be positively correlated with nonverbal tests (i.e., TMT and Coding) but not with verbal tests (i.e., CCC and Digit Span), the results of the hypothesis are mixed. Aside from cWCST scores, correlations between eye tracking data from the cWCST and additional cognitive tests were weak with some select tests showing significant correlations. Specifically, total dwell time on the cWCST distractor cards were weakly correlated with scores on the TMT-A (r = -.24) and the TMT-B (r = -.19). While both the cWCST and TMT require some degree of cognitive flexibility and visual scanning for successful completion, the TMT is a timed task that is relatively more reliant on psychomotor speed. In contreast, the cWCST is an untimed task that requires conceptualization and may be more influenced by working memory and less by speeded visual scanning. The TMT-B is a measure of EF and the weak correlational findings with the cWCST are mostly consistent with previous results that demonstrated the shared variance is small to moderate, at best, between the WCST and TMT (Testa et al., 2012) and the Booklet Category Test which is also purported to examine abstract reasoning and concept formation (Donders & Kirsch, 1990). As well, total dwell time on the cWCST stimulus card was weakly correlated with scores on the CCC (r = -.22). This finding may reflect difficulty holding in mind details about stimulus information, similar to letters provided before the distractor task on the CCC. A weak but significant correlation (r = -.24) was also found between total saccades and scores on the DS WAIS-IV subtest. These findings are somewhat in keeping with results from Ballard et al (1995) who found more fixation time on the stimulus helped to reduce cognitive load on working memory – both tapped by the CCC and DS tests. Together, current results support the heterogeneous nature of EFs, as well as the multifactorial approach to examining EF due to different abilities affecting performance (e.g., Page 66 of 178 cognitive flexibility, inhibition, and concept formation). Moreover, current findings add that this multifactorial structure of EF and tests to measure EF also extend to eye tracking data from the cWCST. The current results show more frequent and moderate correlations between dwell time and scores on the cWCST and some significant but weak correlations with scores on select manualized tests. While saccades and fixations have been shown to correlate with cognitive test scores (Friedman et al., 1995; Klein, Rauh, & Biscaldi, 2010; Yang et al., 2013), these are often with standalone, novel, eye tracking paradigms. Current results expanded previous findings by demonstrating that fixations and saccades have a stronger association with the paired-test scores (cWCST) than other administered cognitive test scores (e.g., CCC, DS, Coding, TMT), suggesting some evidence of discriminant-convergent validity. To put another way, poor discriminant-convergent validity would be observed if fixations and saccades had stronger and more frequent correlations with other cognitive test scores and not with the cWCST scores. 2.4 Study 1 General Discussion Using a process approach, we investigated the construct and criterion validity of eye tracking data during the cWCST. Specifically, AOIs were set on the cWCST to examine: (1) how well fixation dwell times and saccades predict performance based on select scores from the cWCST; (2) how cognitive load influences fixation dwell times and saccades during completion of the cWCST, and (3) how well fixation dwell times and saccades predict scores on verbal and nonverbal manualized cognitive tests. In summary, partial evidence was found to support the construct validity and criterion validity of eye tracking data during the cWCST: dwell time on AOIs of the cWCST predicted performance based on number of categories completed and Page 67 of 178 perseverative errors (aim #1); and fixations and saccades were affected by experimental manipulation of cognitive load (aim #2). Regarding unexpected findings, some results are worth highlighting. While we found support of select eye tracking measures on AOIs predicting test scores on the cWCST, the initial hypothesis of discriminant-convergent validity was not supported. However, results revealed support for an alternative hypothesis by demonstrating fixations and saccades on cWCST AOIs had more frequent and stronger correlations with cWCST scores than other tests administered - manualized paper-pencil tests without the eye tracker. As well, the experimental manipulation of cognitive load provided partial evidence of influencing eye tracking results (fixations and saccades) on the cWCST. Results revealed that when cognitive load was reduced (LCL group) there was a significant decrease in fixations and saccades. In contrast, our manipulation for higher cognitive load (testing + finger tapping) was ineffective and/or insufficient to elicit a change in cWCST scores or eye tracking data; we saw no differences in fixations or saccades between the MCL and HCL group. Fortunately, and important to the replication of methodology in Study 2, while some initial setup was required (e.g., calibration), once testing began, administration was seamless and mirrored the standardized administration of the cWCST. Taken together, the use of eye tracking as a tool for quantifying the process of problem solving (e.g., cWCST) was evident in the results of Study 1. These findings are in keeping with studies supporting the integration (and interest) of technology with neuropsychological assessment (Miller & Barr, 2017; Au et al., 2017; Parsons & Dufferson, 2019; Kessels, 2019; Poreh, 2006). To examine the external validity of these results, the use of eye tracking data with the cWCST was utilized in Study 2 to determine if current findings could be generalized to older adults in clinical and community settings. Page 68 of 178 CHAPTER 3 Study 2: Assessing Criterion and Construct Validity of Eye Tracking Data in Clinical and Community Samples 3.1 Study 2 Purpose The objective of Study 2 was to replicate and expand on the findings of Study 1 by comparing eye tracking data and cognitive test scores between inpatients admitted to hospital after a stroke and age-matched healthy controls from the community. Specifically, Study 2 had the following aims: 1) To investigate the discriminative power of fixations and saccades on AOIs (criterion validity). 2) To investigate how well total and average time per fixation and saccade count, on AOIs of the cWCST, predict cognitive test scores across groups (predictive validity). 3) To explore differences in fixation and saccade counts according to conceptual versus non-conceptual response styles (response process). As applied to clinical populations, process approaches, such as the BPA (Kaplan, 1988), include a qualitative analysis of testing taking (e.g., formulated strategies and error types) to help identify cognitive dysfunction and provide more detailed appreciation for cognitive strengths and weaknesses. Using this method, the focus includes the nature and efficacy of strategies employed by the patient in arriving at a correct or incorrect response, in addition to total scores. Information can be gleaned about (mal)adaptive strategies and behaviours on testing by recording and analyzing the process of problem-solving and decision-making. An example of this approach is Page 69 of 178 used during the Block Design subtest of the WAIS-IV, where the examiner documents any errors or strategies for how the patient works to configure the blocks based on a target design. The BPA has also been demonstrated using drawing tasks, such as the Rey-Osterreith Complex Figure Test, where the examiner documents the process by which the drawing is copied to provide an understanding of planning and organizational skills. As well, the WCST (manualized and computerized versions) offer additional test scores (described in section 1.2.2) that help to understand performance, such as perseverative and non-perseverative responses, failure to maintain set, and number of conceptual level responses. However, despite inclusion of these additional test scores, WCST scores have previously demonstrated only poor to adequate ecological validity for predicting living independence with patients after stroke (Chiu et al., 2017). Moreover, behaviour and extra-test variables, such as approach to testing (e.g., strategies) have been shown to account for a substantial amount of additional variance in outcomes (Chaytor et al., 2006). Along these lines, use of eye tracking has received growing interest in research and clinical settings to augment cognitive testing and predict functional and goal-oriented behaviour (Heitger et al., 2008; Land, 2009; Foulsham, 2015). Subtle qualitative behaviour, such as overt visual attention, that is not easily or reliably available to the examiner or accurately recalled by the examinee (Poreh, 2006), can be captured using eye tracking for quantification and further analysis. Impairment in attention is common after stroke in the acute inpatient setting (Nys et al., 2005; Hochstenbach, Mulder, van Limbeek, Donders, & Schoonderwaldt, 1998). Attentional impairment can result in functional difficulties and challenges with other cognitive activities even when other cognitive domains remain intact (Lezak et al., 2012). Overt attention is purported to be a foundation from which EFs stem and has been shown to predict goal-oriented actions and behaviour (Atkinson et al, 2018; Coull, 1998). Therefore, impairment to visual attention processes Page 70 of 178 may undermine some aspects of EF such as visual problem-solving (Lezak, 2012; Bennett, 2001). Indeed, significant correlations between fixations on novel paradigms designed to assess cognitive function and manualized neuropsychological test scores of global cognition, working memory, and EF have previously been reported (Oyama, Takeda, Ito, Nakajima, Takeya, et al., 2019; Poletti et al. 2017). For example, Oyama et al (2019) monitored eye movements on a brief and novel cognitive screener with an eye tracker and had participants focus their gaze (i.e., fixate) on desired responses. The authors found that eye tracking “cognitive scores” (p. 2) were significantly correlated with traditional neuropsychological tests, and able to differentiate between healthy controls, patients with mild cognitive impairment, and patients with dementia. Following a stroke, patients may exhibit significantly more saccades and fixations during visual search tasks compared to a healthy control group (Mapstone et al., 2003). This discrepancy may reflect a greater increase in cognitive load for patients during the task resulting in less efficient visual search (Walle et al., 2019). As well, Husain et al. (2001) found that after a stroke, patients re-fixated on previously viewed targets during a cancellation task more often than healthy controls suggesting a failure to retain searched locations and less efficient visual search patterns. While previous studies have suggested an association between overt visual attention and performance on tests of EF, and that a stroke can impact both, there are relatively few studies that have directly investigated this with eye tracking. Moreover, none have examined the use of eye tracking during tests of EF in an acute inpatient setting. It is well-established that lower performance on tests of EF, such as the cWCST, is predicted by damage to the prefrontal cortex regions and structures of the brain, often affected by stroke (Yamauchi, Nishii, Higashi, & Kagawa, 2010; Nyhus & Barcelo, 2009; Nagahama, Fukuyama, Yamauchi, et al., 1996; Stuss, Levine, Alexander, et al., 2000). Neuroimaging studies Page 71 of 178 have also found that damage to non-frontal regions, such as the temporal, subcortical, hippocampal, and cerebellar regions are associated with poor performance on the cWCST (Nyhus & Barcelo, 2009). Findings of previous studies have demonstrated that impaired performance on the tasks of EF, including the WCST, are not exclusive to frontal and prefrontal brain regions. Thus, in addition to location of brain lesion(s), it is important to examine other factors that may help elucidate performance on testing. Given the nature of the cWCST as a visually based problem-solving test, this provides the opportunity to utilize eye tracking to examine how saccade and fixation counts on testing (i.e., key cards, stimulus cards, and previously placed cards) predict test scores. Taken together, previous studies have demonstrated that attentional impairment is common after stroke and may influence other cognitive and functional domains and that fixations and saccades correlate with scores on cognitive tests and can differentiate clinical populations from control groups. However, few studies have explored the integration of eye tracking, looking at saccades and fixations, with standardized neuropsychological tests after a stroke. Study 2 aimed to investigate this issue and was the first to explore real-time eye tracking during the cWCST with inpatients admitted to hospital after a stroke. Based on the aims established at the outset of Study 2, the following hypotheses were made: 1. Eye tracking will differentiate between controls and inpatients; a. Participants in the control group will have fewer saccades and fixation counts, overall, during completion of the cWCST, and b. will have fewer fixations and lower average time per fixation on each cWCST AOI (workspace, key cards, and stimulus cards). Page 72 of 178 2. In both groups, fixation on cWCST AOIs will predict number of categories achieved on the cWCST. 3. In both groups, more fixations on the cWCST AOIs will be made during non-conceptual responses (trial and error) compared to conceptual responses (10 sequentially correct responses). 4. In both groups, fixations and saccades on the cWCST AOIs will hold stronger correlations with other visual-versus verbal-based cognitive test scores that are manually administered. 3.2 Methodology 3.2.1 Methodological Changes from Study 1 to Study 2 and 3 Transitioning the study from a laboratory setting to an inpatient clinical setting, some changes to methodology were made to accommodate inpatients after a stroke, the setting, and the new information gleaned from Study 1. The following changes were applied to the methodology and data collection to Studies 2 and 3: 1) Regarding eye tracking measures, to decrease the time of video review and analysis of eye tracking data, the AOIs on the key cards in Study 1 (target and distractors) were combined into one ‘key card AOI’. Collapsing the four AOIs into one on the key cards provided a simplification for data analysis (i.e., one AOI for all key cards rather than four key card AOIs) as well as a buffer zone between AOIs to reduce chance of error during recording (see Orquin, Ashby, & Clarke, 2015) and provide more robust data for saccades and fixations (Hessels et al., 2016). Page 73 of 178 2) Preliminary trials with the inpatients showed the chinrest used in Study 1 to be troublesome for inpatients and ineffective due to a tendency to return to a seated upright position and out of the catchment area of the eye tracker. Given the flexibility of remote eye trackers to allow some rotation of the neck and head, and to maximize participant comfort and increase data acquisition, the chinrest was removed. 3) As noted in Sections 1.4 and 2.2, three fixation variables per AOI were obtained by the eye tracker: total dwell time, average dwell time per fixation, and fixation count. Use of fixation counts replaced total fixation dwell time in Studies 2 and 3. Analyses in Study 2 revealed a high degree of collinearity (r > .80 for stroke and r > .90 in healthy controls) between these two fixation variables. To accommodate the change in AOI placement on the key cards (see methodological change point #1 above) fixation count was used in place of total dwell time as participants would likely be fixating on separate cards in considering their responses, as opposed to dwelling on the entire key card AOI per se. Moreover, fixation count has been shown to provide an index of cognitive load during problem solving (Zagermann et al., 2016). Thus, fixation count and average dwell time per fixation were retained for analyses; total fixation dwell time was removed. 4) Last, the inpatient test battery varied slightly over the course of the study. Due to the variability of test selection in the clinical setting, we chose to include inpatients regardless of the composition of their test battery, as long as the cWCST was completed. This approach was chosen to avoid severely limiting sample size due to Page 74 of 178 variation in the test battery for inpatients (see Measures section below). In contrast, participants in the control group were administered a fixed research test battery comprised of tests consistently administered to the inpatients group. To address variation in testing, standardized scores from tests purported to measure a similar major cognitive domain (e.g., Color Trails Test and Trail Making Test) were subsumed into a composite score (see Song, Lin, & Ward, 2013 for review of transformation options). Tests comprising each cognitive domain are displayed in Table 3. Given the potential for cohort effects using computerized testing with an older adult population (Morrell, Park, Mayhorn, & Kelly, 2000), the CPQ-12 questionnaire was retained to account for any confounding effects regarding computer familiarity and skills. The questionnaires for alertness and anxiety were also kept as these factors are common in the early stages of stroke recovery (Campbell, Murray, Holmes, et al., 2012). A screening measure for depressive symptoms (Patient Health Questionnaire-9) was also included to account for low mood that is often noted during the early stages of recovery from a stroke (Paolucci, 2008). 3.2.2 Participants Participants in Study 2 were recruited from the inpatient rehabilitation unit at Kelowna General Hospital upon referral for a neuropsychological evaluation, and from volunteer and community centres in Kelowna. Ninety-six participants were recruited for the study: 49 inpatients (44% female) recovering from stroke at Kelowna General Hospital’s Rehabilitation Unit, and 47 healthy individuals (59% female) recruited from the community to serve as the control group. In both groups, those with a prior history of a neurological condition (e.g., multiple sclerosis, traumatic brain injury, previous stroke, or dementia), primary presentation of a major psychiatric Page 75 of 178 disorder, comorbid drug or alcohol abuse, or developmental disorder were excluded from the study. After data collection, eight inpatients enrolled in the study were excluded from analyses: one participant had previously completed a neuropsychological assessment that included the WCST, five participants had incomplete eye tracking data (e.g., cataracts or excessive movement during testing), and two participants chose to discontinue with testing. One participant from the control group was excluded due to a technical difficulty with the eye tracker and no data being recorded. After these additional exclusions, the total sample for the study was 87 participants (41 inpatients and 46 controls). Page 76 of 178 Table 5 Cognitive Tests Comprising Each Cognitive Domain and Composite Score Executive Function Memory Visuospatial-constructional Motor Attention Processing Speed Fluency Problem Solving Verbal Visual Constructional Spatial CTT WAIS-IV CD COWAT cWCST SRT ROCFT 3’ WAIS-IV BD ROCFT Copy GPB TMT RBANS CD RBANS VF RAVLT ROCFT 30’ Clock Draw Digit Span RBANS Word/Story Note. CTT = Colour Trails Test; TMT = Trail Making Test; Digit Span = Digit Span subtest from WAIS-IV; CD = Coding subtest; RBANS = Repeatable Neuropsychological Assessment of Neuropsychological Status; RBANS VF = RBANS Verbal Fluency; COWAT = Controlled Oral Word Association Test; SRT = Buschke Selective Reminding Test; RAVLT = Rey Auditory Verbal Learning Test; ROCFT = Rey-Osterreith Complex Figure Test; BD = Block Design subtest WAIS-IV; Clock Draw = Clock Drawing Test; GPB = Grooved Pegboard Page 77 of 178 3.2.3 Materials and Procedures Controlled Oral Word Association Test (COWAT). The COWAT (Benton, 1969), is a test of verbal fluency. The test is comprised of four, 60-second trials: three require the generation of as many words that begin with a given letter (e.g., F, A, or S), and the fourth trial requires generation of as many words that belong to a category (e.g., animals). The test taps the domain of executive function; assessing the ability to retrieve words, as well as monitor and inhibit incorrect responses. The first total score is calculated by summing the first three trials (phonemic fluency) and the second total score is from the fourth trial (semantic fluency). Age and education have been shown to influence performance; education being a strong predictor for FAS score and age being a stronger predictor for category score; while sex did not predict scores (Tombough, Kozak, & Rees, 1999). The consequences of stroke have been shown to have a significant impact on COWAT scores compared to healthy controls, even after controlling for demographics and stroke risk factors (Weinstein, Preis, Beiser, Au, Kelly-Hayes, et al., 2014). Rey-Osterrieth Complex Figure Test. The Rey-O Complex Figure Test (RCFT; Osterrieth, 1944) assesses visuospatial recall, visuospatial recognition, response bias, processing speed, and visuospatial constructional ability. The test-retest reliability of the RCFT ranges from .76 to .89, and inter-rater reliability ranges from .93 to .99 (Lezak, Howieson, Loring, Hannay, Fischer, 2004). The figure consists of 18 elements that are individually scored based on accuracy and placement compared to the presented stimulus (i.e., drawn copy). Scores are obtained from the drawn copy trial, as well as the 3- and 30-minute delay drawing trials, with higher scores equating to better performance. Block Design subtest (WAIS-IV). The WAIS-IV Block Design is a subtest that assesses visuospatial and motor skills (Wechsler, Coalson, & Raiford, 2008). Raw scores are converted to Page 78 of 178 standardized scores based on age-corrected normative data. Participants are presented with blocks, colored red, white, or red and white on each side. A stimulus design is presented, and the blocks are arranged to match the stimulus, starting with two-block, then four block, and then nine-block designs. Trials are discontinued at set times, with better scores being achieved the faster the trial is completed. Scores on the block design are strongly correlated with Full Scale IQ on the WAIS-IV. Grooved Peg Board. The Grooved Peg Board Test (GPB) measures manual dexterity in the left and right hands. Pegs are picked up, one at a time from a dish, and manipulated to match the orientation of each hole in the pegboard. There are five rows of five holes, and performance is based on the time it takes to fill all 25 holes. The test is conducted in two trials, starting with the dominant hand then the non-dominant hand. Time on the GPB is often impaired following stroke (Haaland & Delaney, 1981). Moderate correlations have been found with visual acuity and dominant-hand performance (Schear & Sato, 1989), as well as measures of visual attention (e.g., TMT-B; Strenge et al., 2002). Buschke Selective Reminding Test. The Buschke Selective Reminding Test (SRT; Buschke, 1973) assesses verbal learning and memory; specifically, long-term storage, random long-term retrieval, continuous long-term retrieval, short-term recall, short delay cued recall and recognition, long delay free- and cued-recall, and long delay recognition. Clinically, the SRT has been shown to differentiate healthy individuals from those with brain injury (Levin, Grossman, & Kelly, 1976), dementia and multiple sclerosis (Rao, et al., 1989), and damage between the left and right hippocampi (Sass, Spencer, Kim, et al., 1990). Rey Auditory Verbal Learning and Memory Test. The Rey Auditory Verbal Learning and Memory Test (RAVLT) assesses immediate memory, learning, susceptibility to interference Page 79 of 178 and recognition memory (Rey, 1958). The test consists of five recall trials of a list of 15 words, followed by a distractor list trial of 15 new words, short- and long-delay recall trials, and a recognition trial. Total score on the RAVLT has excellent internal reliability (r = .90; Van den Burg & Kingma, 1999). Stefanova et al (2002) demonstrated scores on the RAVLT for patients after a stroke are significantly lower than healthy controls; the test is sensitive to, and can distinguish between, these two populations. Repeatable Battery of the Assessment of Neuropsychological Status. The Repeatable Battery of the Assessment of Neuropsychological Status (RBANS) is a brief test of cognitive function, consisting of 12 subtests, assessing five domains: immediate memory, visuospatial-constructional skills, language, attention, and delayed memory (Randolph, 1998). Raw scores are converted to age- and education corrected standardized scores in the publisher’s manual. Internal reliability for total score is excellent (r = .86 to .94; Gontkovsky et al., 2004; Randolph, 1998). Wilde et al (2006) reported differences between inpatients with left and right hemisphere strokes on a rehabilitation unit; patients with left hemisphere strokes had lower scores on Language/Verbal Memory indices while those with right hemisphere strokes had lower scores on visuospatial/visual memory indices. Color Trails Test. The Color Trails Test (CTT; Maj, D’Elia, Satz, Janssen, Zaudig, Uchiyama, Starace, Galderisi, & Chervinsky, 1993) assesses speed of attention, sequencing, mental flexibility, visual search, and motor function. The CTT is very similar in nature to the Trail Making Test (see Study 1) but minimizes reliance on language by substituting letters (TMT) for color; thus, completion requires alternating color rather than letter. Additionally, it reduces language requirements and cultural bias. Test-retest reliability is reported to be .64 for part 1 scores Page 80 of 178 and .79 for part 2 scores (Straus et al., 2006). Age- and education-corrected normative data are available in the test manual. Patient Health Questionnaire-9. The Patient Health Questionnaire-9 (PHQ-9; Kroenke, Spitzer, & Williams, 2001) is a screening tool for depression. The measure consists of 9-items, each on a scale of 0 (not at all) to 3 (nearly every day), that are summed to create total scores from 0 to 27. The higher the score, the greater the endorsement of depressive symptomatology. The PHQ-9 was designed to screen for mild, moderate, and severe levels of depression, with scores of 5, 10, 15, and 20, representing mild, moderate, moderately severe, and severe depression, respectively. Scores on the PHQ-9 have been shown to be a significant predictor of engagement in rehabilitation following stroke (Kringle, Terhorst, Butters, & Skidmore, 2018). Information regarding the inpatient’s stroke, demographics, medical history, and assessments pertaining to rehabilitation were obtained via their departmental or electronic medical file. Cognitive testing was conducted by trained psychometrists blinded to the study. Participants in the control group completed an abbreviated test battery that reflected core tests for the inpatients (i.e., WAIS-IV Coding, COWAT-FAS, TMT, and cWCST) that was administered by the primary author or a trained research assistant. Test scores were standardized based on published normative data. Prior to introduction of the eye tracker and administration of the cWCST, participants completed the alertness and anxiety scales, and CPQ-12 (see Study 1 for measure description). Procedures for the eye tracker reflected those of Study 1 and administration of the cWCST was conducted as per the manual. Participants were provided a standard introduction to the computer and eye tracker, including a 9-point calibration to ensure eye movements were being captured in all quadrants of the screen within an acceptable angle (see Section 2.2.1 for procedure from Study 1). Screen-recording was started after calibration and stopped at the completion of the last trial on Page 81 of 178 the cWCST. BeGaze software provided by SMI was used to analyze eye tracking of each participant’s screen recording. Video clips were analyzed frame-by-frame at 1/10-second to ensure standardization of AOI placement/recording into the video, start-stop of video cropping, and when examining within-subject variations on select consecutive trials (10 consecutive correct trials and 10 non-conceptual level responses). Video-recordings were clipped to start after placement of the first card (i.e., trial 1) to exclude eye movements that would have been recorded while participants listened to instructions from the researcher, and therefore, irrelevant to the aim of the study. Video was discontinued after placement of the last trial. To examine within-subject differences between conceptual level response and non-conceptual level responses, video was clipped when a subject placed 10 consecutive correct cards to achieve a category and 10 incorrect responses. Moreover, selection of the incorrect responses was chosen by excluding a string of 10 trials where more than 5 correct responses in a row were made (set loss) and where most responses (5 or more out of 10) were perseverative. These criteria were chosen to better reflect trial-and-error during cWCST trials. 3.3 Results Statistical Analyses All statistical analyses were conducted in SPSS Version 20. Relevant assumption testing and power analyses are reported in Appendix A. Analyses were conducted using standardized scores of cognitive tests and data points from AOI characteristics produced by the software BeGaze for eye movement and AOI characteristics. Preliminary analyses were conducted to investigate group characteristics and assumptions specific to each respective analysis. Non-parametric analyses were employed, or preferable statistical values were interpreted as appropriate, when assumptions were violated or findings would be more robust, respectively. Analyses were Page 82 of 178 conducted in a stepwise approach: first, classification power of eye tracking between two groups was examined; second, differences on AOI characteristics during the cWCST was examined; and last, the association between AOI characteristics and composite scores based on cognitive testing, including the cWCST, were examined. Demographic and Stroke Characteristics Demographics (age, education, sex) and stroke characteristics were assessed prior to the main analyses and are displayed in Table 4. Age of participants ranged from 61 to 91 years (M = 73.8, SD = 7.7) in the control group, and 39 to 97 years (M = 72.2 SD = 13.3) in the inpatient group. Years of education ranged from 9 to 21 years (M = 14.8 SD = 2.5) in the control group and from 7 to 21 years (M = 12.2 SD = 2.9) in the inpatient group. The time from injury to testing for the inpatient group ranged from 2 to 73 days (M = 18.2, SD = 14.9). Stroke occurred in the left hemisphere of 18 inpatients (44%) and in the right hemisphere of 23 inpatients (56%). Ischemic stroke was diagnosed in 33 inpatients (81%) and hemorrhagic stroke was diagnosed in 8 inpatients (19%). Page 83 of 178 Table 6 Demographic Information of the Healthy Control and Stroke Inpatient Groups Control (n = 46) Stroke (n = 41) Age (M, SD) 73.8 (7.7) 72.2 (13.3) Gender (M:F) 19:27 23:18 Education (M, SD) 14.8 (2.5) 12.2 (2.9) Days since injury (M, SD) Handedness (L:R) -- 1:45 18.2 (14.9) 6:35 Stroke Hemisphere Left (n, %) Right (n, %) -- 18 (44%) 23 (56%) Stroke Type Ischemic (n, %) Hemorrhagic (n, %) -- 33 (81%) 8 (19%) Stroke Region Anterior (n, %) Posterior (n, %) -- 31 (76%) 10 (24%) Page 84 of 178 cWCST Scores Between Groups Select cWCST scores were compared between the healthy control group and inpatient group (Table 5). Independent Samples t-tests revealed more categories were achieved by the healthy control group (M = 4.2, SD = 2.2) compared to the inpatient group (M = 1.9, SD = 2.1), t(83) = 4.72, p < .001, d = 1.07. Fewer perseverative errors (T scores) were made by healthy controls (M = 55.4, SD = 12.59) compared to inpatients (M = 46.27, SD = 11.26), t(84) = 3.53, p = .001, d = 0.76. More conceptual level responses (T scores) were made by healthy controls (M = 53.02, SD = 10.77) compared to inpatients (M = 44.51, SD = 10.12), t(84) = 3.77, p < .001, d = 0.81. Time to complete (seconds) the cWCST was shorter for healthy controls (M = 853.3s, SD = 310.0s) than for inpatients (M = 1386.7s, SD = 437.8s), t(74) = 6.22, p < .001, d = 1.41. Page 85 of 178 Table 7 Raw cWCST Scores Between Healthy Controls (n = 46) and Inpatients After a Stroke (n = 41) Variables (raw score) Control Stroke Total Trials 113.1 (20.0) 122.4 (15.1) Correct Responses 72.4 (14.5) 62.7 (17.9) Total Errors 40.6 (23.2) 59.7 (21.0) PR 23.1 (20.3) 40.5 (23.9) PE NPE 20.6 (16.4) 20.0 (11.0) 34.5 (18.6) 25.2 (12.9) CLR Categories Trials 1st Cat FMS L2L 60.8 (19.8) 4.2 (2.2) 29.2 (32.4) 1.4 (1.7) -1.1 (7.14) 42.2 (23.7) 1.9 (2.1) 58.8 (51.8) 1.5 (1.6) -7.9 (9.98) Task Time (sec) 853.2 (310.0) 1383.3 (444.6) Note. Values represent raw scores (standard deviation). Trials = total trials administered, Correct = total correct responses, Total Errors = total incorrect responses, PR = perseverative responses, PE = perseverative errors, NPE = non-perseverative errors, CLR = conceptual level responses, Categories = total categories completed (maximum 6), Trials 1st Cat = trials to complete the 1st category, FMS = failure to maintain response set (i.e., switching response set after at least 5 consecutive correct responses provided), L2L = learning to learn, Task time = total time in seconds to complete cWCST. Page 86 of 178 3.3.1 Hypothesis #1a: Between Group Differences on Total Saccade and Fixation Counts To investigate whether measures of eye tracking during the cWCST could predict group membership (healthy control or inpatient admitted to hospital after a stroke) a binomial logistic regression was run using total saccade and fixation counts. Results of the logistic regression were significant, χ2(2) = 21.6, p < .001. The Hosmer & Lemeshow Test for model fit was not statistically significant, χ2(8) = 5.19, p = .74, suggesting a good fit. The model explained 31% (Nagelkerke R2) of the variance in predicting stroke and correctly classified 74.4% of cases. Sensitivity was 62.2% (correctly predicted a stroke occurred in those who had a stroke) and specificity was 84.4% (correctly predicted to not have had a stroke when a stroke did not occur). Positive predictive value was 74.4% (of all cases predicted of having a stroke, 74.4% were correctly predicted) and negative predictive value was 73.1% (of all cases predicted not having a stroke, 73.1% were correctly predicted). Fixation count was a significant predictor (p = .01), but saccade count was not (p = .20). That is, having a stroke was associated with a significantly higher fixation count (and a non-significant trend for higher saccade count) compared to healthy controls. The area under the ROC curve for fixation count was .75 (95% CI, .65 to .86), defined as a(n) fair/acceptable level of discrimination (Hosmer & Lemeshow, 2000) for the model. A summary of the logistic regression model is displayed in Table 6. Page 87 of 178 Table 8 Logistic Regression Predicting Likelihood of Stroke Based on Fixation and Saccade Counts B SE Wald df OR 95% CI for OR Lower Upper Fixation Count 0.631 0.263 5.76 1 1.88 1.12 3.15 Saccade Count Constant 0.185 -2.71 0.144 0.68 1.66 15.75 1 1 1.20 0.06 0.98 -- 1.60 -- Note. OR = Odds ratio; fixation and saccade counts are total counts from completion of the Computerized Wisconsin Card Sorting Test. The model explained 31% of the variation between healthy controls and inpatients after a stroke and classified 74% of cases. Fixation counts was a significant contributor to the model (p = .01), saccade count was not significant (p = .20); more fixations were associated with increased likelihood of stroke. Page 88 of 178 3.3.2 Hypothesis #1b: Group Differences on cWCST AOIs To further examine the criterion validity, we analyzed fixations on the cWCST key cards, workspace (previously placed cards), and stimulus card AOIs between the two groups using a one-way 2 (group) x 3 (AOI) MANOVA. Results revealed a significant difference between groups on a combination of the AOI fixation counts, λ = .822, F(3, 81) = 5.84, p = .001, η2 = .18. Post-hoc analyses were conducted using non-parametric Mann-Whitney U tests. Results revealed significant differences on key card fixation count between stroke (Mdn = 1018) and healthy controls (Mdn = 673, U = 620, p = .014); as well as workspace fixation count between stroke (Mdn = 1126) and healthy controls (Mdn = 610, U = 540, p = .002). No significant difference was found on stimulus fixation count between stroke (Mdn = 555) and healthy controls (Mdn = 607, U = 839, p = .59). The results suggest more fixations occur on the workspace and key cards, but not on the stimulus card, of the cWCST after a stroke when compared to healthy age-matched individuals. Discussion Study 2, Hypothesis #1a & 1b Fixation count during the cWCST was able to differentiate between healthy controls and inpatients. Number of fixations made during completion of the cWCST were significantly lower for healthy individuals than those who had experienced a stroke, and specifically, on the key cards and workspace (previously placed cards). These findings are similar to the findings of Study 1 that found increasing cognitive load (LCL versus MCL and HCL) resulted in increased dwell time on cWCST AOIs. Moreover, Study 2 findings support previous clinical studies that have shown eye movements are effective at differentiating normal and abnormal brain function (Leigh & Kennard, 2004; Mapstone et al., 2003), and differentiate healthy controls from mild cognitive impairment and dementia (Oyama et al., 2019; Pavisic, Firth, Parsons, Rego, Shakespeare, et al., 2017); Parkinson’s disease (Amador et al., 2006); autism spectrum disorder (Fujioka, Inohara, Okamoto, Page 89 of 178 Masuya, Ishitobi, et al., 2016); schizophrenia (Benson et al., 2012), and post-concussive syndrome (Heitger et al., 2009). Moreover, current findings corroborate previous studies that found patients after a stroke have a higher number of fixations and saccades on visual search tasks (Mapstone et al., 2003; Hussain et al., 2001). This discrepancy in eye movements is ostensibly due to less efficient visual processing and task difficulty (i.e., high cognitive load) (Walle et al., 2019; Lu Pinke, 2014; Tokuda, 2008; Vlaskamp & Hooge, 2006). Common cognitive challenges post-stroke (e.g., attention, working memory, cognitive flexibility) may result in more fixations and saccades during the cWCST that is further associated with longer time on individual trials and the overall test. Increasing task complexity (cognitive load) has been shown to increase fixation count on a battery of tests assessing EF (Lu Pinke, 2014). Extending the findings of Study 1, current results support the criterion validity using eye tracking during the cWCST to differentiate between healthy controls and inpatients admitted to hospital after a stroke. 3.3.3 Hypothesis #2: Fixation on AOIs Predicting cWCST Performance We investigated whether fixation count on AOIs of the cWCST predicted number of categories completed (a common outcome measure of the cWCST used in clinical and research settings; Chiu et al., 2017; Strauss et al., 2006). Two hierarchical regression analyses were run, one for each group, and both consisted of two levels. Anxiety level and computer proficiency score were entered into the first level, and fixation count on each AOI and saccade count into the second level. Demographic information was not included in the model as it is accounted for during test scoring when converting raw scores to standardized T scores. For predicting number of categories achieved on the cWCST in the healthy control group, Model 1 (anxiety and computer proficiency) accounted for 10% (adjusted R2 = 6%) of the variance. This was not a significant change from no model F(2, 42) = 2.27, p = .12. Addition of Model 2 Page 90 of 178 (fixation count on AOIs and total saccade count) led to an increase in R2 of 36%, representing a significant change from Model 1, F(4, 38) = 6.40, p < .001. Model 2 was a significant predictor of number of categories achieved in healthy controls, F(6, 38) = 5.41, p < .001, R2 = 46% (adj. R2 = 38%). Total number of saccades was a significant predictor in the final model (β = -0.48, p = .02). For predicting number of categories achieved on the cWCST in the stroke group, Model 1 accounted for 4% (adj. R2 = 1%) of the variance, F(2, 37) = 0.72, p = .49. Addition of Model 2 increased R2 by 37%, representing a significant change from Model 1, F(4, 33) = 5.18, p = .002. Model 2 was a significant predictor of number of categories achieved in inpatients after a stroke, F(6, 33) = 3.80, p = .005, R2 = 41% (adj. R2 = 30%). Significant predictors were fixation count on key cards (β = -0.63, p = .02) and CPQ-12 score (β = 0.35, p = .02). Using Fisher’s z transformation, the correlations of the final models were compared between the healthy control and inpatient admitted to hospital after a stroke. The two models did not differ between these groups, Z = 0.31, p = .76. Eye tracking results were able to predict scores on the cWCST for both groups; albeit, with differing eye tracking characteristics for each group. Discussion Study 2, Hypothesis #2 Concurrent validity of fixation count on AOIs and total saccades during completion of the cWCST were investigated. Above and beyond subjective reports of anxiety and computer proficiency, number of fixations and saccades increased the accounted for variance in predicting categories achieved for healthy controls and inpatients after stroke by 36% and 37%, respectively. Current results corroborate previous research suggesting fixations on problem tasks were associated with standardized cognitive tests of global function, working memory, and EF (Oyama et al., 2019; Poletti et al. 2017). These findings also extend previous research showing that time Page 91 of 178 spent on card-inspection and reflection after feedback during the cWCST predicted number of categories achieved (Somsen, 2007). Current results provide support for a process approach with eye tracking, suggesting fixations help predict test performance in healthy and inpatient stroke populations. Notably, task mastery on the cWCST, represented here by consecutive correct responses to complete a target category, is associated with fewer saccades in healthy older adults and fewer fixations on key cards in inpatients after a stroke. The former coincides with previous studies showing less efficient (e.g., sporadic) saccades predict poor performance on reading tests (Bax, 2013) and memory retrieval (Johansson & Johansson, 2014). The latter coincides with findings by Hu and colleagues (2017) who found “high performers” had fewer fixations on an AOI representing the problem operation (workspace) area during a visual problem-solving task compared to “low performers” (p. 304). Together, results of hypothesis 2 suggest that cWCST can be predicted by fixations and saccades, but specific predictors are dependent on the population. In keeping with findings by McDonald (2002 suggesitng previous computer experience should be considered when interpreting test scres, we also found computer proficiency (CPQ-12) was significant in predicting number of categories on the cWCST for people after a stroke. Moreover, alternative testing modalities (e.g., manualized versions) or additional orientation and/or practice with the computerized test prior to administration should be considered to minimize the influence of extraneous factors on test results. Current results would suggest this will be important when assessing elderly inpatients, specifically after a stroke. 3.3.4 Hypothesis #3: Differences on AOI Fixations Based on Response-Type A three-way, 2 (group) x 3 (AOI) x 2 (response type), mixed-ANOVA was conducted to investigate how fixation count differed between the two groups, on each AOI, as they completed conceptual and non-conceptual level responses. There was a significant two-way interaction Page 92 of 178 between response-type (conceptual and non-conceptual) and AOIs, F(2, 106) = 20.47, p < .001, partial η2 = .28. Follow-up simple main effects of response-types revealed a statistically significant difference in average fixation count on key cards for conceptual responses (M = 50.0 SD = 40.1) and non-conceptual responses (M = 95.0 SD = 53.4), F(1, 59) = 76.15, p < .001, partial η2 = .56, a difference of 45 fixations, 95% CI [34.0 to 54.2]. A statistically significant difference was found for average fixation count on workspace for conceptual responses (M = 54.0, SD = 30.6) and non-conceptual responses (M = 90.1, SD = 49.5), F(1, 59) = 48.9, p < .001, partial η2 = .45, a difference of 36 fixations, 95% CI [25.8 to 46.6]. A statistically significant difference was found for average fixation count on stimulus card for conceptual responses (M = 40.8 SD = 19.4) and non-conceptual responses (M = 51.0 SD = 22.3), F(1, 59) = 28.0, p < .001, partial η2 = .34, a difference of 11.8 fixations, 95% CI [7.3 to 16.2]. Examining the simple main effect of AOI, we found that the mean fixation count was significantly different between AOIs in the conceptual level responses, F(1.72, 94.54) = 3.69, p = .035, partial η2 = .06. Pairwise comparison, with a Bonferroni correction (critical p value = .013), revealed a significant difference in fixation count between workspace (M = 56.1 SD = 30.2) and stimulus card (M = 40.8, SD = 19.4; p = .006), a difference of 15 fixations, 95% CI [3.7 to 26.9]. A statistically significant difference was found between fixation counts within AOIs in the non-conceptual level responses, F(2, 114) = 19.87, p < .001, partial η2 = .26. Pairwise comparisons, also with a Bonferroni correction, revealed a significant difference in fixation count between key card (M = 95.4, SD = 52.8) and stimulus card (M = 51.0, SD = 22.3; p < .001), a difference of 44 fixations, 95% CI [25.6 to 63.1], as well as workspace (M = 92.5, SD = 48.6) and stimulus card (p < .001), a difference of 42 fixations, 95% CI [25.3 to 57.6]. Results are displayed in Figure 4. Page 93 of 178 Figure 4 Average Fixation Counts on Each Area of Interest of the Computerized Wisconsin Card Sorting Test Note. Areas of interest (AOI) = key cards, workspace, or stimulus card by conceptual or non-conceptual/trial-and-error (T&E) responses. Over 10 consecutive trials, on average, runs of conceptual responses had fewer fixations across all AOIs compared to runs of non-conceptual responses. Within conceptual responses, there were significantly more fixations on the workspace than the stimulus card. Within the non-conceptual responses, there were significantly more fixations on both the workspace and key cards than the stimulus card. Page 94 of 178 Discussion Study 2, Hypothesis #3 We examined AOI fixation counts based on response type: conceptual responses (10 correct trials to achieve a category) and non-conceptual responses (10 trial-and-error responses attempting to determine category). No between-group (stroke versus control) differences were found, but within-group differences were found for all AOI fixation counts and response type. That is, more fixations were made on all the AOIs during non-conceptual responses compared to conceptual responses, with differences of 45, 36, and 12 fixations for key cards, workspace, and stimulus cards, respectively. Regardless of response type, fewer fixations were made to the stimulus card than the workspace or key cards. The relative similarity between fixations on workspace and key cards may be due to the program permitting selection of placement by clicking on either the workspace or the key cards and/or checking previously placed cards before placement. Fewer fixations to the stimulus cards may be because, once details of the card were obtained, more attention was directed toward considering the workspace and placement options regardless of conceptual or non-conceptual (trial-and-error) response types. Once the concept of the category was achieved, fixations on all AOIs significantly decreased. This finding was in keeping with previous studies showing distinct eye movement patterns are observed over the course of problem solving (Hodgson et al., 2000; Hu et al., 2017), and specifically, that fixations and saccades toward problem-relevant information are more likely to produce correct responses (Grant and Spivey, 2003). Here, during non-conceptual responses, relatively more fixations were made across all AOIs, in keeping with findings by Ballard et al (1995) who reported that during construction of a block design, fixations on the target design reduced the cognitive load placed on working memory. Current findings build on this as well as the results of Study 1 (fewer fixations in the LCL group) in that relatively fewer fixations were Page 95 of 178 observed with conceptual versus non-conceptual responses. Similarly, we found once a category was identified and conceptual responses were being made (represented by consecutive correct responses until category completion) fewer fixations were made to non-target areas (e.g., workspace, stimulus card). Overall, results of hypothesis 3 support the quantification of a process approach (see Poreh, 2000; Au et al., 2017) and construct validity of eye tracking data by showing distinct differences in eye movement patterns during stages of problem solving (conceptual versus trial-and-error) during the cWCST. 3.3.5 Hypothesis #4: Correlation of Eye Tracking with Other Cognitive Tests and Demographics Spearman’s Rho correlations were run to examine the association between eye tracking (fixation count and average time per fixation on each AOI, and total saccade count), demographics, and manually administered cognitive tests. Composite scores for each cognitive domain (see Section 3.2.1) were used for analyses with the inpatient group. In the control group, correlation between age and eye tracking measures ranged from weak to moderate (r = .10 to .49); a significant correlation was found with total saccade count (r = .49, p = .001). Sex and education were not significantly correlated with eye tracking measures. Correlations between eye tracking measures and sex were weak to moderate (r = .00 to .24) and with education were weak to moderate (r = .00 to .22). Compared with other cognitive tests, only one significant correlation emerged between performance on TMT-A and average time per fixation on the stimulus card (r = .36, p = .02) and total saccade count (r = -.38, p = .01). No significant correlation was found between scores on Coding or COWAT and eye tracking measures. Correlation results are presented in Table 7. Page 96 of 178 Table 9 Spearman Rho Correlations Between Select Cognitive Test Scores and Eye Tracking Data From the Areas of Interest on the Computerized Wisconsin Card Sorting Test for the Control Group (n = 46) TMT-A TMT-B WAIS-IV Coding COWAT-FAS COWAT-Category Key cards Fix Count Average Fixation -.30 .02 -.20 .00 -.19 -.12 .-23 -.01 -.11 -.11 Workspace Fix Count Average Fixation .01 .26 .14 -.07 -.01 .02 .05 .22 .02 -.02 Stimulus Fix Count Average Fixation .00 .36* -.10 .28 .02 .05 -.09 .20 -.19 .00 Total Fixation -.20 -.16 -.08 -.13 -.08 Total Saccades -.38* -.22 -.18 -.14 -.07 Note. TMT-A/B = Trail Making Test Part A and B; WAIS-IV Coding = Weschler Adult Intelligence Test – Fourth Edition Coding subtest; COWAT = Controlled Oral Word Association Test. Fix count is the total number of fixations made on the AOI. Average fixation = average time (ms) per fixation. Total fixation and saccades were the total number made across all AOIs during completion of the cWCST. * = p < .05; significant correlations are in bold font. Page 97 of 178 In the stroke group, age and eye tracking measures ranged from weak to strong (r = .17 to .59); a significant correlation was found with saccade count (r = .59, p < .001). Correlations between eye tracking measures and sex were weak to moderate (r = .10 to .47); significant correlations were found between sex and average time per fixation on each AOI (key card, workspace, and stimulus; r = -.43, -.47, and -.38, respectively, all p values < .05) with less time found with females. Correlations between eye tracking measures and education ranged from weak to moderate (r = .01 to .22); no significant correlations were found. Comparing screening measures for computer proficiency, mood, anxiety, and alertness with fixations on the AOIs no significant correlations emerged with PHQ-9 or anxiety scores (all p values > .05). Scores on the CPQ-12 were significantly correlated with fixation count on Workspace (r = -.34) and total saccade count (r = -.40). Level of alertness was significantly correlated with key card average fixation (r = .44). Orientation scores (person, time, and place) were significantly correlated with key card average fixation (r = -.41). Examining composite scores, correlations with the visual-based EF composite ranged from weak to moderate (r = .00 to -.44) with significant correlations emerging for key card and stimulus AOI fixation counts. In contrast, verbal-based EF composite fell in the weak range (r = .02 to -.19). Correlations with visual-based memory composite score ranged from weak to moderate (r = -.01 to -.29). Correlations with verbal-based memory composite score ranged from weak to moderate (r = .10 to -.27). Correlations with visuospatial-constructional skills composite scores ranged from weak to moderate (r = -.02 to -.42). Motor speed was also included and showed weak to moderate correlations (r = .01 to -.30). Correlation results for composite scores and eye tracking data are presented in Table 8. Page 98 of 178 Table 10 Spearman Rho Correlations for Cognitive Domain Composite Scores and Fixations on the Computerized Wisconsin Card Sorting Test for Inpatients After Stroke (n = 36)ǂ EF Visual EF Verbal Memory Visual Memory Verbal Visuospatial Constructional Motor Speed Key cards Fix Count Average Fixation -.44* -.35* .14 .02 -.01 -.29 .19 -.25 -.42* -.31 -.06 -.30 Workspace Fix Count Average Fixation -.29 .00 -.13 .02 .08 -.22 .10 -.27 -.15 -.02 -.08 .03 Stimulus Fix Count Average Fixation -.37* -.23 .02 -.07 -.13 -.28 .21 -.16 -.26 -.27 .15 -.12 Total Fixation -.43* -.03 -.03 .15 -.25 .01 Total Saccades -.22 -.19 .23 .22 -.13 .12 Note. EF-Visual tests: cWCST nonperseverative errors, Coding WAIS-IV, Trail Making Test Part B/Color Trails Test – 2; EF-Verbal tests: Controlled Oral Word Association Test FAS and Category/RBANS Fluency; Memory Visual: Rey Osterreith Complex Figure 3-minute and 30-minute delays; Memory Verbal: Total List and Long Delay Free Recall of Buschke Selective Reminding Test/California Verbal Learning Test – Second Edition/Rey Auditory Verbal Learning Test/RBANS List Learning and Delay; Visuospatial Constructional: Block Design (WAIS-IV) and Rey-Osterreith Complex Figure Copy; Motor Speed: Grooved Peg Board Dominant and Non-dominant hands. Average fixation = average time per fixation (ms). Fixation count is the total number of fixations made on the AOI. Total fixation and saccades = total number made across all AOIs during completion of the cWCST. Correlations on composites were only computed for the inpatient stroke group, because they were tested with a comprehensive battery, not applicable to the control group. ǂ = Some cases could not be included in analysis because of abbreviated testing that prevented computation of composite scores. * = p < .05; significant correlations are in bold font. Page 99 of 178 Discussion Study 2, Hypothesis #4 Investigating the discriminant and convergent validity in these two older adult populations, we examined how well fixations and saccades correlated with visual and verbal tests of various cognitive abilities (based on test scores or composite scores). Similar to Study 1 and contrary to predictions, there were few significant correlations between fixations and saccades with neuropsychological test scores (or composite scores) in both groups, regardless of the visual or verbal nature of the tests. However, also similar to Study 1, there was evidence of convergent-discriminant validity for eye tracking data such that fixations and saccades converged with cWCST scores but showed more divergence from other cognitive test scores. Notably, there were more significant correlations associated with the visual EF composites scores than other cognitive composite scores. While findings partially support attention serving as a basis for EF, such as problem solving (Atkinson et al., 2018; Lezak et al., 2012; Paelecke-Habermann et al., 2005; Bennett, 2001; Coull, 1998), evidence for the convergent-discriminant validity is not as strong as initially anticipated. Previous studies have also demonstrated correlations between saccades on experimental paradigms and performance on neuropsychological tests (Friedman et al., 1995; Yang et al., 2013; Crawford, Higham, Renvoize, Patel, Dale, et al., 2005). While our results revealed weak to moderate correlations with other test scores (control group) this could be a result of using differing eye tracking data compared to previous studies (e.g., saccadic parameters on novel paradigms such as pro- and anti-saccadic tasks) and with other clinical populations (e.g., patients with Alzheimer’s disease or schizophrenia). A positive correlation was found between age and number of saccades in both groups and is in keeping with previous studies that have suggested elderly adults have less the executive control (e.g., inhibition) of saccadic eye movements (Crawford, Smith, & Berry, 2017; Abel & Page 100 of 178 Douglas, 2007; Peltsch, Hemraj, Garcia, & Munoz, 2011). Performance on the cWCST (categories and trials administered) has been shown to decline with age (Strauss et al., 2006). The current observation that more saccades were associated with increasing age may be due to more trials administered and time spent on the cWCST, representing declining executive abilities. Self-reported measures of mood and anxiety had little association with fixations and saccades on the cWCST. In contrast to results from Poletti et al (2017) indicating depressive symptoms affected perceived usability of eye tracking, we found no association in the current study. Differences may highlight the importance of using non-invasive equipment for eye tracking in a clinical or research context to reduce intimidation or adverse reactions to more contemporary methods of cognitive assessment. Here, we also measured computer experience (CPQ-12), alertness, and orientation, and found each to be associated with fixations and saccades during testing. Taken together, findings suggest individual factors such as mood, anxiety, alertness, and computer proficiency should be screened as they may affect both engagement and results from eye tracking on cognitive tests. 3.4 Study 2 General Discussion Study 2 sought to replicate and extend the findings of Study 1 by examining the construct and criterion validity of eye tracking data during the cWCST in healthy older adults and inpatients after stroke. Further evidence of criterion validity for eye tracking during the cWCST was provided with results revealing significantly more fixations occurring on the key cards and workspace for inpatients after a stroke and that overall fixation count was able to adequately differentiate the two groups. Further investigation of the construct validity of eye tracking data was explored by examining how fixations and saccades predicted test performance and differed between conceptual- and non-conceptual responses on the cWCST. Findings revealed distinct patterns Page 101 of 178 between the two types of responses with significantly fewer fixations on all AOIs during conceptual versus non-conceptual (trial-and-error) responses. As well, discriminant-convergent validity was also examined by comparing eye tracking results with other verbal and visual cognitive test scores. Similar to the findings of Study 1, results did not support our initial hypothesis for the convergent-discriminant validity but once again demonstrated an alternative to this hypothesis. Specifically, fixations and saccades predicted performance on the cWCST but were weakly correlated with scores on manualized tests and cognitive domain composite scores except the visual-based EF composite score. This finding again supports the assumption that attention is an underlying mechanism for higher-order cognitive processes (Lezak et al., 2012; Coull, 1998) and in keeping with findings by Atkinson et al (2018) that suggest overt visual attention is associated with goal-directed behavior. In addition, consideration should be given to screening for comfort and skill on a computer prior to testing with older adults who have suffered a stroke, given this cohort may be more likely to report uncertainty or difficulty during computerized testing (McDonald, 2002). Here, lower scores on the CPQ-12 reflecting less experience, were associated with more fixations on previously placed cards (workspace) and more total saccades during the cWCST. This may suggest people scan the screen and fixate on their choices more due to unfamiliarity of the test modality (computer) prior to responding across trials. Thus, it is suggested that traditional paper-pencil modalities be considered, or additional time be granted, for those who have limited experience with computers. Overall, the results of Study 2 provided support for the validity of eye tracking data during cognitive testing for inpatients after stroke. Study 3 aimed to further investigate the ecological validity of eye tracking data in the same inpatient stroke sample during rehabilitation. Page 102 of 178 Chapter 4 Study 3: Ecological Validity of Eye Tracking Data During Recovering from a Stroke 4.1 Study 3 Purpose The objective of Study 3 was to expand on the findings of Study 2, by assessing the ecological validity (veridicality) of eye tracking using scores on a functional outcome measure after stroke. Specifically, Study 3 aimed to: 1) Investigate if fixations and saccades on the cWCST predict admission, discharge, and change in functional independence for inpatients who have experienced a stroke. Neuropsychological assessments are often employed to infer function, potential for recovery, and treatment recommendations (Kibby, Schmiterr-Edgecombe, & Long, 1998). In this context, cognitive testing would benefit most, not from replacing traditional testing methods, but from the addition of contemporary technology into existing approaches (Parsons & Duffield, 2019; Miller & Barr, 2017). Moreover, the integration of contemporary methods is gaining traction and has now become an integral part of longitudinal cohort studies (e.g., Framingham Heart Study) to obtain more detailed qualitative information regarding performance on cognitive assessments (Au et al., 2017). Understanding the process of test performance, in a structured and controlled environment, can help to translate results more readily to function in daily living (Franzen & Wilhelm, 1996; Goldstein, 1996; Burgess et al., 2006). However, a recent critical review of neuropsychological assessments by Donders (2019) insisted that more research was required to ascertain the incremental value of neuropsychological testing to improving diagnostic accuracy and functional outcomes in clinical practice. Indeed, as a discipline, neuropsychology continues to Page 103 of 178 strive for methods of assessment and testing to improve inferences made from the test setting to daily functioning (Kessels, 2019; Lezak et al., 2012). Neuropsychologists often make inferences about how performance on a battery of cognitive tests will predict independence on tasks of daily living such as self-care, money management, cooking, driving, return-to-work/school. Tests of EF have been shown to adequately predict function in the short- and long-term (Nys et al., 2005). However, while previous studies have shown adequate predictive validity of EF test scores, some have argued that the weak to modest correlations with functional outcomes suggests there is room for improvement (Chaytor & Schmitter-Edgecombe, 2003). Generally, accounting for testing factors outside of traditional scores has been shown to drastically improve inferences regarding daily function (Kaplan, 1989; Chaytor et al., 2006; Lamar et al., 2010). Computerized tests and alternative computerized formats for traditional tests have been developed to provide more precise and accurate measures of cognitive function. However, as noted before, the majority of cognitive test batteries employed by neuropsychologists largely consist of traditional paper-pencil tests (Rabin et al., 2014; Rabin et al., 2016). Therefore, the integration of novel methods into traditional assessments must be conducted in a discrete manner that preserves the standardized approach while offering incremental value to the test results. In this way, technological advances in neuroscience may help improve shortcomings of the ecological validity of EF test scores used in neuropsychological assessments (Chaytor et al., 2006; Kessels, 2019). Eye tracking has been gaining interest to understand the relation with daily tasks (Hayhoe & Ballard, 2005; Land, 2009; Ashraf, Sodergren, Mylonas, Sing, et al., 2018). Fixations and saccades have been shown to have utility in being able to understand actions and behaviours more precisely. Distinct patterns in eye movement are reported even in more common daily tasks, such Page 104 of 178 as reading (Smith et al., 2014) and walking while interacting with objects and people (Foulsham, 2015). Moreover, gaze and fixation patterns have been shown to predict level of mastery on a task (Sailer, et al. 2005) as well as evaluate progress and refine teaching methods in training clinicians (Ashraf et al., 2018) and pilots (Skyarekova & Skultety, 2019). While previous research has demonstrated the utility of eye tracking on routine daily tasks and more critical and demanding job tasks, there is scarce evidence of how it may help neuropsychologists predict functional outcomes for inpatients after a stroke. Exploring methods and data that could improve inferences and predictions of function after a stroke is important. The patient, family, and rehabilitation team often look to neuropsychologists to provide information regarding cognitive functions in relation to patient discharge planning, prognosis for recovery, and reintegration back into the home and community. That said, Study 3 aimed to expand on findings of previous research and Study 2 by investigating the concurrent and predictive validity of fixations and saccades during the cWCST for predicting functional outcome scores. The following hypothesis was proposed: 1. Total saccade and fixation counts as well as fixations on the cWCST AOIs (count and average time per fixation) will predict admission and discharge scores based on a measure of functional independence, above and beyond cWCST scores alone. 4.2 Methodology 4.2.1 Participants Study 3 included the same inpatient sample obtained for Study 2. 4.2.2 Materials and Procedures Functional Independence Measure. The Functional Independence Measure (FIM; Granger, Hamilton, Keith, & Zielezny, & Sherwin, 1986) assesses function on self-care, Page 105 of 178 continence, transfers, mobility, communication, and social cognition. The 18-items are rated on a Likert-type scale from 1 (total assistance required) to 7 (complete independence); total score ranges from 18 to 126. The FIM is one of the most frequently used measures for assessing rehabilitation outcome after stroke (Sangha, Lipson, Foley, Salter, Bhogal, et al., 2005) and a primary source of functional status data in Canadian Institute for Health Information data system. The FIM has demonstrated high internal consistency (α coefficient > .84) and high responsiveness to clinical change (standardized response mean [effect size] > 1.2) for inpatients after a stroke during rehabilitation (Hsueh, Lin, Jeng, & Hsieh, 2002). Data from Study 2, including participant demographics, cWCST test scores (categories achieved and perseverative errors), and eye tracking data (AOI fixation counts and average time per fixation, as well as total saccade count) were retained for Study 3. Occupational therapists (OT) assessed inpatients within 1-week of admission to the rehabilitation unit and again 1-2 weeks prior to discharge. Administration of the FIM was included in both assessments. Baseline and discharge FIM scores were collected retrospectively from the participant’s occupational therapy records. Change scores were calculated by subtracting the baseline from discharge scores. In cases with missing FIM scores, a psychometrist familiar with the FIM and blinded to the current study, retroactively calculated scores by reviewing OT progress notes and assessment reports within a 1-week period of the participant’s neuropsychological testing. Statistical Analyses Preliminary analyses included investigation of assumptions for the FIM scores, correlation analyses between cWCST scores, fixations (count and average time per fixation), and total saccade count with FIM scores. Correlation results helped to determine variables entered into subsequent Page 106 of 178 hierarchical linear regressions to investigate how fixations and saccades predicted FIM scores, above and beyond cWCST scores. 4.3 Results Preliminary Analyses We first examined if FIM admission, discharge, or change scores differed based on hemisphere of stroke using independent samples t tests with a Bonferroni correction (critical p value = .006). No significant differences emerged between hemisphere and FIM scores (all p values > .05). Results are displayed in Table 9. Page 107 of 178 Table 11 Baseline, Discharge, and Change Scores on the Functional Independence Measure for Inpatients After Stroke (n = 41) All Inpatients with Stroke (n = 41) Stroke Left Hemisphere (n = 18) Stroke Right Hemisphere (n = 23) FIM Baseline 87.1 (19.8) 90.8 (16.6) 84.3 (22.0) FIM Discharge 106.3 (14.2) 108.4 (13.2) 104.7 (15.0) FIM Change 19.2 (12.4) 17.6 (9.8) 20.4 (14.3) Note. There were no significant differences on FIM scores between left- and right-hemisphere strokes (all p values > .05). Page 108 of 178 Next, we examined correlations between FIM scores, total saccades, and fixations (i.e., total count, as well as count and average time per fixation on each AOI) from the cWCST. Results revealed weak to moderate correlations between average time per fixation on AOIs and FIM scores (r = -.03 to -.51). Notably, a significant correlation emerged between average time per fixation on key cards and FIM admission score (r = -.51, p = .001) and change in FIM score (r = .38, p = .02). Weak correlations were found between fixation count on AOIs and FIM scores (r = .00 to .19). Weak to moderate correlations were found between total saccade and fixation counts and FIM scores (r = .09 to -.33). Notably, a significant negative correlation emerged between total saccade count and FIM discharge score (r = -.33, p = .04). Next, we examined correlations between scores on the FIM and cWCST (perseverative errors, conceptual responses, trials administered, and categories achieved). Results revealed cWCST perseverative errors (T scores) and categories achieved were positively correlated with FIM scores at admission (r = .40, p = .009; r = .36, p = .02, respectively) and discharge (r = .42, p = .006, r = .37, p = .016, respectively). Previous findings demonstrated poor performance on the WCST was associated with poor outcomes after stroke (Greve et al., 1999). Accordingly, lower admission and discharge FIM scores were associated with more perseverative errors and fewer categories achieved on the cWCST. Correlation results are displayed in Table 10. Page 109 of 178 Table 12 Correlations Between Scores on the Functional Independence Measure and Computerized Wisconsin Card Sorting Test, and Eye Tracking Data FIM Admission FIM Discharge FIM Change Key cards Fix Count Average Fixation -.19 -.51* -.19 -.27 .19 .38* Workspace Fix Count Average Fixation .00 -.12 -.12 -.07 .03 -.04 Stimulus Fix Count Average Fixation .03 -.07 -.08 -.09 -.10 -.16 Total Fixation -.16 -21 .15 Total Saccades -.18 -.31 .09 cWCST Categories .25 .37* -.22 cWCST PE (T) .37* .42* -.17 cWCST NPE (T) -.13 -.04 .12 cWCST CLR (T) .09 .13 -.05 cWCST Trials (raw) -29 -.25 .23 Note. T = standardized T-score based on age and education. FIM = Functional Independence Measure. cWCST PE = Perseverativee Errors (higher score indicates fewer errors), NPE = non-perseverative errors, CLR = conceptual level responses, Trials = trials administered (max 128). Average Fixation = average amount of time (ms) during each fixation on the AOI. * p < .05; significant correlations are displayed in bold font. Page 110 of 178 Hypothesis #1: Fixations on AOIs from the cWCST will predict scores on the FIM during rehabilitation, above and beyond cWCST scores. Three hierarchical regressions were computed, based on the results of the preliminary correlation analyses, noted above, to investigate the prediction of FIM scores from cWCST scores and eye tracking. First, FIM admission score was examined by entering cWCST PE and categories achieved in the first level and average time per fixation on key cards into the second level. The first model of cWCST scores was significant in predicting FIM admission, F(2, 34) = 3.93, p = .03, R2 = .19 (adjusted R2 = .14). Model 2, addition of the average time per fixation on key cards, led to an increase in R2 of 19%, representing a significant change from Model 1, F(1, 33) = 10.16, p = .003, R2 = .38 (adjusted R2 = .32). Average time per fixation on key cards was a significant predictor of Model 2 (β = -0.44, p = .03) and cWCST categories achieved approached significance (β = 0.28, p = .07). Second, FIM discharge score was examined using the same models (cWCST and eye tracking) as above. The first model was significant in predicting FIM discharge, F(2, 34) = 3.81, p = .03, R2 = .18 (adjusted R2 = .14). Model 2 accounted for an additional 3% of variance and was not a significant improvement over Model 1, F(1, 33) = 1.10, p = .30, R2 = .21 (adjusted R2 = .14). No significant predictors emerged in the final model. Third, we examined change in FIM scores from baseline to discharge - representing change in functional independence during rehabilitation. Model 1 was not significant in predicting FIM change scores, F(2, 34) = 0.80, p = .46, R2 = .05 (adjusted R2 = .00). Model 2 accounted for an additional 25% of variance and was a significant improvement over Model 1, F(1, 33) = 11.49, p = .002, R2 = .29 (adjusted R2 = .23). Moreover, average time per fixation on key cards was a Page 111 of 178 significant predictor in the model (β = 0.50, p = .002). Results of the hierarchical regression analyses are displayed in Tables 11a-c. Page 112 of 178 Table 13a Multiple Hierarchical Regression Predicting Functional Independence Measure Admission Scores (N = 41) FIM Admission Score Model 1 Model 2 Variable B β B β Constant 59.79 -- 99.98 -- cWCST PE (T) 0.41 0.27 0.28 0.19 cWCST Cat (raw) 2.02 0.25 2.34 0.28 Key Card Avg Fix -- -- -0.12 -0.44** R2 (Adjusted) / F .19(.14) / 3.93* .38(.32) / 6.71** ΔR2 / F for ΔR2 -- .19 / 10.16** Note. cWCST PE = perseverativee error T scores; cWCST Cat = number of categories completed; Key Card Avg Fix = Average time per fixation on keycard AOI. * = p < .05, ** < .01. Page 113 of 178 Table 13b Multiple Hierarchical Regression Predicting Functional Independence Measure Discharge Scores (N = 41) FIM Discharge Score Model 1 Model 2 Variable B β B β Constant 86.88 -- 97.53 -- cWCST PE (T) 0.29 0.19 0.26 0.24 cWCST Cat (raw) 1.40 0.24 1.48 0.25 Key Card Avg Fix -- -- -0.03 -0.17 R2 (Adjusted) / F .18(.14) / 3.81* .21(.14) / 2.91* ΔR2 / F for ΔR2 -- .02 / 1.10 Note. cWCST PE = perseverative error T scores; cWCST Cat = number of categories completed; Key Card Avg Fix = Average time per fixation on keycard AOI. * = p < .05, ** < .01. Page 114 of 178 Table 13c Multiple Hierarchical Regression Predicting Functional Independence Measure Change Scores (N = 41) FIM Change Score Model 1 Model 2 Variable B β B β Constant 27.40 -- -1.21 -- cWCST PE (T) -0.12 -0.13 -.03 -0.03 cWCST Cat (raw) -0.64 -0.12 -0.87 -0.17 Key Card Avg Fix -- -- 0.09 0.50** R2 (Adjusted) / F .04(.00) / 0.80 .29(.23) / 4.53** ΔR2 / F for ΔR2 -- .25 / 11.49** Note. cWCST PE = perseverative error T scores; cWCST Cat = number of categories completed; Key Card Avg Fix = Average time per fixation on keycard AOI. * = p < .05, ** < .01. Page 115 of 178 Discussion Study 3, Hypothesis #1 Results of Study 3 provided partial evidence for the ecological validity of fixations from the cWCST. Specifically, findings suggested that function on admission and improvement during rehabilitation were predicted by average fixations on key cards, above and beyond cWCST scores. These findings add to the evidence provided by Study 1 and 2 that demonstrated fixations (dwell time, average time per fixation, and/or fixation count) differentiated between cognitive load groups (Study 1), predicted cWCST scores (Study 1 and 2), were influenced by response-type (conceptual versus trial-and-error; Study 2), and differentiated between healthy controls and inpatients recovering from a stroke (Study 2). Specifically, results of Study 3 expanded on these findings by providing supporting evidence for the concurrent validity (FIM admission scores) as well as predictive validity (FIM change score) of fixations on the cWCST. Notably, FIM admission and discharge scores were also predicted by a combination of cWCST number of categories completed and perseverative errors. Better performance (fewer perseverative errors and more categories achieved) on the cWCST was associated with higher admission and discharge FIM scores. Findings of Study 3 were also in keeping with previous studies that have shown fixations predict basic daily activities (Foulsham, 2015; Sailer et al., 2005), mediate contemplation of action into behavior (Bulling & Wedel, 2019; Land, Mennie, & Rusted, 1999), reading (Bax, 2013), level of mastery with more complex motor tasks (Vickers, 1992; Land & McLeod, 2000; Richardson & Spivey, 2004) and complex reasoning tasks (Smith et al., 2014; Thomas & Lleras, 2007). Moreover, current findings support previous studies demonstrating that eye tracking can help to detect functional decline in patients with cognitive impairment (Seligman & Giovannetti, 2015). Longer time per fixation has been shown to differentiate high performers from low performers on visual analytic problems (Hu et al., 2017) with attention being directed toward relevant solution-Page 116 of 178 information (He, Sui, Chaparro, & Strohl, 2014). Here, we found that better recovery of function (higher FIM change scores) was associated with longer average time per fixation on the key cards of the cWCST. Unexpectedly, average time per fixation was also negatively correlated with, and predicted, FIM admission scores. Thus, longer average fixation time was associated with greater initial impairment (low FIM score) and better recovery (larger change in FIM score), whereas shorter average fixation time was associated with lower initial impairment (high FIM score) and less recovery over time. This finding may be due to inpatients who present with lower FIM scores experiencing relatively greater cognitive load, which has been associated with longer fixation times (Vlaskamp & Hooge, 2006; Jacob & Karn, 2003). In such cases, inpatients who initially presented with low FIM scores also had the potential for the highest change in FIM scores (i.e. most recovery of function) and may have had a longer stay on the unit to recover. In contrast, inpatients admitted to rehabilitation with relatively higher functional scores may have had less room to improve or may have improved on specific (but critical) activities that were only reflected as small or minor changes on the FIM (e.g., climbing stairs, dressing) that allowed them to be discharged safely from hospital. As well, scores on the FIM at admission and at discharge were predicted by perseverative errors and categories completed on the cWCST. These results suggest the cWCST is predictive of baseline function after stroke and outcome at discharge but is less sensitive to change in function during rehabilitation. Taken together, select cWCST scores and fixations were significantly correlated and predicted functional outcomes and recovery after stroke. Moreover, average time per fixation on cWCST key cards were significant predictors of initial functional status and recovery during rehabilitation above and beyond cWCST scores, suggesting there is benefit to including eye tracking during testing. Page 117 of 178 4.4 Study 3 General Discussion The aim of Study 3 was to investigate the ecological validity of eye tracking data from the cWCST. It was important to address this topic, because neuropsychological assessments are often asked to infer function and behaviour in everyday living for patients. While previous cohort studies have benefitted from the use of contemporary technology to augment the process approach during neuropsychological assessments (Au et al., 2017), less is known about the benefits for inpatients recovering after a stroke. Findings of Study 3 demonstrated that eye tracking fixations during the cWCST predicted scores on the FIM, an assessment of independence on basic daily tasks such as self-care, locomotion, and transfers - all tasks that require visual attention to various stimuli in the environment. Specifically, average time per fixation predicted functional independence at baseline during rehabilitation after a stroke as well as the change from baseline to discharge (recovery). This proposed benefit of eye tracking with the cWCST also supported recommendations by Nyhus and Barcelo (2009) that suggested the integration of contemporary technology to improve the validity of scores from the WCST. Study 3 extended the results of Study 1 and 2 by demonstrating that select fixations on the cWCST predicted functional outcomes after a stroke, above and beyond scores on the cWCST. Overall, findings support eye tracking as a potential method to augment test results and improve the ecological validity of neuropsychological assessments for inpatients recovering from a stroke. Page 118 of 178 Chapter 5 Conclusion: Overall Discussion, Clinical Implications, Limitations, and Future Research 5.1 Overall Discussion and Clinical Implications Neuropsychological assessments are considered the ‘gold standard’ for the assessment of cognitive functions (Kessels, 2019; Lezak, 2012). As a discipline, neuropsychology strives to improve the tools and methods used for clinical decision-making (Strauss, 2006). One example of advancing methodology is the Boston Process Approach, which has helped elucidate test performance by examining test-taking behaviour and error analysis (Milberg et al., 2009; Kaplan, 1988). This approach has provided improvement in diagnostic accuracy and differential diagnosis (Lamar et al., 2010). More recently, advances in computer and neuroscience technology have provided powerful methods to augment qualitative approaches by quantifying behaviour and testing methods (Poreh, 2006; Au et al., 2017). While neuropsychology has lagged behind other disciplines for integrating contemporary technology into assessments (Jagaroo, 2009; Parsey & Schmitter-Edgecombe, 2012; Parsons, 2016; Miller & Barr, 2017), interest has been increasing over the last decade (Kessels, 2019; Parsons & Dufferson, 2019; Casaletto & Heaton, 2017; Miller & Barr, 2017; Au et al., 2017; Parsons, McMahon, & Kane, 2016; Poreh, 2006) with computerized tests becoming more mainstream among early career neuropsychologists (Rabin et al., 2014), and computers generally becoming more prevalent for scoring and interpreting measures (Parsey & Schmitter-Edgecombe, 2013). Given that contemporary technology (e.g., computers, laptops, cell phones) is ubiquitous in our society, it is reasonable for neuropsychologists to adopt assessment methods that reflect the status quo of daily activities. Page 119 of 178 This dissertation was in accordance with the exponential increase in interest across disciplines and fields of study for utilizing eye tracking to augment our understanding and assessment of cognitive abilities (Salvucci & Goldberg, 2000; Hodgson et al., 2000; Jacob & Karn, 2003; Leigh & Kennard, 2004; Richardson & Spivey, 2004; Thomas & Lleras, 2007; Van Gog & Scheiter, 2009; Nisiforou & Laghos, 2013; Bax, 2013; Bochynska & Laeng, 2015), engagement in daily activities (Hayhoe & Ballard, 2005; Sailer et al., 2005; Land, 2009; Gidlof, et al., 2013; Foulsham, 2015), and clinical diagnoses (Rayner, 1998; Amador et al., 2006; Heitger et al., 2008; Benson et al., 2012; Seligman & Giovannetti, 2015; Upshaw et al., 2019). However, most studies have utilized novel cognitive/eye tracking paradigms or require bulky equipment that could interfere with standardized test procedures during neuropsychological testing. Despite the growing use of eye trackers in clinical and research settings, few studies have specifically investigated the integration of eye tracking in real-time during neuropsychological assessment. Further research has been necessary to explore this type of supplementary method and specifically examine how eye tracking can augment our understanding of cognitive functions and daily activities after a stroke. Moreover, digital technology has allowed previously qualitative information to be captured and analyzed for greater precision with neuropsychological assessments (Kessels, 2019). This can be helpful for neuropsychologists who are asked to assess inpatients after a stroke and comment on cognitive functions in relation to independence and treatment/discharge planning. Neuropsychology has maintained its position as the gold standard for assessing cognitive functions by utilizing tests and methods that are vetted and empirically validated. In keeping with this, the current dissertation investigated various aspects of validity for saccades and fixations during cognitive testing in effort to provide a novel source of qualitative information that could help augment test results and inform clinical decision-making and treatment planning. Page 120 of 178 With a process approach in mind, this dissertation addressed the validity of eye tracking data during the cWCST based on the following topics: (1) whether fixations and saccades differ across cognitive load on the cWCST (construct validity); (2) whether fixations and saccades predict scores on the cWCST and manually administered cognitive tests (criterion validity); (3) whether fixations and saccades on the cWCST differentiate between healthy individuals and inpatients after a stroke (criterion validity); (4) if saccades and fixations differ based on response-type on the cWCST (construct validity); and (5) how well fixations and saccades predict recovery of function during rehabilitation (ecological validity). Study 1 addressed points 1 and 2; Study 2 addressed points 2, 3, and 4; and Study 3 addressed point 5. The aims of these studies were in keeping with the growing literature on the use of eye trackers in clinical and research settings to examine cognition, function, and diagnoses, as well as the increasing interest and support of implementing contemporary technology into neuropsychological assessments. Aspects of validity, clinical implications, limitations, and directions for future research are discussed in further detail below. 5.1.1 Construct Validity of Eye Tracking Data from the cWCST Findings from Study 1 and 2 provided mixed, but overall supportive, evidence for the construct validity of eye tracking data during administration of the cWCST. In Study 1, we manipulated cognitive load and investigated the association of fixations and saccades with test scores on the cWCST as well as additional cognitive tests. Fixation dwell time on AOIs and saccade count predicted select scores on the cWCST. Moreover, cognitive load affected fixations and saccades during the cWCST, such that low cognitive load resulted in significantly shorter fixation dwell times, and fewer saccades, compared to higher levels of cognitive load. In Study 2, response types (i.e., 10 consecutive trials of conceptual versus non-conceptual/trial-and-error) Page 121 of 178 showed distinct differences in fixation count on AOIs, specifically on previously placed cards (workspace) and key cards. Across both Study 1 and 2, convergent-discriminant validity was mixed. Fixations correlated with select scores on the cWCST (as well as a visual EF composite score in Study 2). In contrast, correlations were weakly correlated with most other visual and verbal cognitive tests (or composite scores) in Study 1 and 2. This was in contrast to hypotheses for eye tracking data to demonstrate stronger correlations with visual versus verbal tests. Alternatively, convergent-discriminant validity was supported in Study 1 and 2 based on fixations and saccades demonstrating stronger and more frequent correlations with cWCST scores compared to additionally administered cognitive test scores. This finding is reasonable given observations during a test (or activity) are most applicable to the test being administered and may or may not generalize to other cognitive tests. However, this did raise concern that eye tracking data obtained during administration of the cWCST may then not generalize to daily activities outside the test setting (examined more directly in Study 3). Overall, while there was general support for the construct validity of fixations and saccades across the studies of this dissertation, continued exploration is warranted with larger sample sizes that may allow for more sophisticated analyses (e.g., principle component analysis and confirmatory factor analysis). This would also provide further refinement of the otherwise large amount of eye tracking data available from the software into manageable and clinically meaningful information. 5.1.2 Criterion Validity of Eye Tracking Data from the cWCST To further explore the use of eye tracking in distinguishing between healthy individuals and inpatients after a stroke, Study 2 examined the criterion validity of fixations and saccades during the cWCST. Eye tracking data effectively predicted group membership. While previous studies have found fixations and saccades to be effective at differentiating between clinical Page 122 of 178 populations and healthy controls, these have often been conducted with paradigms such as pro- and anti-saccadic eye movements or novel cognitive test paradigms. Study 2 sought to integrate eye tracking into an established computerized cognitive test, the cWCST, with little to no interruption to standardized test procedures. Findings were in keeping with past research and demonstrated that real-time eye tracking, during the cWCST, was able to maintain standardized procedures and distinguish between healthy individuals and inpatients who had experienced a stroke. Specifically, having a stroke was associated with significantly more total saccade count during testing, as well as fixations on AOIs placed on the cWCST, compared to age-matched healthy individuals. Moreover, fixations and saccades on the cWCST AOIs predicted test performance , supporting the concurrent validity (subtype of criterion validity) of eye tracking data during cognitive testing. Note that this finding was also replicated across different population samples between Study 1 and 2, providing more robust evidence supporting the criterion validity. 5.1.3 Ecological Validity of Eye Tracking Data from the cWCST The final focus of this dissertation was to examine the ecological validity of eye tracking during neuropsychological assessment of inpatients who had experienced a stroke. Specifically, Study 3 investigated how well fixations and saccades from the cWCST, a well-established test of EF, predicted functional independence based on clinician ratings of the FIM for inpatients upon admission to, and discharge from, a rehabilitation unit. This final phase of the dissertation was important as neuropsychological assessments are often conducted to help infer function in daily life and predict outcomes. Moreover, there has been growing interest in implementing contemporary technology to augment neuropsychological assessments (Parsons, 2016; Miller & Barr, 2017; Casaletto & Heaton, 2017; Au et al., 2017; Kessels, 2019). Study 2 and 3 provided evidence that, while protecting the integrity of standardized testing of the cWCST, eye tracking Page 123 of 178 improved the prediction of functional independence for inpatients recovering from a stroke, above and beyond cWCST scores. Notably, across Study 1 and 2, there was also evidence supporting the external validity of eye tracking from the experimental to clinical setting. Specifically, fixations and saccades predicted scores on the cWCST and discriminated between groups (manipulation of cognitive load in Study 1 and healthy versus stroke inpatient groups in Study 2). In an effort to investigate the ecological validity of fixations and saccades from the cWCST, functional assessments were completed primarily by occupational therapists (and some by a psychometrist) blind to the study. Overall, this dissertation, and specifically Study 3, provided preliminary evidence suggesting that specific fixation data can help predict basic daily function for inpatients recovering from a stroke. However, further investigation is warranted for other daily activities such as work or school performance that often relies on use of computers and technology. As noted above, technology is ubiquitous in our society and therefore eye tracking during computerized tests may serve as an especially useful method of analyzing visual attention on specific activities; thus, providing greater precision with neuropsychological assessment, as noted by Kessels (2019). 5.1.4 Clinical Implications The current dissertation aimed to provide preliminary investigations, using a process approach, of using eye tracking data with cognitive testing during neuropsychological assessments. Overall, findings generally supported multiple aspects of validity for eye tracking data during completion of the cWCST. As well, results of this dissertation reflect the growing interest of employing technological advances of cognitive neuroscience into cognitive test settings to leverage their capabilities for potentially improving ecological validity and assessment precision (Bilder, 2011; Parsons, 2016; Kessels, 2019). While concerns have been raised about the limitations and drawbacks of replacing traditional methods of testing with contemporary Page 124 of 178 technology, the general conclusion here is that eye tracking should not supplant traditional paper-pencil or computerized testing but be used in addition to traditional testing methods. Previous eye tracking studies have provided evidence of being associated with cognitive abilities, daily function, and diagnoses of neurological disorders and brain injury. This dissertation adds to the growing body of research by providing evidence for the validity of eye tracking data during administration of the cWCST to enrich neuropsychological assessments of inpatients recovering from a stroke. Current findings (see Study 3) reflected the impressions of Poreh (2006) that indicated eye tracking allows for quantification of overt visual attention to assist in making inferences about daily functioning. Inclusion of eye tracking may also be directive in providing more targeted interventions by examining overt visual attention during test taking. For example, eye tracking allows clinicians to determine whether responses are made with or without attending to other options, such as performances seen in individuals with visuospatial neglect (Upshaw et al., 2019). As well, directing eye movements to goal-relevant information or in distinct sequences can help with conceptualization on problem tasks (Grant & Spivey, 2003; Rouinfar et al., 2014). To better understand planning, organization, and decision-making, real-time eye tracking provides opportunity to observe what is being attended to, in what order, and for how long. Such processes often happen too quickly for introspection, and interruption from an examiner to inquire via patient self-reflection may influence subsequent test behaviour (Poreh, 2006). In addition to observation and self-reports, previous studies have shown eye tracking can provide further insight to task engagement and learning styles (Scheiter & Van Gog, 2009; Land, 2009; Foulsham, 2015). Further, directing overt visual attention has been shown to facilitate learning and problem solving on visual tasks (Grant & Spivey, 2003; Thomas & Lleras, 2007; Johansson & Johansson, 2014; Page 125 of 178 Brunye et al., 2019). Following a stroke, rehabilitation teams may inquire about learning capacity of patients who would benefit from implementing strategies to compensate for functional deficits. In regard to learning, Study 2 demonstrated differing patterns of fixations between conceptual and non-conceptual responses. This finding may open up possible avenues of investigation for using eye tracking data with cognitive test results to help to inform opinion and recommendations on learning and implementation of new strategies for the patient. Interestingly, eye tracking technology is rapidly evolving, and even over the course of the current dissertation, new models of wireless eyewear have become available that permit real-time eye tracking during daily activities (e.g., Tobii, iMotions, Pupil Labs). Certainly, with the vast amount of technology that is becoming available, there is great potential to understand and elucidate the complexities of the mind and cognitive functions. They also provide opportunity to examine and adapt current theories about cognition and behaviour. New methods of investigation, such as eye tracking, allow researchers and clinicians to obtain an understanding of brain-behaviour relationships that was not available at the level it is now. For neuropsychology, the approach to diagnosis, assessment, and intervention using newly available technology will be accompanied by many opportunities as well as challenges (Bilder et al., 2011). However, as with many newly available products on the market, research regarding the analytic challenges and empirical evidence to support their claims of efficacy is lacking (MacInnes, Iqbal, Pearson, & Johnson, 2018). The influx of gadgets and technology available on the market warrant investigation and empirical evidence to guide the understanding and utility of contemporary technology and to integrate it into research and clinical settings. Regarding the brain-behaviour relationship, neuropsychology is uniquely positioned to examine many of these claims, in both settings. Page 126 of 178 5.2 Limitations of Dissertation While the current study has many strengths and builds upon past research, there are limitations to the current dissertation that must be addressed. First, the primary focus was investigating aspects of validity for integrating eye tracking with the cWCST without additional examination of the reliability to provide a fuller account of the psychometric properties. While the external validity was assessed, to some extent, here by integrating eye tracking with the cWCST across samples of three different populations (i.e., university students, healthy elderly, and inpatients with stroke), reliability was not a primary aim of the current research project but an important topic for future research. Notably, previous studies have demonstrated good to excellent test-retest reliability of eye tracking paradigms for cognitive assessments (Plomecka, Baranczuk-Turska, Pfeiffer, & Langer, 2020; Howell, Brilliant, Master, & Meehan, 2020). Specific to the current methods, future studies may consider examining split-half reliability by investigating eye tracking measures on AOIs during consecutively correct responses across multiple categories on the cWCST. One would expect that similar results would be obtained during these ten conceptual responses for each category, and similarly, that comparable results would be obtained during non-conceptual responses when trial-and-error is occurring. Nevertheless, direct investigation of the reliability with the current paradigm is warranted. Second, results of Study 2 and 3 were based on a convenience sample of inpatients who experienced a stroke. As previously noted, stroke can have deleterious effects on oculomotor movement and, in more extreme cases, level of arousal. Findings here, and integration of eye tracking, may be only generalizable to those inpatients that retain control of eye movements and do not experience double- or blurred vision that would interfere with viewing visual stimuli on a computer. Moreover, as with most patients who engage in neuropsychological testing, a certain Page 127 of 178 degree of cognitive capacity must be demonstrated for the patient to understand and follow instruction. In this case, all patients were able to complete a comprehensive battery of cognitive testing as part of their evaluation. That said, the same test battery was not used consistently in Study 2, necessitating post-hoc computation of composite scores. While this provided common scores for cognitive domains, despite differing test batteries, it also limits the generalization of findings. While the cWCST has been reported to be the most frequently utilized test of EF (Rabin et al., 2014), further research is necessary to investigate the association of fixations and saccades with other computerized cognitive tests. Third, FIM scores in Study 3 were primarily completed by occupational therapists who were blind to the current study; however, in some cases this data was unavailable (e.g., admission or discharge scores not completed). In such cases, a psychometrist familiar with FIM scoring procedures, and also blind to the study, completed the FIM scoring based on retrospective review of the patient’s rehabilitation chart. While this departs from the standardized approach of having occupational therapists complete all FIMs with the patients who participated, scoring was based on objective data recorded in the charting. Regardless, this does not preclude the potential for inter-rater differences. Fourth, results of the current study are based upon integration of eye tracking with a specific test of EF. The degree of specification with the current study (e.g., eye tracking hardware/software, test choice, AOI placement, eye tracking data selected for analysis, video recording parameters) may limit generalization to future studies and populations. Change to any of these may influence outcomes found; thus, replication and extension is warranted. However, it is important to point out that despite the novel methodology of this dissertation, results were Page 128 of 178 generally in keeping with previous findings of eye tracking and cognitive abilities, including problem solving. 5.3 Areas for Future Research The integration of contemporary technology into neuropsychological evaluations open an array of possibilities in how to assess cognitive function. Eye tracking alone has a multitude of opportunities, such as developing a better understanding between cognitive function and behaviour. Thus, future research is warranted to establish further empirical evidence for the clinical application of eye tracking and to direct how findings can be implemented into practice for neuropsychology. However, there is much work to be done to bridge the gap between the promising field of cognitive neuroscience in eye tracking and the clinical setting of neuropsychological assessments. While neuropsychologists have generally been relatively hesitant to implement newer technology (e.g., virtual reality, eye tracking, computerized assessments) into practice, there has been a steady increase in interest and research, and to a lesser extent, clinical application over the past two decades. For example, a virtual reality-based version of the Multiple Errands Test (MET), a performance-based assessment that examines executive functioning based on tasks of daily living, was designed to make testing more accessible across settings (Raspelli, Pallavicini, Carelli, Morganti, Poletti, Corra, et al., 2012). This addressed some of the practical shortcomings raised about the original MET (see Logie et al., 2011), while still being able to differentiate between individuals with brain injury and healthy controls. As well, further empirical investigation regarding the feasibility of implementing eye tracking into the clinical setting for neuropsychological assessment would be of interest. Relative to the current studies, integration of eye tracking appeared seamless in that standardized administration appeared to be upheld with Page 129 of 178 little to no obvious interruption during testing. As well, the graphic interface provides ease of AOI placement/configuration (e.g., key cards, previously placed cards, and stimulus card) that reduces chance of error related to AOI proximity or small size (see Orquin et al., 2015). Moreover, the test provides a convenient start point (trial 1) and end point (screen that displays “end of test”) as natural markers to crop video-recordings with the eye tracker – convenient for future studies wanting to replicate the methodology. Regarding the present dissertation, future studies should consider the use of eye tracking with other visual-based computerized cognitive tests to determine the relationship between eye movements and test scores. Moreover, results of Study 1 suggest more challenging methods to manipulate cognitive load should be included in future studies with experimental designs. For example, cognitive tests used in clinical settings often employ backward counting tasks as a method to interfere or distract from target information (e.g., California Verbal Learning Test - Short Form, Auditory Consonant Trigrams) and may provide a more effective manipulation for cognitive load. As well, pupil size may serve as a marker for emotional arousal during testing (Wang, Baird, Huang, Coutinho, Brien, & Munoz, 2018; Bradley, Miccoli, Escrig, & Lang, 2008). This can have important implications for assessments because emotional affect can influence cognitive functions, such as attention, learning, and memory (Tyng, Amin, Saad, & Malik, 2017), and therefore, cognitive test results. Additionally, effort testing has become an increased area of interest, specifically in the context of forensic neuropsychology (Greher & Wodushek, 2017). Subtle behavioural cues, such as patterns of gaze during testing, may provide an additional measure of effort. Eye tracking could be embedded into most computerized tests that require attending to visual stimuli and used to explore difficulties with response inhibition, working memory, and/or questionable effort (e.g., fixations consistently towards correct responses before Page 130 of 178 making an incorrect response). Last, recent research has translated traditional cognitive tests, such as the Trail Making Test, to allow completion using eye gaze (Hicks et al., 2013). This allows testing to be completed by patients who may suffer from impaired motor/sensory function due to injury or disease process (e.g., ALS) and are otherwise unable to engage in tests with a motor component, such as the TMT. Similar methods could be used for the cWCST to circumvent physical limitations that preclude testing. Moreover, certain phenomena resulting from brain injury, such as hemispatial neglect, appear to hold promise for examination with eye tracking to better elucidate this complex issue that can be detrimental to recovery and integration back into the home and community. Eye tracking shows considerable potential in experimental and clinical practice to advance our understanding of brain-behaviour relationships. As Bilder et al (2011) concluded, neuropsychology has evolved over the decades since its inception; however, this has usually been in response to other healthcare professions leveraging contemporary technologies. Neuropsychology is aptly positioned to integrate contemporary technologies, like eye tracking, into practice and to be a leader in guiding their developments. It is hoped that the results of the current dissertation might provide additional avenues for further exploration. Page 131 of 178 References Abdul-sattar, A. B., & Godab, T. (2013). Predictors of functional outcome in Saudi Arabian patients with stroke after inpatient rehabilitation, 33, 209–216. http://doi.org/10.3233/NRE-130947 Abel, L. A., & Douglas, J. (2007). Effects of age on latency and error generation in internally mediated saccades. Neurobiology of Aging, 28(4), 627-637. Aho, K., Harmsen, P., Hatano, S., Marquardsen, J., Smirnov, V. E., & Strasser, T. (1980). Cerebrovascular disease in the community: results of a WHO collaborative study. Bulletin of the World Health Organization, 58(1), 113. Alvarez, J. A., & Emory, E. (2006). Executive function and the frontal lobes: a meta-analytic review. Neuropsychology review, 16(1), 17-42. Amador, S. C., Hood, A. J., Schiess, M. C., Izor, R., & Sereno, A. B. (2006). Dissociating cognitive deficits involved in voluntary eye movement dysfunctions in Parkinson’s disease patients. Neuropsychologia, 44(8), 1475–1482. http://doi.org/10.1016/j.neuropsychologia.2005.11.015 Amieva, H., Phillips, L., & Della Sala, S. (2003). Behavioural dysexecutive symptoms in normal aging. Brain and cognition, 53(2), 129-132. Anastasi, A., & Urbina, S. (1997). Psychological testing. Prentice Hall:Pearson Education. Anderson, P. J. (2008). Towards a developmental model of executive function. Executive functions and the frontal lobes: A lifespan perspective, 3, 21. Page 132 of 178 Arboix, A., Arbe, G., García-Eroles, L., Oliveres, M., Parra, O., & Massons, J. (2011). Infarctions in the vascular territory of the posterior cerebral artery: clinical features in 232 patients. BMC research notes, 4(1), 329. Armstrong, T., & Olatunji, B. O. (2009). What they see is what you get: Eye tracking of attention in the anxiety disorders. Psychological Science Agenda, 23(3). Arrieux, J. P., Cole, W. R., & Ahrens, A. P. (2017). A review of the validity of computerized neurocognitive assessment tools in mild traumatic brain injury assessment. Concussion, 2(1), CNC31. Atkinson, M. A., Simpson, A. A., & Cole, G. G. (2018). Visual attention and action: How cueing, direct mapping, and social interactions drive orienting. Psychonomic bulletin & review, 25(5), 1585-1605. https://doi.org/10.3758/s13423-017-1354-0 Au, R., Piers, R. J., & Devine, S. (2017). How technology is reshaping cognitive assessment: Lessons from the Framingham Heart Study. Neuropsychology, 31(8), 846. Axelrod, B. N., Goldman, R. S., Heaton, R. K., Curtiss, G., Thompson, L. L., Chelune, G. J., & Kay, G. G. (1996). Discriminability of the Wisconsin Card Sorting Test using the standardization sample. Journal of clinical and experimental neuropsychology, 18(3), 338-342. Baddeley, A. D., & Hitch, G. J. (1994). Developments in the concept of working memory. Neuropsychology, 8(4), 485. Baddeley, A. D. (2001). Is working memory still working?. American Psychologist, 56(11), 851. Ballard, D. H., Hayhoe, M. M., & Pelz, J. B. (1995). Memory representations in natural tasks. Journal of Cognitive Neuroscience, 7(1), 66-80. Page 133 of 178 Barceló, F. (2001). Does the Wisconsin card sorting test measure prefrontal function. The Spanish journal of psychology, 4(1), 79-100. Barker-Collo, S., & Feigin, V. (2006). The impact of neuropsychological deficits on functional stroke outcomes. Neuropsychology Review, 16(2), 53–64. http://doi.org/10.1007/s11065-006-9007-5 Barker-Collo, S., Starkey, N., Lawes, C. M., Feigin, V., Senior, H., & Parag, V. (2012). Neuropsychological profiles of 5-year ischemic stroke survivors by Oxfordshire stroke classification and hemisphere of lesion. Stroke, 43(1), 50-55. Barnes, J. J., Dean, A. J., Nandam, L. S., O'Connell, R. G., & Bellgrove, M. A. (2011). The molecular genetics of executive function: role of monoamine system genes. Biological psychiatry, 69(12), e127-e143. Barr, W. B., & McCrea, M. (2001). Sensitivity and specificity of standardized neurocognitive testing immediately following sports concussion. Journal of the International neuropsychological Society, 7(6), 693-702. DOI: https://doi.org/10.1017/S1355617701766052 Bartzokis, G., Lu, P. H., Heydari, P., Couvrette, A., Lee, G. J., Kalashyan, G., ... & Mintz, J. (2012). Multimodal magnetic resonance imaging assessment of white matter aging trajectories over the lifespan of healthy individuals. Biological psychiatry, 72(12), 1026-1034. Bauer, R. M., Iverson, G. L., Cernich, A. N., Binder, L. M., Ruff, R. M., & Naugle, R. I. (2012). Computerized neuropsychological assessment devices: joint position paper of the American Page 134 of 178 Academy of Clinical Neuropsychology and the National Academy of Neuropsychology. Archives of Clinical Neuropsychology, 27(3), 362-373. Bax, S. (2013). The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing, 30(4), 441-465. Bednarik, R., & Tukiainen, M. (2007). Validating the restricted focus viewer: A study using eye-movement tracking. Behaviour research methods, 39(2), 274-282. Bell‐McGinty, S., Podell, K., Franzen, M., Baird, A. D., & Williams, M. J. (2002). Standard measures of executive function in predicting instrumental activities of daily living in older adults. International Journal of Geriatric Psychiatry, 17(9), 828-834. Bennett, T. L. (2001). Neuropsychological evaluation in rehabilitation planning and evaluation of functional skills. Archives of Clinical Neuropsychology, 16(3), 237-253. https://doi.org/10.1093/arclin/16.3.237 Benson, P. J., Beedie, S. A., Shephard, E., Giegling, I., Rujescu, D., & Clair, D. S. (2012). Simple viewing tests can detect eye movement abnormalities that distinguish schizophrenia cases from controls with exceptional accuracy. Biological psychiatry, 72(9), 716-724. Benton, A. L. (1969). Development of a multilingual aphasia battery: Progress and problems. Journal of the Neurological Sciences, 9, 39-48. Bergstrom, J. R., & Schall, A. (Eds.). (2014). Eye tracking in user experience design. Elsevier. Bilder, R. M. (2011). Neuropsychology 3.0: evidence-based science and practice. Journal of the international Neuropsychological Society, 17(01), 7-13. Bochynska, A., & Laeng, B. (2015). Tracking down the path of memory: eye scanpaths facilitate retrieval of visuospatial information. Cognitive processing, 16(1), 159-163. Page 135 of 178 Bogod, N. M., Mateer, C. A., & Macdonald, S. W. (2003). Self-awareness after traumatic brain injury: A comparison of measures and their relationship to executive functions. Journal of the International Neuropsychological Society: JINS, 9(3), 450. Bonato, M. (2012). Neglect and extinction depend greatly on task demands: a review. Frontiers in human neuroscience, 6, 195. Boone, M., Chillon, J. M., Garcia, P. Y., Canaple, S., Lamy, C., Godefroy, O., & Bugnicourt, J. M. (2012). NIHSS and acute complications after anterior and posterior circulation strokes. Ther Clin Risk Manag, 8, 87-93. Boone, K. B., Miller, B. L., Lesser, I. M., Hill, E., & D'Elia, L. (1990). Performance on frontal lobe tests in healthy, older individuals. Developmental Neuropsychology, 6(3), 215-223. Boot, W. R., Charness, N., Czaja, S. J., Sharit, J., Rogers, W. A., Fisk, A. D., ... & Nair, S. (2015). Computer proficiency questionnaire: assessing low and high computer proficient seniors. The Gerontologist, 55(3), 404-411. Bowling, A. C., Hindman, E. A., & Donnelly, J. F. (2012). Prosaccade errors in the antisaccade task: differences between corrected and uncorrected errors and links to neuropsychological tests. Experimental brain research, 216(2), 169-179. Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J. (2008). The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology, 45(4), 602-607. Brunyé, T. T., Drew, T., Weaver, D. L., & Elmore, J. G. (2019). A review of eye tracking for understanding and improving diagnostic interpretation. Cognitive research: principles and implications, 4(1), 1-16. Page 136 of 178 Bulling, A., & Wedel, M. (2019). Pervasive eye-tracking for real-world consumer behaviour analysis. In A Handbook of Process Tracing Methods (pp. 27-44). Routledge. Burgess, P. W., Alderman, N., Evans, J., Emslie, H., & Wilson, B. A. (1998). The ecological validity of tests of executive function. Journal of the international neuropsychological society, 4(06), 547-558. Burgess, P. W., Alderman, N., Forbes, C., Costello, A., LAURE, M. C., Dawson, D. R., ... & Channon, S. (2006). The case for the development and use of “ecologically valid” measures of executive function in experimental and clinical neuropsychology. Journal of the international neuropsychological society, 12(02), 194-209. Buchsbaum, B. R., Greer, S., Chang, W. L., & Berman, K. F. (2005). Meta‐analysis of neuroimaging studies of the Wisconsin Card‐Sorting task and component processes. Human brain mapping, 25(1), 35-45. https://doi.org/10.1002/hbm.20128 Buschke, H. (1973). Selective reminding for analysis of memory and learning. Journal of Verbal Learning and Verbal Behaviour, 12(5), 543-550. Cahn-Weiner, D. A., Boyle, P. A., & Malloy, P. F. (2002). Tests of executive function predict instrumental activities of daily living in community-dwelling older individuals. Applied neuropsychology, 9(3), 187-191. Cahn-Weiner, D. A., Ready, R. E., & Malloy, P. F. (2003). Neuropsychological predictors of everyday memory and everyday functioning in patients with mild Alzheimer's disease. Journal of geriatric psychiatry and neurology, 16(2), 84-89. Page 137 of 178 Casaletto, K. B., & Heaton, R. K. (2017). Neuropsychological assessment: Past and future. Journal of the International Neuropsychological Society: JINS, 23(9-10), 778. doi: 10.1017/S1355617717001060 Cassvan, A., Ross, P. L., Dyer, P. R., & Zane, L. (1976). Lateralization in stroke syndromes as a factor in ambulation. Archives of physical medicine and rehabilitation, 57(12), 583-587. Chan, E., Khan, S., Oliver, R., Gill, S. K., Werring, D. J., & Cipolotti, L. (2014). Underestimation of cognitive impairments by the Montreal Cognitive Assessment (MoCA) in an acute stroke unit population. Journal of the neurological sciences, 343(1), 176-179. Chan, R. C., Shum, D., Toulopoulou, T., & Chen, E. Y. (2008). Assessment of executive functions: Review of instruments and identification of critical issues. Archives of clinical neuropsychology, 23(2), 201-216. Chaytor, N., & Schmitter-Edgecombe, M. (2003). The ecological validity of neuropsychological tests: A review of the literature on everyday cognitive skills. Neuropsychology review, 13(4), 181-197. Chaytor, N., Schmitter-Edgecombe, M., & Burr, R. (2006). Improving the ecological validity of executive functioning assessment. Archives of clinical neuropsychology, 21(3), 217-227. Chiu, E., Wu, W., Hung, J., & Tseng, Y. (2017). Validity of the wisconsin card sorting test in patients with stroke. Disability and Rehabilitation, 1-5. http://dx.doi.org/10.1080/09638288.2017.1323020 Chlopan, B. E., Hagen, R. L., & Russell, E. W. (1990). Lateralized anterior and posterior lesions and performance on Digit Span and Russell's revision of the Wechsler Memory Scale. Journal of consulting and clinical psychology, 58(6), 855. Page 138 of 178 Collette, F., Van der Linden, M., Laureys, S., Delfiore, G., Degueldre, C., Luxen, A., & Salmon, E. (2005). Exploring the unity and diversity of the neural substrates of executive functioning. Human brain mapping, 25(4), 409-423. Collins, F. S., & Riley, W. T. (2016). NIH’s transformative opportunities for the behavioural and social sciences. Science Translational Medicine, 8, 366ed314 Conti, J., Sterr, A., Brucki, S. M. D., & Conforto, A. B. (2015). Diversity of approaches in assessment of executive functions in stroke: limited evidence?. Eneurologicalsci, 1(1), 12-20. Cooper, L., Gale, A., Darker, I., Toms, A., & Saada, J. (2009, February). Radiology image perception and observer performance: How does expertise and clinical information alter interpretation? Stroke detection explored through eye-tracking. In SPIE Medical Imaging (pp. 72630K-72630K). International Society for Optics and Photonics. Corbetta, M., Akbudak, E., Conturo, T. E., Snyder, A. Z., Ollinger, J. M., Drury, H. A., ... & Shulman, G. L. (1998). A common network of functional areas for attention and eye movements. Neuron, 21(4), 761-773. Coull, J. T. (1998). Neural correlates of attention and arousal: insights from electrophysiology, functional neuroimaging and psychopharmacology. Progress in neurobiology, 55(4), 343-361. https://doi.org/10.1016/S0301-0082(98)00011-2 Cowan, N. (2001). Metatheory of storage capacity limits. Behavioural and brain sciences, 24(01), 154-176. Page 139 of 178 Crawford, T. J., Higham, S., Renvoize, T., Patel, J., Dale, M., Suriya, A., & Tetley, S. (2005). Inhibitory control of saccadic eye movements and cognitive impairment in Alzheimer’s disease. Biological psychiatry, 57(9), 1052-1060. Crawford, T. J., Smith, E. S., & Berry, D. M. (2017). Eye gaze and aging: Selective and combined effects of working memory and inhibitory control. Frontiers in Human Neuroscience, 11, 563. Cullen, B., O’neill, B., Evans, J. J., Coen, R. F., & Lawlor, B. A. (2007). A review of screening tests for cognitive impairment. Journal of Neurology, Neurosurgery & Psychiatry, 78(8), 790-799. D’Elia, L., Satz, P., Uchiyama, C., & White, T. (1996). Color Trails Test. Odessa, FL: Psychological Assessment Resources. De Jong, T. (2010). Cognitive load theory, educational research, and instructional design: some food for thought. Instructional Science, 38, 105-134. doi:10.1007/s11251-009-9110-0 Delazer, M., Sojer, M., Ellmerer, P., Boehme, C., & Benke, T. (2018). Eye-tracking provides a sensitive measure of exploration deficits after acute right MCA stroke. Frontiers in neurology, 9, 359. Delis, D. C., Kramer, J. H., Kaplan, E., & Ober, B. A. (2000). California Verbal Learning Test – second edition. Adult version. Manual. San Antonio, TX: Psychological Corporation. Demakis, G. J. (2003). A meta-analytic review of the sensitivity of the Wisconsin Card Sorting Test to frontal and lateralized frontal brain damage. Neuropsychology, 17(2), 255. http://dx.doi.org/10.1037/0894-4105.17.2.255 Diamond, A. (2013). Executive functions. Annual review of psychology, 64, 135-168. Page 140 of 178 Dodrill, C. B. (1997). Myths of neuropsychology. The Clinical Neuropsychologist, 11(1), 1-17. Donders, J. (2019). The incremental value of neuropsychological assessment: A critical review. The Clinical Neuropsychologist, 34(1), 56-87. Dong, W., Yan, B., Johnson, B. P., Millist, L., Davis, S., Fielding, J., & White, O. B. (2013). Ischaemic stroke: the ocular motor system as a sensitive marker for motor and cognitive recovery. Journal of Neurology, Neurosurgery & Psychiatry, 84(3), 337-341. Doricchi, F., Guariglia, P., Figliozzi, F., Silvetti, M., Bruno, G., & Gasparini, M. (2005). Causes of cross-over in unilateral neglect: between-group comparisons, within-patient dissociations and eye movements. Brain, 128(6), 1386-1406. Duchowski, A. T. (2002). A breadth-first survey of eye-tracking applications. Behaviour Research Methods, Instruments, & Computers, 34(4), 455-470. Faria, C. D. A., Alves, H. V. D., & Charchat-Fichman, H. (2015). The most frequently used tests for assessing executive functions in aging. Dementia & Neuropsychologia, 9(2), 149-155. Faul, F., Erdfelder, E., Buchner, A., & Lang, A. G. (2013). G* Power Version 3.1. 7 [computer software]. Uiversität Kiel, Germany. Feigin, V. L., Parag, V., & Senior, H. (2010). Part 2 : Cognition and functional outcomes 5 years poststroke. Feldstein, S.N., Keller, F.R., Portman, R.E., Durham, R.L., Klebe, K.J., & Davis, H.P. (1999). A comparison of computerized and standard versions of the wisconsin card sorting test. The Clinical Neuropsychologist, 13(3), 303–313. Page 141 of 178 Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). “Mini-mental state”: a practical method for grading the cognitive state of patients for the clinician. Journal of psychiatric research, 12(3), 189-198. Fortuny, L. I. A., & Heaton, R. K. (1996). Standard versus computerized administration of the Wisconsin Card Sorting Test. The Clinical Neuropsychologist, 10(4), 419-424. Foulsham, T. (2015). Eye movements and their functions in everyday tasks. Eye, 29(2), 196-199. Foulsham, T., Walker, E., & Kingstone, A. (2011). The where, what and when of gaze allocation in the lab and the natural environment. Vision research, 51(17), 1920-1931. Franzen, M. D., & Wilhelm, K. L. (1996). Conceptual foundations of ecological validity in neuropsychological assessment. Friedman, L., Kenny, J. T., Jesberger, J. A., Choy, M. M., & Meltzer, H. Y. (1995). Relationship between smooth pursuit eye-tracking and cognitive performance in schizophrenia. Biological Psychiatry, 37(4), 265-272. Fujioka, T., Inohara, K., Okamoto, Y., Masuya, Y., Ishitobi, M., Saito, D. N., ... & Narita, K. (2016). Gazefinder as a clinical supplementary tool for discriminating between autism spectrum disorder and typical development in male adolescents and adults. Molecular autism, 7(1), 19. Gammeri, R., Iacono, C., Ricci, R., & Salatino, A. (2020). Unilateral spatial neglect after stroke: Current insights. Neuropsychiatric Disease and Treatment. Gates, N. J., & Kochan, N. A. (2015). Computerized and on-line neuropsychological testing for late-life cognition and neurocognitive disorders: are we there yet?. Current opinion in psychiatry, 28(2), 165-172. Page 142 of 178 Gazzaniga, M.S., Ivry, R.B., Mangun, G.R., 2014. Cognitive neuroscience: The biology of the mind. 4th ed. New York: W.W. Norton & Co. George, M. G., Tong, X., Kuklina, E. V., & Labarthe, D. R. (2011). Trends in stroke hospitalizations and associated risk factors among children and young adults, 1995–2008. Annals of neurology, 70(5), 713-721. doi: 10.1002/ana.22539 Gidlöf K, Wallin A, Dewhurst R, Holmqvist K (2013). Using eye tracking to trace a cognitive process: Gaze behaviour during decision making in a natural environment. Journal of Eye Movement Research 6(1):3, 1-14. Glozman, J. M. (2007). AR Luria and the history of Russian neuropsychology. Journal of the History of the Neurosciences, 16(1-2), 168-180. https://doi.org/10.1080/09647040600550368 Godefroy, O., Fickl, A., Roussel, M., Auribault, C., Bugnicourt, J. M., Lamy, C., ... & Petitnicolas, G. (2011). Is the Montreal cognitive assessment superior to the mini-mental state examination to detect poststroke cognitive impairment?. Stroke, 42(6), 1712-1716. Goldberg, J. H., & Wichansky, A. M. (2003). Eye tracking in usability evaluation: A practitioner's guide. In the Mind's Eye (pp. 493-516). North-Holland. Goldstein, K. (1942). Aftereffects of brain injuries in war: their evaluation and treatment. The application of psychologic methods in the clinic. Oxford, England: Grune & Stratton. Goldstein, G. (1996). Functional considerations in neuropsychology. In R. J. Sbordone & C. J. Long (Eds.), Ecological validity of neuropsychological testing (pp. 75-89). Delray Beach, FL, England: Gr Press/St Lucie Press, Inc. Page 143 of 178 Gontkovsky, S. T., Beatty, W. W., & Mold, J. W. (2004). Repeatable battery for the assessment of neuropsychological status in a normal, geriatric sample. Clinical Gerontologist, 27(3), 79-86. Gothe, N. P., Fanning, J., Awick, E., Chung, D., Wójcicki, T. R., Olson, E. A., … McAuley, E. (2014). Executive Function Processes Predict Mobility Outcomes in Older Adults. Journal of the American Geriatrics Society, 62(2), 285–290. http://doi.org/10.1111/jgs.12654 Granger, C. V., Hamilton, B. B., Keith, R. A., Zielezny, M., & Sherwin, F. S. (1986). Advances in functional assessment for medical rehabilitation. Topics in geriatric rehabilitation, 1(3), 59-74. Grant, E. R., & Spivey, M. J. (2003). Eye movements and problem solving: Guiding attention guides thought. Psychological Science, 14(5), 462-466. Grant, D. A., & Berg, E. (1948). A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of Experimental Psychology, 38(4), 404–411. https://doi.org/10.1037/h0059831 Greher, M. R., & Wodushek, T. R. (2017). Performance validity testing in neuropsychology: Scientific basis and clinical application—A brief review. Journal of Psychiatric Practice, 23(2), 134-140. Greiffenstein, M., Baker, J., & Gola, T. (1994). Validation of malingered amnesia measures with a large clinical sample. Psychological Assessment, 6, 218-224. Greve, K. W., Bianchini, K. J., Hartley, S. M., & Adams, D. (1999). The Wisconsin Card Sorting Test in stroke rehabilitation: Factor structure and relationship to outcome. Archives of Clinical Neuropsychology, 14(6), 497-509. Page 144 of 178 Gualtieri, C. T., & Johnson, L. G. (2006). Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Archives of Clinical Neuropsychology, 21(7), 623-643. Gutiérrez Pérez, C., Sävborg, M., Påhlman, U., Cederfeldt, M., Knopp, E., Nordlund, A., ... & Tarkowski, E. (2011). High frequency of cognitive dysfunction before stroke among older people. International journal of geriatric psychiatry, 26(6), 622-629. Haaland, K. Y., & Delaney, H. D. (1981). Motor deficits after left or right hemisphere damage due to stroke or tumor. Neuropsychologia, 19(1), 17-27. Hachinski, V., Iadecola, C., Petersen, R. C., Breteler, M. M., Nyenhuis, D. L., Black, S. E., ... & Vinters, H. V. (2006). National Institute of Neurological Disorders and Stroke–Canadian stroke network vascular cognitive impairment harmonization standards. Stroke, 37(9), 2220-2241. Halstead, W. C., Carmichael, H. T., & Bucy, P. C. (1946). Prefrontal lobotomy: A preliminary appraisal of the behavioural results. American Journal of Psychiatry, 103(2), 217-228. Han, D. Y., Anderson, A. J., Jones, J. E., Hermann, B. P., & Sattin, J. A. (2014). Neuropsychology in multidisciplinary stroke care: Clinical feasibility of the NINDS-CSN vascular cognitive impairment harmonization standards. International Scholarly Research Notices, 2014. Hanks, R., Rapport, L., Millis, S., & Deshpande, S. (1999). Measures of executive functioning as predictors of functional ability and social integration in a rehabilitation sample. Archives of Physical Medicine and Rehabilitation, 80, 1030-1037. Page 145 of 178 Hatano, S. (1976). Experience from a multicentre stroke register: a preliminary report. Bulletin of the World Health Organization, 54(5), 541. Harvey, P. D. (2012). Clinical applications of neuropsychological assessment. Dialogues in clinical neuroscience, 14(1), 91. Hayhoe, M., & Ballard, D. (2005). Eye movements in natural behaviour. Trends in cognitive sciences, 9(4), 188-194. He, J., Siu, C., Chaparro, B., & Strohl, J. (2014). Mobile. In Eye Tracking in User Experience Design (pp. 255-290). Morgan Kaufmann. Heaton, R. K., Chelune, G. J., Curtiss, G., Kay, G. G., & Talley, J. L. (1993). Wisconsin card sorting test. Psychological Assessment Resources. Hedna, V. S., Bodhit, A. N., Ansari, S., Falchook, A. D., Stead, L., Heilman, K. M., & Waters, M. F. (2013). Hemispheric differences in ischemic stroke: is left-hemisphere stroke more common?. Journal of Clinical Neurology, 9(2), 97-102. Heitger, M. H., Jones, R. D., & Anderson, T. J. (2008, August). A new approach to predicting postconcussion syndrome after mild traumatic brain injury based upon eye movement function. In Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE (pp. 3570-3573). IEEE. Heruti, R. J., Lusky, A., Dankner, R., Ring, H., Dolgopiat, M., Barell, V., ... & Adunsky, A. (2002). Rehabilitation outcome of elderly patients after a first stroke: effect of cognitive status at admission on the functional outcome. Archives of physical medicine and rehabilitation, 83(6), 742-749. Page 146 of 178 Hessels, R. S., Kemner, C., van den Boomen, C., & Hooge, I. T. (2016). The area-of-interest problem in eyetracking research: A noise-robust solution for face and sparse stimuli. Behaviour research methods, 48(4), 1694-1712. Heuer, S., & Hallowell, B. (2015). A novel eye-tracking method to assess attention allocation in individuals with and without aphasia using a dual-task paradigm. Journal of communication disorders, 55, 15-30. Hicks, S. L., Sharma, R., Khan, A. N., Berna, C. M., Waldecker, A., Talbot, K., & Turner, M. R. (2013). An eye-tracking version of the trail-making test. PloS one, 8(12), e84061. Hochstenbach, J., Mulder, T., van Limbeek, J., Donders, R., & Schoonderwaldt, H. (1998). Cognitive decline following stroke: a comprehensive study of cognitive decline following stroke. Journal of clinical and experimental neuropsychology, 20(4), 503-517. Hochstenbach, J. B., Anderson, P. G., van Limbeek, J., & Mulder, T. T. (2001). Is there a relation between neuropsychologic variables and quality of life after stroke?. Archives of physical medicine and rehabilitation, 82(10), 1360-1366. Hodgson, T. L., Bajwa, A., Owen, A. M., & Kennard, C. (2000). The strategic control of gaze direction in the Tower of London task. Journal of Cognitive Neuroscience, 12(5), 894-907. Horton Jr, A. M., & Reynolds, C. R. (2015). Ralph M. Reitan: Evidenced Based Before Evidence Based Was Cool. Archives of Clinical Neuropsychology, 30(8), 740-747. https://doi.org/10.1093/arclin/acv070 Howell, D. R., Brilliant, A. N., Master, C. L., & Meehan III, W. P. (2020). Reliability of objective eye-tracking measures among healthy adolescent athletes. Clinical Journal of Sport Medicine, 30(5), 444-450. Page 147 of 178 Hsueh, I. P., Lin, J. H., Jeng, J. S., & Hsieh, C. L. (2002). Comparison of the psychometric characteristics of the functional independence measure, 5 item Barthel index, and 10 item Barthel index in patients with stroke. Journal of Neurology, Neurosurgery & Psychiatry, 73(2), 188-190. Hu, Y., Wu, B., & Gu, X. (2017). An eye tracking study of high-and low-performing students in solving interactive and analytical problems. Journal of Educational Technology & Society, 20(4), 300-311. Husain, M., Mannan, S., Hodgson, T., Wojciulik, E., Driver, J., & Kennard, C. (2001). Impaired spatial working memory across saccades contributes to abnormal search in parietal neglect. Brain, 124(5), 941-952. Hüttermann, S., Noël, B., & Memmert, D. (2018). Eye tracking in high-performance sports: Evaluation of its application in expert athletes. International Journal of Computer Science in Sport, 17(2), 182-203. Ivnik, R. J., Malec, J. F., Smith, G. E., Tangalos, E. G., Petersen, R. C., Kokmen, E., & Kurland, L. T. (1992). Mayo's older Americans normative studies: updated AVLT norms for ages 56 to 97. The Clinical Neuropsychologist, 6(S1), 83-104. Jacob, R. J., & Karn, K. S. (2003). Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. Mind, 2(3), 4. Jagaroo, V. (2009). Neuroinformatics for neuropsychology. In Neuroinformatics for Neuropsychology (pp. 25-84). Springer US. Page 148 of 178 Jefferson, A. L., Paul, R. H., Ozonoff, A. L., & Cohen, R. A. (2006). Evaluating elements of executive functioning as predictors of instrumental activities of daily living (IADLs). Archives of clinical Neuropsychology, 21(4), 311-320. Jodzio, K., & Biechowska, D. (2010). Wisconsin card sorting test as a measure of executive function impairments in stroke patients. Applied neuropsychology, 17(4), 267-277. Johansson, R. (2013). Tracking the Mind's Eye: Eye movements during mental imagery and memory retrieval (Vol. 155). Johansson, R., & Johansson, M. (2014). Look here, eye movements play a functional role in memory retrieval. Psychological Science, 25(1), 236-242. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological review, 87(4), 329. Julayanont, P., Ruthirago, D., & DeToledo, J. C. (2016). Isolated left posterior insular infarction and convergent roles in verbal fluency, language, memory, and executive function. Proceedings (Baylor University. Medical Center), 29(3), 295. Kachouri, H., Laatar, R., Borji, R., Rebai, H., & Sahli, S. (2020). Using a dual‐task paradigm to investigate motor and cognitive performance in children with intellectual disability. Journal of Applied Research in Intellectual Disabilities, 33(2), 172-179. Kaplan, E. (1988). The process approach to neuropsychological assessment. Aphasiology, 2(3-4), 309-311. Karatekin C. Eye tracking studies of normative and atypical development. Developmental Review. 2007 doi: 10.1016/j.dr.2007.06.006 Page 149 of 178 Karnath, H. O., Niemeier, M., & Dichgans, J. (1998). Space exploration in neglect. Brain: a journal of neurology, 121(12), 2357-2367. Keith, R. A., Granger, C. V., Hamilton, B. B., & Sherwin, F. S. (1987). The functional independence measure. Adv Clin Rehabil, 1, 6-18. Keller, J., Gorges, M., Horn, H. T., Aho-Özhan, H. E., Pinkhardt, E. H., Uttner, I., ... & Lulé, D. (2015). Eye-tracking controlled cognitive function tests in patients with amyotrophic lateral sclerosis: a controlled proof-of-principle study. Journal of neurology, 262(8), 1918-1926. Kerchner, G. A., Racine, C. A., Hale, S., Wilheim, R., Laluz, V., Miller, B. L., & Kramer, J. H. (2012). Cognitive processing speed in older adults: relationship with white matter integrity. PloS one, 7(11), e50425. Kerkhoff, G., & Schenk, T. (2012). Rehabilitation of neglect: an update. Neuropsychologia, 50(6), 1072-1079. Kessels, R. P. (2019). Improving precision in neuropsychological assessment: bridging the gap between classic paper-and-pencil tests and paradigms from cognitive neuroscience. The Clinical Neuropsychologist, 33(2), 357-368. https://doi.org/10.1080/13854046.2018.1518489 Kibby, M. Y., Schmitter-Edgecombe M., & Long, C. J. (1998). Ecological Validity of Neuropsychological Tests: Focus on the California Verbal Learning Test and the Wisconsin Card Sorting Test. Archives of Clinical Neuropsychology, 13(6),523-534. Kissela, B. M., Khoury, J. C., Alwell, K., Moomaw, C. J., Woo, D., Adeoye, O., … Kleindorfer, D. O. (2012). Age at stroke: Temporal trends in stroke incidence in a large, biracial Page 150 of 178 population . Neurology, 79(17), 1781–1787. http://doi.org/10.1212/WNL.0b013e318270401d Kizony, R., Levin, M. F., Hughey, L., Perez, C., & Fung, J. (2010). Cognitive load and dual-task performance during locomotion poststroke: a feasibility study using a functional virtual environment. Physical therapy, 90(2), 252-260. Klein, C., Fischer, B., Hartnegg, K., Heiss, W. H., & Roth, M. (2000). Optomotor and neuropsychological performance in old age. Experimental brain research, 135(2), 141-154. Krauzlis, R. J., Goffart, L., & Hafed, Z. M. (2017). Neuronal control of fixation and fixational eye movements. Phil. Trans. R. Soc. B, 372(1718), 20160205. Kringle, E. A., Terhorst, L., Butters, M. A., & Skidmore, E. R. (2018). Clinical predictors of engagement in inpatient rehabilitation among stroke survivors with cognitive deficits: An exploratory study. Journal of the International Neuropsychological Society: JINS, 24(6), 572. Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ‐9: validity of a brief depression severity measure. Journal of general internal medicine, 16(9), 606-613. Kulke, L. V., Atkinson, J., & Braddick, O. (2016). Neural differences between covert and overt attention studied using EEG with simultaneous remote eye tracking. Frontiers in human neuroscience, 10, 592. Lai, M. L., Tsai, M. J., Yang, F. Y., Hsu, C. Y., Liu, T. C., Lee, S. W. Y., ... & Tsai, C. C. (2013). A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational research review, 10, 90-115. Page 151 of 178 Lamar, M., Price, C. C., Giovannetti, T., Swenson, R., & Libon, D. J. (2010). The dysexecutive syndrome associated with ischaemic vascular disease and related subcortical neuropathology: a Boston process approach. Behavioural neurology, 22(1-2), 53-62. Lamy, D., & Zoaris, L. (2009). Task-irrelevant stimulus salience affects visual search. Vision research, 49(11), 1472-1480. Land, M. F. (2009). Vision, eye movements, and natural behaviour. Visual neuroscience, 26(01), 51-62. Land, M. F., & McLeod, P. (2000). From eye movements to actions: how batsmen hit the ball. Nature neuroscience, 3(12), 1340-1345. Land, M., Mennie, N., & Rusted, J. (1999). The roles of vision and eye movements in the control of activities of daily living. Perception, 28(11), 1311-1328. Lange, F., Kröger, B., Steinke, A., Seer, C., Dengler, R., & Kopp, B. (2016). Decomposing card-sorting performance: Effects of working memory load and age-related changes. Neuropsychology, 30(5), 579. Lavie, N. (2010). Attention, distraction, and cognitive control under load. Current directions in psychological science, 19(3), 143-148. Lawrence, V., Houghton, S., Douglas, G., Durkin, K., Whiting, K., & Tannock, R. (2004). Executive function and ADHD: A comparison of children’s performance during neuropsychological testing and real-world activities. Journal of attention disorders, 7(3), 137-149. Lees, R., Fearon, P., Harrison, J. K., Broomfield, N. M., & Quinn, T. J. (2012). Cognitive and mood assessment in stroke research. Stroke, 43(6), 1678-1680. Page 152 of 178 Leigh, R. J., & Kennard, C. (2004). Using saccades as a research tool in the clinical neurosciences. Brain, 127(3), 460-477. Leininger, S., Strong, C. A. H., & Donders, J. (2014). Predictors of outcome after treatment of mild traumatic brain injury: A pilot study. The Journal of head trauma rehabilitation, 29(2), 109-116. Leone, C., Feys, P., Moumdjian, L., D’Amico, E., Zappia, M., & Patti, F. (2017). Cognitive-motor dual-task interference: a systematic review of neural correlates. Neuroscience & Biobehavioural Reviews, 75, 348-360. Levin, H. S., Grossman, R. G., & Kelly, P. J. (1976). Aphasic disorder in patients with closed head injury. Journal of Neurology, Neurosurgery & Psychiatry, 39(11), 1062-1070. Lezak, M., & Assessment, D. N. (1995). Oxford University Press. New York. Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. (2012). Neuropsychological assessment.(5thedn) Oxford University Press. New York, NY. Litman, R. E., Hommer, D. W., Clem, T., Ornsteen, M. L., Ollo, C., & Pickar, D. (1991). Correlation of Wisconsin Card Sorting Test performance with eye tracking in schizophrenia. Am J Psychiatry, 148(11), 1580-1582. Lodha, N., Patel, P., Casamento-Moran, A., Hays, E., Poisson, S. N., & Christou, E. A. (2019). Strength or motor control: What matters in high-functioning stroke? Frontiers in neurology, 9, 1160. Logie, R. H., Trawley, S., & Law, A. (2011). Multitasking: Multiple, domain-specific cognitive functions in a virtual environment. Memory & cognition, 39(8), 1561-1574. Page 153 of 178 Loring, D. W. (2010). History of neuropsychology through epilepsy eyes. Archives of Clinical Neuropsychology, 25(4), 259-273. https://doi.org/10.1093/arclin/acq024 Lou, Y., Cai, H., Liu, X., & Li, X. (2019). Effects of self-enhancement on eye movements during reading. Frontiers in Psychology, 10, 343. Lu Pinke, M. L. (2014). The Development of an Eye-Tracking Method to Assess Cognitive Flexibility Using a Switching-Task Paradigm. MacInnes, J. J., Iqbal, S., Pearson, J., & Johnson, E. N. (2018). Wearable eye-tracking research: Automated dynamic gaze mapping and accuracy/precision comparisons across devices. Unpublished manuscript. Retrieved online: https://www.biorxiv.org/content/biorxiv/early/2018/06/28/299925.full.pdf Mackay, J., Mensah, G. A., Mendis, S., & Greenlund, K. (2004). The atlas of heart disease and stroke. World Health Organization. Maj, M., D'Elia, L., Satz, P., Janssen, R., Zaudig, M., Uchiyama, C., ... & Chervinsky, A. (1993). Evaluation of two new neuropsychological tests designed to minimize cultural bias in the assessment of HIV-1 seropositive persons: a WHO study. Archives of Clinical Neuropsychology, 8(2), 123-135. Malec, J. F. (2004). Comparability of Mayo-Portland Adaptability Inventory ratings by staff, significant others and people with acquired brain injury. Brain Injury, 18(6), 563-575. Malec, J. F., & Lezak, M. D. (2008). Manual for the Mayo-Portland Adaptability Inventory (MPAI-4) for adults, children and adolescents. Page 154 of 178 Malec, J. F., Kragness, M., Evans, R. W., Finlay, K. L., Kent, A., & Lezak, M. D. (2003). Further Psychometric Evaluation and Revision of the Mayo‐Portland Adaptability Inventory in a National Sample. The Journal of head trauma rehabilitation, 18(6), 479-492. Manchester, D., Priestley, N., & Jackson, H. (2004). The assessment of executive functions: Coming out of the office. Brain injury, 18(11), 1067-1081. Mapstone, M., Weintraub, S., Nowinski, C., Kaptanoglu, G., Gitelman, D. R., & Mesulam, M. M. (2003). Cerebral hemispheric specialization for spatial attention: spatial distribution of search-related eye fixations in the absence of neglect. Neuropsychologia, 41(10), 1396-1409. Marcotte, T. D., Scott, J. C., Kamat, R., & Heaton, R. K. (2010). Neuropsychology and the prediction of everyday functioning. The Guilford Press. Marquine, M. J., Attix, D. K., Goldstein, L. B., Samsa, G. P., Payne, M. E., Chelune, G. J., & Steffens, D. C. (2010). Differential patterns of cognitive decline in anterior and posterior white matter hyperintensity progression. Stroke, 41(9), 1946-1950. Matsumoto, H., Terao, Y., Yugeta, A., Fukuda, H., Emoto, M., Furubayashi, T., ... & Ugawa, Y. (2011). Where do neurologists look when viewing brain CT images? An eye-tracking study involving stroke cases. PloS one, 6(12), e28928. Mccue, M., & Aitken, S. (1998). Ecological Validity of Assessment of Problem Solving and Executive Abilities. Rehabilitation Psychology, 43(2), 181. McDonald, A. S. (2002). The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments. Computers & Education, 39(3), 299-312. Page 155 of 178 Mele, M. L., & Federici, S. (2012). Gaze and eye-tracking solutions for psychological research. Cognitive processing, 13(1), 261-265. Melton, A. W. (1963). Implications of short-term memory for a general theory of memory. Journal of verbal Learning and verbal Behaviour, 2(1), 1-21. Milberg, W. P., Hebben, N., Kaplan, E., Grant, I., & Adams, K. M. (2009). The Boston process approach to neuropsychological assessment. Neuropsychological assessment of neuropsychiatric and neuromedical disorders, 42-65. Milionis, H. J., Liberopoulos, E., Goudevenos, J., Bairaktari, E. T., Seferiadis, K., & Elisaf, M. S. (2005). Risk factors for first-ever acute ischemic non-embolic stroke in elderly individuals. International journal of Cardiology, 99(2), 269-275. Miller, J. B., & Barr, W. B. (2017). The technology crisis in neuropsychology. Archives of Clinical Neuropsychology, 32(5), 541-554. https://doi.org/10.1093/arclin/acx050 Miller, E. L., Murray, L., Richards, L., Zorowitz, R. D., Bakas, T., Clark, P., & Billinger, S. A. (2010). Comprehensive overview of nursing and interdisciplinary rehabilitation care of the stroke patient. Stroke, 41(10), 2402-2448. Milner, B. (1963). Effects of different brain lesions on card sorting: the role of the frontal lobes. Journal of American Medical Association, Neurology, 9, 90-100. doi: 10.1001/archneur.1963.00460070100010 Mirelman, A., Shema, S., Maidan, I., & Hausdorff, J. M. (2018). Gait. In Handbook of clinical neurology (Vol. 159, pp. 119-134). Elsevier. Page 156 of 178 Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive psychology, 41(1), 49-100. Morris, J. (2009). Effects of right hemisphere strokes on personality functioning. Topics in stroke rehabilitation, 16(6), 425-430. Mozzaffarian, D., Benjamin, E. J., Go, A. S., Arnett, D. K., Blaha, M. J., Cushman, M., ... & Huffman, M. D. (2015). Executive summary. Circulation, 131(4), 434-441. Mueller, B. A., Lim, K. O., Hemmy, L., & Camchong, J. (2015). Diffusion MRI and its role in neuropsychology. Neuropsychology review, 25(3), 250-271. Naess, H., Waje-Andreassen, U., Thomassen, L., Nyland, H., & Myhr, K. M. (2006). Health-related quality of life among young adults with ischemic stroke on long-term follow-up. Stroke, 37(5), 1232-1236. Nagahama, Y., Fukuyama, H., Yamauchi, H., Matsuzaki, S., Konishi, J., Shibasaki, H., & Kimura, J. (1996). Cerebral activation during performance of a card sorting test. Brain, 119(5), 1667-1676. Nalder, E. J., Clark, A. J., Anderson, N. D., & Dawson, D. R. (2017). Clinicians’ perceptions of the clinical utility of the Multiple Errands Test for adults with neurological conditions. Neuropsychological rehabilitation, 27(5), 685-706. https://doi.org/10.1080/09602011.2015.1067628 Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbonneau, S., Whitehead, V., Collin, I., ... & Chertkow, H. (2005). The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society, 53(4), 695-699. Page 157 of 178 Ng, Y. S., Stein, J., Ning, M., & Black-Schaffer, R. M. (2007). Comparison of clinical characteristics and functional outcomes of ischemic stroke in different vascular territories. Stroke, 38(8), 2309-2314. Niehorster, D. C., Cornelissen, T. H., Holmqvist, K., Hooge, I. T., & Hessels, R. S. (2018). What to expect from your remote eye-tracker when participants are unrestrained. Behaviour research methods, 50(1), 213-227. Nisiforou, E. A., & Laghos, A. (2013). Do the eyes have it? Using eye tracking to assess students cognitive dimensions. Educational Media International, 50(4), 247-265. Nouh, A., Remke, J., & Ruland, S. (2014). Ischemic posterior circulation stroke: a review of anatomy, clinical presentations, diagnosis, and current management. Frontiers in neurology, 5, 30. Nuthmann, A., & Malcolm, G. L. (2016). Eye guidance during real-world scene search: The role color plays in central and peripheral vision. Journal of Vision, 16(2), 3-3. Nunnally, JC.; Bernstein, IH. Psychometric Theory. 3rd ed. McGraw-Hill College Division; New York: 1994. Nyhus, E., & Barceló, F. (2009). The Wisconsin Card Sorting Test and the cognitive assessment of prefrontal executive functions: a critical update. Brain and cognition, 71(3), 437-451. Nys, G. M. S., Van Zandvoort, M. J. E., De Kort, P. L. M., Jansen, B. P. W., Van der Worp, H. B., Kappelle, L. J., & De Haan, E. H. F. (2005). Domain-specific cognitive recovery after first-ever stroke: a follow-up study of 111 cases. Journal of the International Neuropsychological Society : JINS, 11(7), 795–806. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/16519259 Page 158 of 178 Nys, G. M. S., Zandvoort, M. J. E. Van, Kort, P. L. M. De, & Worp, H. B. Van Der. (2005). The prognostic value of domain-specific cognitive abilities in acute first-ever stroke, 821–827. Nys, G. M., Zandvoort, M. J., de Kort, P. L., Jansen, B. P., de Haan, E. H., & Kappelle, L. J. (2007). Cognitive disorders in acute stroke: Prevalence and clinical determinants. Cerebrovascular Disease, 23, 408-416. DOI: 10.1159/000101464 O’Donnell, L. J., & Westin, C. F. (2011). An introduction to diffusion tensor image analysis. Neurosurgery Clinics, 22(2), 185-196. Orquin, J. L., Ashby, N. J., & Clarke, A. D. (2016). Areas of interest as a signal detection problem in behavioral eye‐tracking research. Journal of Behavioral Decision Making, 29(2-3), 103-115. Osterrieth, P. A. (1944). Le test de copie d'une figure complexe; contribution à l'étude de la perception et de la mémoire [Test of copying a complex figure; contribution to the study of perception and memory]. Archives de Psychologie, 30, 206–356. Ozonoff, S. (1995). Reliability and validity of the Wisconsin card sorting test in studies of autism. Neuropsychology, 9(4), 491. Oztop, P., Ayas, S., Ustaomer, K., Cosar, S. N. S., & Yemisci, O. U. (2013). Functional outcomes in anterior and posterior circulation ischemic strokes/Anterior ve posterior dolasimli iskemik inmedeki fonksiyonel sonuclar. Turkish Journal of Physical Medicine and Rehabilitation, 59(1), 13-18. Paelecke-Habermann, Y., Pohl, J., & Leplow, B. (2005). Attention and executive functions in remitted major depression patients. Journal of affective disorders, 89(1-3), 125-135. https://doi.org/10.1016/j.jad.2005.09.006h Page 159 of 178 Påhlman, U., Gutierrez-Perez, C., Sävborg, M., Knopp, E., & Tarkowski, E. (2011). Cognitive function and improvement of balance after stroke in elderly people: the Gothenburg cognitive stroke study in the elderly. Disability and Rehabilitation, 33(21-22), 1952-1962. Pavisic, I. M., Firth, N. C., Parsons, S., Rego, D. M., Shakespeare, T. J., Yong, K. X., ... & Carton, A. M. (2017). Eyetracking metrics in young onset Alzheimer’s disease: a window into cognitive visual functions. Frontiers in neurology, 8, 377. Paolucci, S., Antonucci, G., Pratesi, L., Traballesi, M., Lubich, S., & Grasso, M. G. (1998). Functional outcome in stroke inpatient rehabilitation: predicting no, low and high response patients. Cerebrovascular Diseases, 8(4), 228-234. Papegaaij, S., Hortobagyi, T., Godde, B., Kaan, W. A., Erhard, P., & Voelcker-Rehage, C. (2017). Neural correlates of motor-cognitive dual-tasking in young and old adults. PloS one, 12(12), e0189025. Park, B., & Brünken, R. (2015). The rhythm method: A new method for measuring cognitive load—An experimental dual‐task study. Applied Cognitive Psychology, 29(2), 232-243. Park, K. C., Yoon, S. S., & Rhee, H. Y. (2011). Executive dysfunction associated with stroke in the posterior cerebral artery territory. Journal of Clinical Neuroscience, 18(2), 203-208. Parsey, C. M., & Schmitter-Edgecombe, M. (2013). Applications of technology in neuropsychological assessment. The Clinical Neuropsychologist, 27(8), 1328-1361. Parsons, T. D. (2016). Clinical Neuropsychology and Technology: What’s New and How We Can Use It. Springer. Page 160 of 178 Parsons, T. D., McMahan, T., & Kane, R. (2018). Practice parameters facilitating adoption of advanced technologies for enhancing neuropsychological assessment paradigms. The Clinical Neuropsychologist, 32(1), 16-41. Parsons, T. D., Duffield, T., & Asbee, J. (2019). A comparison of virtual reality classroom continuous performance tests to traditional continuous performance tests in delineating ADHD: a meta-analysis. Neuropsychology review, 1-19. Parton, A., Malhotra, P., & Husain, M. (2004). Hemispatial neglect. Journal of Neurology, Neurosurgery & Psychiatry, 75(1), 13-21. Patel, M. D., Coshall, C., Rudd, A. G., & Wolfe, C. D. (2002). Cognitive impairment after stroke: clinical determinants and its associations with long‐term stroke outcomes. Journal of the American Geriatrics Society, 50(4), 700-706. Peltsch, A., Hemraj, A., Garcia, A., & Munoz, D. P. (2011). Age-related trends in saccade characteristics among the elderly. Neurobiology of aging, 32(4), 669-679. Pendlebury, S. T., & Rothwell, P. M. (2009). Prevalence, incidence, and factors associated with pre-stroke and post-stroke dementia: a systematic review and meta-analysis. The Lancet Neurology, 8(11), 1006-1018. Pendlebury, S. T., Mariz, J., Bull, L., Mehta, Z., & Rothwell, P. M. (2012). MoCA, ACE-R, and MMSE versus the National Institute of Neurological Disorders and Stroke–Canadian Stroke Network vascular cognitive impairment harmonization standards neuropsychological battery after tia and stroke. Stroke, 43(2), 464-469. Perna, R., Loughan, A. R., & Talka, K. (2012). Executive functioning and adaptive living skills after acquired brain injury. Applied Neuropsychology: Adult, 19(4), 263-271. Page 161 of 178 Pinke, Melissa Lu, "The Development of an Eye-Tracking Method to Assess Cognitive Flexibility Using a Switching-Task Paradigm" (2014). Theses and Dissertations. Paper 598. Plomecka, M. B., Barańczuk-Turska, Z., Pfeiffer, C., & Langer, N. (2020). Aging Effects and Test–Retest Reliability of Inhibitory Control for Saccadic Eye Movements. Eneuro, 7(5). Poletti, B., Carelli, L., Solca, F., Lafronza, A., Pedroli, E., Faini, A., ... & Meriggi, P. (2017). An eye-tracking controlled neuropsychological battery for cognitive assessment in neurological diseases. Neurological Sciences, 38(4), 595-603. Pollock, A., St George, B., Fenton, M., & Firkins, L. (2012). Top ten research priorities relating to life after stroke. The Lancet Neurology, 11(3), 209. Pollock, A., Hazelton, C., Henderson, C. A., Angilley, J., Dhillon, B., Langhorne, P., & Shahani, U. (2011). Interventions for visual field defects in patients with stroke. The Cochrane Library. Poole, A., & Ball, L. J. (2006). Eye tracking in HCI and usability research. In Encyclopedia of human computer interaction (pp. 211-219). IGI Global. Poreh, A. M. (2000). Forum the quantified process approach: An emerging methodology to neuropsychological assessment. The Clinical Neuropsychologist, 14(2), 212-222. Poreh, A. M. (2006). A brief introduction to the Quantified Process Approach. In A. M. Poreh (Ed.), Studies on neuropsychology, neurology and cognition. The quantified process approach to neuropsychological assessment (p. 3–15). Taylor & Francis. Poulin, V., Korner-Bitensky, N., Dawson, D. R., & Bherer, L. (2012). Efficacy of executive function interventions after stroke: a systematic review. Topics in stroke rehabilitation, 19(2), 158-171. Page 162 of 178 Prentice, K. J., Gold, J. M., & Buchanan, R. W. (2008). The Wisconsin Card Sorting impairment in schizophrenia is evident in the first four trials. Schizophrenia research, 106(1), 81-87. Purves D, Augustine GJ, Fitzpatrick D, et al., editors. Sunderland (MA): Sinauer Associates; 2001. Rabin, L. A., Barr, W. B., & Burton, L. A. (2005). Assessment practices of clinical neuropsychologists in the United States and Canada: A survey of INS, NAN, and APA Division 40 members. Archives of Clinical Neuropsychology, 20(1), 33-65. Rabin, L. A., Spadaccini, A. T., Brodale, D. L., Grant, K. S., Elbulok-Charcape, M. M., & Barr, W. B. (2014). Utilization rates of computerized tests and test batteries among clinical neuropsychologists in the United States and Canada. Professional Psychology: Research and Practice, 45(5), 368. Rabin, L. A., Paolillo, E., & Barr, W. B. (2016). Stability in test-usage practices of clinical neuropsychologists in the United States and Canada over a 10-year period: A follow-up survey of INS and NAN members. Archives of Clinical Neuropsychology, acw007. Rabinovici, G. D., Stephens, M. L., & Possin, K. L. (2015). Executive dysfunction. CONTINUUM: Lifelong Learning in Neurology, 21(3 Behavioural Neurology and Neuropsychiatry), 646. Randolph, C., Tierney, M. C., Mohr, E., & Chase, T. N. (1998). The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): preliminary clinical validity. Journal of clinical and experimental neuropsychology, 20(3), 310-319. Page 163 of 178 Rao, S. M., Aubin-Faubert, P. S., & Leo, G. J. (1989). Information processing speed in patients with multiple sclerosis. Journal of Clinical and Experimental Neuropsychology, 11(4), 471-477. Raspelli, S., Pallavicini, F., Carelli, L., Morganti, F., Poletti, B., Corra, B., ... & Riva, G. (2011). Validation of a neuro virtual reality-based version of the multiple errands test for the assessment of executive functions. Rayner, K., & Pollatsek, A. (1981). Eye movement control during reading: Evidence for direct control. The Quarterly journal of experimental psychology, 33(4), 351-373. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological bulletin, 124(3), 372. Rayner, K., & Reingold, E. M. (2015). Evidence for direct cognitive control of fixation durations during reading. Current opinion in behavioural sciences, 1, 107-112. Reese, C., Suhr, J., & Riddle, T. (2012). Exploration of malingering indices in the Wechsler Adult Intelligence Scale - Fourth Edition Digit Span Subtest. Archives of Clinical Neuropsychology, 27(2), 176-181. https://doi.org/10.1093/arclin/acr117 Reingold, E. M. (2014). Eye tracking research and technology: Towards objective measurement of data quality. Visual cognition, 22(3-4), 635-652. Reitan, R. M., & Wolfson, D. (1985). The Halstead-Reitan neuropsychological test battery: Theory and clinical interpretation (Vol. 4). Reitan Neuropsychology. Reitan, R. M., & Wolfson, D. (1994). A selective and critical review of neuropsychological deficits and the frontal lobes. Neuropsychology review, 4(3), 161-198. https://doi.org/10.1007/BF01874891 Page 164 of 178 Rey, A. (1958). L'examen clinique en psychologie. [The clinical examination in psychology]. Presses Universitaries De France. Richardson, D. C., & Johnson, S. P. (2008). Eye tracking research in infants and adults. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 30, No. 30). Richardson, D. C., & Spivey, M. J. (2004). Eye tracking: Characteristics and methods. Encyclopedia of biomaterials and biomedical engineering, 3, 1028-1042. Robbins, T. W., & Roberts, A. C. (2007). Differential regulation of fronto-executive function by the monoamines and acetylcholine. Cerebral cortex, 17(suppl_1), 151-160. Rohr, C. S., Okon-Singer, H., Craddock, R. C., Villringer, A., & Margulies, D. S. (2013). Affect and the brain's functional organization: a resting-state connectivity approach. PloS one, 8(7), e68015. Rouinfar, A., Agra, E., Larson, A. M., Rebello, N. S., & Loschky, L. C. (2014). Linking attentional processes and conceptual problem solving: visual cues facilitate the automaticity of extracting relevant information from diagrams. Frontiers in psychology, 5, 1094. Rowe, F. (2011). Prevalence of ocular motor cranial nerve palsy and associations following stroke. Eye, 25(7), 881-887. Sailer, U., Flanagan, J. R., & Johansson, R. S. (2005). Eye–hand coordination during learning of a novel visuomotor task. Journal of Neuroscience, 25(39), 8833-8842. Salvucci, D. D., & Goldberg, J. H. (2000, November). Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 symposium on Eye tracking research & applications (pp. 71-78). Page 165 of 178 Sand, K. M., Midelfart, A., Thomassen, L., Melms, A., Wilhelm, H., & Hoff, J. M. (2013). Visual impairment in stroke patients–a review. Acta Neurologica Scandinavica, 127(s196), 52-56. Sandercock, P., Collins, R., Counsell, C., & Farrell, B. (1997). The International Stroke Trial (IST): a randomised trial of aspirin, subcutaneous heparin, both, or neither among 19 435 patients with acute ischaemic stroke. The Lancet, 349(9065), 1569. Sangha, H., Lipson, D., Foley, N., Salter, K., Bhogal, S., Pohani, G., & Teasell, R. W. (2005). A comparison of the Barthel Index and the Functional Independence Measure as outcome measures in stroke rehabilitation: patterns of disability scale usage in clinical trials. International Journal of Rehabilitation Research, 28(2), 135-139. Sass, K. J., Spencer, D. D., Kim, J. H., Westerveld, M., Novelly, R. A., & Lencz, T. (1990). Verbal memory impairment correlates with hippocampal pyramidal cell density. Neurology, 40(11), 1694-1694. Sbordone, R. J., & Long, C. (1996). Ecological validity of neuropsychological testing. CRC Press. Scheiter, K., & Van Gog, T. (2009). Introduction: Using eye tracking in applied research to study and stimulate the processing of information from multi-representational sources. Applied Cognitive Psychology. Schneider, A. T., Pancioli, A. M., Khoury, J. C., Rademacher, E., Tuchfarber, A., Miller, R., ... & Broderick, J. P. (2003). Trends in community knowledge of the warning signs and risk factors for stroke. Jama, 289(3), 343-346. Page 166 of 178 Schroeder, R. W., Twumasi-Ankrah, P., Baade, L. E., & Marshall, P. S. (2012). Reliable Digit Span: A systematic review and cross-validation study. Assessment, 19(1), 21-30. Scott, J., & Mayo, A. M. (2018). Instruments for detection and screening of cognitive impairment for older adults in primary care settings: A review. Geriatric Nursing, 39(3), 323-329. Seligman, S. C., & Giovannetti, T. (2015). The potential utility of eye movements in the detection and characterization of everyday functional difficulties in mild cognitive impairment. Neuropsychology review, 25(2), 199-215. Shallice, T. I. M., & Burgess, P. W. (1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114(2), 727-741. https://doi.org/10.1093/brain/114.2.727 Silver, C. H. (2000). Ecological validity of neuropsychological assessment in childhood traumatic brain injury. The Journal of head trauma rehabilitation, 15(4), 973-988. Smith, N. D., Glen, F. C., Mönter, V. M., & Crabb, D. P. (2014). Using eye tracking to assess reading performance in patients with glaucoma: A within-person study. Journal of ophthalmology, 2014. Somsen, R. J. (2007). The development of attention regulation in the Wisconsin Card Sorting Task. Developmental science, 10(5), 664-680. Spencer, R. J., Axelrod, B. N., Drag, L. L., Waldron-Perrine, B., Pangilinan, P. H., & Bieliauskas, L. A. (2013). WAIS-IV reliable digit span is no more accurate than age corrected scaled score as an indicator of invalid performance in a veteran sample undergoing evaluation for mTBI. The Clinical Neuropsychologist, 27(8), 1362-1372. Page 167 of 178 Squire, R. F., Steinmetz, N. A., & Moore, T. (2012). Frontal eye field. Scholarpedia, 7(10), 5341. Spooner, D. M., & Pachana, N. A. (2006). Ecological validity in neuropsychological assessment: A case for greater consideration in research with neurologically intact populations. Archives of clinical neuropsychology, 21(4), 327-337. https://doi.org/10.1016/j.acn.2006.04.004 Srikanth, V. K., Thrift, A. G., Saling, M. M., Anderson, J. F., Dewey, H. M., Macdonell, R. A., & Donnan, G. A. (2003). Increased risk of cognitive impairment 3 months after mild to moderate first-ever stroke. Stroke, 34(5), 1136-1143. Stalbovs, K., Scheiter, K., & Gerjets, P. (2015). Implementation intentions during multimedia learning: using if-then plans to facilitate cognitive processing. Learning and Instruction, 35, 1-15. Stefanova, E. D., Kostic, V. S., Ziropadja, L., Markovic, M., & Ocic, G. (2002). Serial position learning effects in patients with aneurysms of the anterior communicating artery. Journal of clinical and experimental neuropsychology, 24(5), 687-694. Steinmetz J-P, Brunner M, Loarer E, Houssemand C. (2010). Incomplete psychometric equivalence of scores obtained on the manual and the computer version of the Wisconsin Card Sorting Test? Psychological Assessment, 22(1), 199–202. Stuss, D. T., Levine, B., Alexander, M. P., Hong, J., Palumbo, C., Hamer, L., ... & Izukawa, D. (2000). Wisconsin Card Sorting Test performance in patients with focal frontal and posterior brain damage: effects of lesion location and test structure on separable cognitive processes. Neuropsychologia, 38(4), 388-402. Page 168 of 178 Stuss, D., Levine, B. (2002). Adult clinical neuropsychology: Lessons from studies of the frontal lobes. Annual Review of Psychology, 53, 401-433. DOI: 10.1146/annurev.psych.53.100901.135220 Su, C. Y., Wuang, Y. P., Chang, J. K., Guo, N. W., & Kwan, A. L. (2006). Wisconsin Card Sorting Test performance after putaminal hemorrhagic stroke. The Kaohsiung Journal of Medical Sciences, 22(2), 75-84. Suchan, J., & Karnath, H. O. (2011). Spatial orienting by left hemisphere language areas: a relic from the past? Brain, 134(10), 3059-3070. Sullivan, E. V., & Bigler, E. D. (2015). Neuroimaging’s role in neuropsychology: introduction to the special issue of neuropsychology review on neuroimaging in neuropsychology. Neuropsychology Review, 25 (221). https://doi.org/10.1007/s11065-015-9296-7 Sun, J. H., Tan, L., & Yu, J. T. (2014). Post-stroke cognitive impairment: epidemiology, mechanisms and management. Annals of translational medicine, 2(8). Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285. Sweller, J., Van Merrienboer, J. J., & Paas, F. G. (1998). Cognitive architecture and instructional design. Educational psychology review, 10(3), 251-296. Tatemichi, T. K., Desmond, D. W., Stern, Y., Paik, M., Sano, M., & Bagiella, E. (1994). Cognitive impairment after stroke: frequency, patterns, and relationship to functional abilities. Journal of Neurology, Neurosurgery & Psychiatry, 57(2), 202-207. Teasell, R., Neuroscience, N. B., & Heitzner, J. (n.d.). 2 . Clinical Consequences of Stroke, (July 2008), 1–19. Page 169 of 178 Testa, R., Bennett, P., & Ponsford, J. (2012). Factor analysis of nineteen executive function tests in a healthy adult population. Archives of Clinical Neuropsychology, 27(2), 213-224. https://doi.org/10.1093/arclin/acr112 Thomas, L. E., & Lleras, A. (2007). Moving eyes and moving thought: on the spatial compatibility between eye movements and cognition. Psychonomic Bulletin & Review, 14(4), 663–668. http://doi.org/10.3758/BF03196818 Tien, A.Y., Spevack, T.V., Jones, D.W., Pearlson, G.D., Schlaepfer, T.E., & Strauss, M.E. (1996). Computerized wisconsin card sorting test: Comparison with manual administration. The Kaohsiung Journal of Medical Sciences, 12(8), 479–485. Tokuda, S. (2008). Using saccadic eye movements to measure mental workload. Journal of Vision, 8(17), 86-86. Tombaugh, T. N., & McIntyre, N. J. (1992). The mini‐mental state examination: a comprehensive review. Journal of the American Geriatrics Society, 40(9), 922-935. Tombaugh, T. N., Kozak, J., & Rees, L. (1999). Normative data stratified by age and education for two measures of verbal fluency: FAS and animal naming. Archives of clinical neuropsychology, 14(2), 167-177. Tupper, D.E. Neuropsychol Rev (1999) 9: 1. https://doi.org/10.1023/A:1025634702965 Tyng, C. M., Amin, H. U., Saad, M. N., & Malik, A. S. (2017). The influences of emotion on learning and memory. Frontiers in psychology, 8, 1454. Upshaw, J. N., Leitner, D. W., Rutherford, B. J., Miller, H. B., & Libben, M. R. (2019). Allocentric Versus Egocentric Neglect in Stroke Patients: A Pilot Study Investigating the Page 170 of 178 Assessment of Neglect Subtypes and Their Impacts on Functional Outcome Using Eye Tracking. Journal of the International Neuropsychological Society: JINS, 25(5), 479-489. van den Burg, W., & Kingma, A. (1999). Performance of 225 Dutch school children on Rey's Auditory Verbal Learning Test (AVLT): parallel test-retest reliabilities with an interval of 3 months and normative data. Archives of Clinical Neuropsychology, 14(6), 545-559. Van Gog, T., & Scheiter, K. (2010). Eye tracking as a tool to study and enhance multimedia learning. Learning & Instruction, 20, 95-99. Vernet, M., Quentin, R., Chanes, L., Mitsumasu, A., & Valero-Cabré, A. (2014). Frontal eye field, where art thou? Anatomy, function, and non-invasive manipulation of frontal regions involved in eye movements and associated cognitive operations. Vlaskamp, B. N., & Hooge, I. T. C. (2006). Crowding degrades saccadic search performance. Vision research, 46(3), 417-425. Van Wyk, A., Eksteen, C. A., Becker, P. J., & Heinze, B. M. (2016). A cross-sectional survey and cross-sectional clinical trial to determine the prevalence and management of eye movement disorders and vestibular dysfunction in post-stroke patients in the sub-acute phase: protocol. Frontiers in Neurology, 7. Vickers, J. N. (1992). Gaze control in putting. Perception, 21(1), 117-132. Wagle, J., Farner, L., Flekkøy, K., Bruun Wyller, T., Sandvik, L., Fure, B., ... & Engedal, K. (2011). Early post-stroke cognition in stroke rehabilitation patients predicts functional outcome at 13 months. Dementia and geriatric cognitive disorders, 31(5), 379-387. Walle, K. M., Nordvik, J. E., Becker, F., Espeseth, T., Sneve, M. H., & Laeng, B. (2019). Unilateral neglect post stroke: Eye movement frequencies indicate directional hypokinesia Page 171 of 178 while fixation distributions suggest compensational mechanism. Brain and behaviour, 9(1), e01170. doi:10.1002/brb3.1170 Wang, C. A., Baird, T., Huang, J., Coutinho, J. D., Brien, D. C., & Munoz, D. P. (2018). Arousal effects on pupil size, heart rate, and skin conductance in an emotional face task. Frontiers in Neurology, 9, 1029. Wechsler, D., Coalson, D. L., & Raiford, S. E. (2008). WAIS-IV. Wechsler Adult Intelligence Scale: Fourth Edition. Technical and interpretative manual. San Antonio, TX: NCS Pearson. Weinstein, G., Preis, S. R., Beiser, A. S., Au, R., Kelly‐Hayes, M., Kase, C. S., ... & Seshadri, S. (2014). Cognitive performance after stroke–The F ramingham H eart S tudy. International Journal of Stroke, 9, 48-54. Whyte, E., Skidmore, E., Aizenstein, H., Ricker, J., & Butters, M. (2011). Cognitive Impairment in Acquired Brain Injury: A Predictor of Rehabilitation Outcomes and an Opportunity for Novel Interventions. PM & R : The Journal of Injury, Function, and Rehabilitation, 3(6 0 1), S45–S51. http://doi.org/10.1016/j.pmrj.2011.05.007 Wild, K., Howieson, D., Webbe, F., Seelye, A., & Kaye, J. (2008). The status of computerized cognitive testing in aging: A systematic review. Alzheimer’s & Dementia : The Journal of the Alzheimer’s Association, 4(6), 428–437. http://doi.org/10.1016/j.jalz.2008.07.003 Wilson, B. A. (1993). Ecological validity of neuropsychological assessment: Do neuropsychological indexes predict performance in everyday activities?. Applied and Preventive Psychology, 2(4), 209-215. Page 172 of 178 Yamauchi, H., Nishii, R., Higashi, T., Kagawa, S., & Fukuyama, H. (2011). Silent cortical neuronal damage in atherosclerotic disease of the major cerebral arteries. Journal of Cerebral Blood Flow & Metabolism, 31(3), 953-961. Yang, Q., Wang, T., Su, N., Xiao, S., & Kapoula, Z. (2013). Specific saccade deficits in patients with Alzheimer’s disease at mild to moderate stage and in patients with amnestic mild cognitive impairment. Age, 35(4), 1287-1298. Zagermann, J., Pfeil, U., & Reiterer, H. (2016). Measuring Cognitive Load using Eye Tracking Technology in Visual Computing. In BELIV'16: Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization (pp. 78-85). Zimiles, H. (1996). Rethinking the validity of psychological assessment. American Psychologist, 51(9), 980-981. http://dx.doi.org/10.1037/0003-066X.51.9.980 Zimmerman, D. L., Ownsworth, T., O'Donovan, A., Roberts, J., & Gullo, M. J. (2016). Independence of hot and cold executive function deficits in high-functioning adults with autism spectrum disorder. Frontiers in human neuroscience, 10, 24. Page 173 of 178 APPENDIX A: Assumption Testing Study 1 Preliminary Analyses Test for normality was conducted on age, education, level of alertness and anxiety, and computer proficiency (CPQ-12). Shapiro-Wilk tests suggested violations of normality for all variables listed (p < .05); thus, nonparametric measures were employed to explore differences between cognitive load groups. Results of Kruskal-Wallis analyses revealed no significant differences between cognitive load groups along each of these variables (all p values > .05). Likewise, Mann-Whitney U analyses revealed no significant differences for order of administration on these same variables (all p values > .05). On the cWCST, number of perseverative errors and categories completed did not differ by order of administration after Bonferroni-correction was applied (p values > .025). Study 1, Hypothesis 1 (MANOVA and mixed-measures ANOVA): Univariate outliers were examined via boxplots (3 x interquartile range) and multivariate outliers were investigated via Mahalanobis Distance (critical value = 18.47; p < .001): two cases were identified as extreme outliers and removed from analyses. Pillai’s Trace was interpreted due to unequal group sizes and IVs deviating from a normal distribution (Olson, 1974). The assumption of homogeneity of variance-covariance matrices, assessed by Box’s test, was met (p = .20). Homogeneity of variance was also met (all p values > .01). A power analysis with the following parameters: effect size = 0.25, alpha = .05, power = .80, groups = 3, and measures = 4, estimated the required sample size in the study to be 113 participants. One outlier was identified, based on studentized residual of 3.10 for dwell time on Page 174 of 178 the distraction AOI. This outlier was retained given it was on one variable of four in the model, marginally surpassed the cut-off of 3.0, and was unlikely to affect results. All other assumptions were met for analyses. Study 1, Hypothesis 2 (Hierarchical Regressions): Assumptions were tested prior to analyses. Due to significant multicollinearity between fixation measures (i.e., dwell time, counts, and revisits), only dwell time and average time per fixation on AOIs and total saccades were retained. Durbin-Watson was used to assess for autocorrelation in residuals and found to be within normal limits (1.84). VIF values were examined for potential collinearity and found to all be within normal limits (i.e., 1 to 10). Standardized residuals revealed no extreme outlier that surpassed the +/- 3.0 cut-off. Examination of leverage points revealed two cases above the recommended cut-off (> 0.50 cut-off). Cook’s distance values revealed no extreme cases (all <1.0 cut-off). Extreme cases were filtered from analyses. Study 2 Preliminary Analyses Group demographics: Test of normality with the Shapiro-Wilk test revealed non-normal distribution for age in both groups (p values < .05) and normal distribution of education in both groups (p values > .05), and time since injury for the inpatients after a stroke group violated normality (p < .001). Visual inspection of the Q-Q-plots showed only minor deviations from normality for age in both groups. A chi-square analysis revealed no significant difference for sex ratio between the two groups, χ2(1) = 3.03, p = .08. Z-scores (Z +/- 3.29) and boxplots were inspected for extreme outliers (three times the interquartile range) on age, education, and time since injury (inpatients after a stroke). One outlier for time since injury was identified (Z = 3.79; Page 175 of 178 days = 73). This case was retained as this variable was not significantly correlated with primary outcomes (p > .05), and it is not uncommon for patients to have extended stays on the unit. Preliminary assumptions were checked for independent samples t-tests analyses between the two groups. Standardized T scores were used for analyses; higher scores indicated better performance. Shapiro Wilk tests for normality revealed normal distribution in both groups for T scores on Perseverative Errors and Conceptual Level Responses (p values > .05); normality was violated for number of trials and categories completed (p values < .05). Visual inspection of Q-Q plots revealed a similar pattern of normal and non-normal distribution on these scores, similar results were also found with Levene’s Test for equality of variance. Study 2, Hypothesis 1a (Logistic Regression): Independent variables were examined for outliers; raw data were converted to Z-scores with a cut-off of +/- 3.29. Four outliers emerged from the inpatients after a stroke on saccade count (4.31 and 4.30) and fixation count (4.38 and 3.47). These cases were filtered from respective analyses sensitive to outliers that included these two variables. Collinearity was assessed with correlation analysis: there was modest correlation between the saccade and fixation count (r = .77), but within the .90 cut-off for collinearity (Tabachnik & Fidell, 2014). Both measures were retained for analyses. The Box-Tidwell procedure revealed both IVs to be linearly related to the logit of the DV (all p values > .05). No standardized residuals were detected with a cut-off value of 3 SDs. Study 2, Hypothesis 1b (MANOVA): Potential outliers were explored by use of boxplots and transformation of the IVs to Z-scores. Two cases (inpatients after a stroke) were flagged as outliers (Z = 5.31, 3.98, and 3.55) and removed from analysis. Results of the Shapiro-Wilk test revealed no significant violation of normality (all p values > .001) and visual inspection of Q-Q plots revealed the same. Pearson Page 176 of 178 correlation revealed no multicollinearity between the IVs, (r = .32 to .44; all p values < .01). Visual inspection of scatterplots for both groups revealed a linear relationship of the IVs. Mahalanobis Distance, with a cut-off score of 16.23 and alpha = .05, revealed no significant multivariate outliers. Box’s test of equality of variance revealed homogeneity of variance-covariance matrices (p = .014). Levene’s Test of Homogeneity of Variance revealed heterogeneity with the key card fixation count (p < .001). Thus, post-hoc analyses were conducted using non-parametric Mann-Whitney U tests. Study 2, Hypothesis 2 (Hierarchical Linear Regression) Assumption testing revealed independence of residuals (Durbin-Watson = 1.91; range 0 to 4, optimal = 2.0); however, test of homoscedasticity was not met (no linear relationship between IV and residuals). Thus, we ran the regression using the weighted least squares approach. Assumption of multicollinearity was assessed, and met, by examining correlation matrices of the IVs (all r values < .80), as well as VIF (all values < 10) and 1/VIF (all values > 1). Examination of the studentized deleted residuals revealed no outliers greater than +/- 3.0 SD. Leverage value for each case was within the acceptable limit (cut-off 0.5). Cook’s Distance value for each case was also within the acceptable limit (cut-off 1.0). Mahalanobis’ Distance indicated no multivariate outliers (all values below critical-value of 22.46). Last, there was approximate normality of the studentized residuals, based on inspection of histogram and P-P Plot. Given the results of the assumption tests, results of the multiple hierarchical linear regression can be interpreted with confidence. Study 2, Hypothesis 3 (Mixed-Measures ANOVA): A power analysis with the following parameters was run to determine adequate sample size: effect size (f) = .20, α = .05, β = .80, groups = 2, and measurements = 6. The estimated total Page 177 of 178 sample size required was 44 participants. Outliers were investigated with visual inspection of boxplots. No extreme outliers (+/- 3 SD from the median) were observed. Due to a high degree of multicollinearity between fixation counts and dwell time, the latter was dropped. Fixation counts on each AOI (key card, workspace, and stimulus) were included on two response-types (conceptual and non-conceptual). Inspection of Q-Q plots and results of the Shapiro Wilk tests suggested no extreme deviation from normality on the IVs (all p values > .001). Results of Levene’s Test revealed homogeneity of variance on all IVs (all p values > .05). For the two-way interaction effect, the assumption of sphericity was assumed based on the results of Mauchley’s Test of Sphericity, χ2 (2) = 0.10, p = .95. Study 3 Hypothesis 1 Preliminary Results Assumption testing revealed independence of residuals (Durbin-Watson = 1.72). Homoscedasticity of the data was met. Shapiro-Wilk test revealed a normal distribution of FIM admission and change scores (p = .70 and .48, respectively) and borderline normality of distribution for FIM discharge scores (p = .001). Assumption of multicollinearity was assessed, and met, by examining correlation matrices of the IVs (all r values < .80), as well as VIF (all values < 10) and 1/VIF (all values > 1). Examination of the studentized deleted residuals revealed no outliers greater than +/- 3.0 SD. Cook’s Distance value for each case was also within the acceptable limit (cut-off 1.0). Mahalanobis’ Distance indicated no multivariate outliers (all p values above .05 for critical value). Last, there was approximate normality of the studentized residuals, based on inspection of histogram and P-P Plot. Given the results of the assumption tests, results of the multiple hierarchical linear regression can be interpreted with confidence. Page 178 of 178 APPENDIX B: Study 1 Instructions Low Cognitive Load Instructions: [Provide instructions for cWCST, then…] “For this test, I will provide you with the correct responses: You will first match the color of the cards, after 10 consecutively correct responses you will then match the shape of the card, after another 10 consecutively correct responses you will then match to the number of symbols on the card, and then return to color, then shape, then number again. Do you understand what you are to do? Do you have any questions? Let me know when you are finished.” Medium Cognitive Load Instructions: [As per test manual instructions] High Cognitive Load Instructions: [Provide instructions for cWCST, then…] “Begin tapping your index/pointer finger with the hand not operating the computer mouse at the beat of the metronome, that I will start shortly. Keep tapping your finger at 1-tap/second as you work on this card sorting task. If you forget to keep tapping, I will remind you.” [TURN METRONOME ON. ALLOW PARTICIPANT TO ASK QUESTIONS AND GET COMFORTABLE. REQUEST THEY START TAPPING INDEX FINGER IF NOT ALREADY STARTED. START cWCST.]
Thesis/Dissertation
eng
Psychology
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Updating our approach to neuropsychological assessments : examining the validity of eye tracking with the computerized Wisconsin Card Sorting Test
Text