Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

The LF-ASD brain-computer interface : preliminary on-line identification of imagined finger flexions.. Bozorgzadeh, Ziba 2000

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata

Download

Media
ubc_2001-0008.pdf [ 4.7MB ]
Metadata
JSON: 1.0065267.json
JSON-LD: 1.0065267+ld.json
RDF/XML (Pretty): 1.0065267.xml
RDF/JSON: 1.0065267+rdf.json
Turtle: 1.0065267+rdf-turtle.txt
N-Triples: 1.0065267+rdf-ntriples.txt
Original Record: 1.0065267 +original-record.json
Full Text
1.0065267.txt
Citation
1.0065267.ris

Full Text

The LF-ASD Brain-Computer Interface: Preliminary On-Line Identification of Imagined Finger Flexions in Spontaneous EEG by Ziba Bozorgzadeh B.A.Sc, University of British Columbia, 1996 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Applied Science in THE FACULTY OF GRADUATE STUDIES (Department of Electrical and Computer Engineering) We accept this thesis as conforming to the required standard The University of British Columbia Vancouver, B.C., V6T 1Z4 December 2000 © Ziba Bozorgzadeh, 2000 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department of & The University of British Columbia Vancouver, Canada Date DE-6 (2/88) ABSTRACT The brain-computer interface (BCI) has emerged as a potential and radically new mode of communication for users with neuromuscular impairments since it provides a communication channel based on human brain activity as opposed to peripheral nerves and muscles. The main goal of this research study was to test the effectiveness of the Low-Frequency Asynchronous Switch Design (LF-ASD) BCI technique, which has been designed to detect imagined finger flexion patterns in an asynchronous control environment. A system was developed to study the LF-ASD with imaginary movements in subjects with spinal cord injuries. The new system was evaluated in two studies in which able-bodied and spinal cord injured subjects were asked to control the LF-ASD through movement imagination. In addition to the self-reported errors (false positives and false negatives), another class of errors, "lucky hits" (LHs), was recognized and a methodology for its estimation was introduced. In Study 1, two able-bodied subjects were asked to imagine right-hand index finger flexions. The estimated system performance (including the LH estimation results) for Study 1 produced true positive (hit) rates in the range of 57% to 68% with corresponding false positive rates in the range of 1.2% to 3.4%. The algorithm was modified to reduce false positive rates and was tested in a second study with two subjects with spinal cord injuries. In Study 2, the estimated system performance (including the LH estimation results) produced hit rates in the range of 34.7% to 41.3% with corresponding false positive rates in the range ii of 0.1% to 1.0%. The results of these studies provided strong evidence that able-bodied subjects as well as SCI subjects can activate the on-line LF-ASD with imagined/attempted right-hand index finger flexions. The ensemble averages of the single-trial bipolar difference of FC1-C1 electrode signals, as well as the monopolar electrode signals d, Cz and C2 for uncontaminated hits for both studies provided further evidence that the LF-ASD feature set can be used to detect imagined voluntary movements by subjects with spinal cord injuries. The results of this research provide positive initial indications that the LF-ASD can be activated with imagined/attempted movements. This was a fundamental step in the development of the LF-ASD BCI system given that its design was based on two unverified assumptions that 1) imagined movements will have enough similarity with actual movements to drive the LF-ASD, and 2) people with spinal cord injuries can produce consistently detectable imagined voluntary movement-related patterns similar to able-bodied people. The work of this thesis provides further evidence that a BCI system based on the LF-ASD technique may be possible to assist people with a high level of motor impairment to control devices through the preparation or imagination of motor-related tasks. iii CONTENTS ABSTRACT ii LIST OF TABLES viLIST OF FIGURES x ACKNOWLEDGEMENTS xiii CHAPTER 1 INTRODUCTION 1 1.1 Research Goals 3 1.2 Overview 4 CHAPTER 2 BACKGROUND 5 2.1 Existing BCI Techniques2.1.1 BCI Research at the Wadsworth Center 6 2.1.2 The Graz BCI System 8 2.1.3 VEP-BasedBCI2.1.4 P300-Based BCI 9 2.1.5 The Thought Translation Device (TTD) 10 2.1.6 BCI Research at the University of British Columbia 11 2.1.6.1 The LF-ASD 13 2.1.6.2 On-Line Evaluation of the LF-ASD 16 2.2 Use of Imagined Movements in the LF-ASD2.3 Need for Self-Report of Subject Intent 17 iv CHAPTER 3 SYSTEM DESIGN 19 3.1 S ystem Hardware Description 20 3.2 System Version 1 Operation 2 3.3 System Version 2 Operation 6 3.4 Codebook Selection 8 CHAPTER 4 EVALUATION METHODOLOGY 29 4.1 Study 1 Methodology: Able-bodied Subjects 24.1.1 Study 1 Experimental Setup 29 4.1.2 Study 1 Subject Selection 30 4.1.3 Study 1 Method for Data Collection 31 4.2 Study 2 Methodology: SCI Subjects 2 4.2.1 Study 2 Experimental Setup4.2.2 Study 2 Subject Selection 34.2.3 Study 2 Method for Data Collection 33 4.3 Method for Lucky Hit Estimation 4 4.3.1 Problem Overview 35 4.3.2 Detailed Estimation Methodology 37 4.3.3 Computation of Lucky Hits 40 CHAPTER 5 RESULTS & DISCUSSION 2 5.1 Results of Study 1 45.2 Results of Study 2 9 5.3 Overall Discussion 56 CHAPTER 6 CONCLUSIONS 63 6.1 Summary of Contributions 5 6.2 Suggested Future Work 66 REFERENCES 67 APPENDIX A TABLES 72 V A. 1 Glossary of Abbreviations 73 A. 2 Encoding Tables 4 APPENDIX B PROCEDURES, INSTRUCTIONS & FORMS 75 B. l BCI System Apparatus 76 B.l.l Equipment & AccessoriesB.l.2 EEG SuppliesB.l.3 Lab Supplies 76 B.2 Set-Up Procedure for the BCI System 78 B.2.1 Initial Set-UpB.2.2 Subject Preparation 79 B.2.3 Standard Experimental Procedure Per Session 80 B.2.4 Post-Experimental Clean-Up 81 B.2.5 Post-Analysis ToolsB.3 Study 1: Instruction Sheet for the Researcher 82 B.3.1 TestSession 1: lst75Trials 8B.3.2 Test Session 2: 2nd 75 Trials 3 B.4 Study 2: Instruction Sheet for the Researcher 84 B.4.1 Session 1: Familiarization and Customization 8B.4.2 Session 2: 1st 75 Trials 89 B.4.3 Session 3: 2nd 75 TrialsB.5 Consent Form 91 B.6 Subject Information Form 92 B.7 LAT-24 R Handedness Inventory 3 APPENDIX C LUCKY HIT ESTIMATION PLOTS 94 vi LIST OF TABLES Table 2.1: Performance of the ASDs [6] 12 Table 2.2: Optimal Feature Dimensions [6] 5 Table 4.1: Study 1 Participant Information 31 Table 4.2: Study 2 Participant Information 3 Table 4.3: BCI System Outputs 36 Table 4.4: Section 4.3.2 Reference of Notations 37 Table 5.1: Study 1 Modified Online LF-ASD Performance with Imagined Movements by Abie-Bodied Subjects 42 Table 5.2: Study 1 Lucky Hit Estimation 45 Table 5.3: Study 1 Modified Online LF-ASD Performance with Actual Movements 49 Table 5.4: Study 2 Modified Online LF-ASD Performance with Imagined Movements by SCI Subjects 50 Table 5.5: Study 2 Lucky Hit Estimation 53 Table 5.6: Study 2 Modified Online LF-ASD Performance with Only 50 Imagined Movements by SCI Subjects 56 Table 5.7: Max-Min Differences in Amplitude of Averaged Monopolar Signals for all Uncontaminated Hits for the 1-Second Period Centering the Detected Hit.... 61 Table 5.8: Average Input (FCZ) Peak Amplitude 62 vii Table A.2.1: EOG Encoding Index 74 Table A.2.2: EEG Encoding IndexTable B.l: Electrode Montage for the Grass Electroencephalograh EEG Amplifier, Model 8-18 C 78 Table B.2: EEG Amplifier Settings for the Grass Electroencephalograh EEG Amplifier, Model 8-18 Cviii LIST OF FIGURES Figure 2.1: EEG Electrode Placement 14 Figure 3.1: BCI System Components 20 Figure 3.2: System Block Diagram 1 Figure 3.3: LF-ASD Experimental Display 2Figure 3.4: System Version 1 Operation Flow Diagram 23 Figure 3.5: Timeline Diagram of the System Version 1 Operation for Managing the Three Cases of Hit, FP and FN 24 Figure 3.6: Decision Boundary Scaling Diagram 26 Figure 4.1: Classification Process 35 Figure 5.1: Study 1, Able-bodied Subject 1 - Filtered Averages of FCi-Ci, Q and EOG for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker 44 Figure 5.2: Study 1, Able-bodied Subject 2 - Filtered Averages of FCi-Ci, d and EOG for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker 44 Figure 5.3: Study 1, Able-bodied Subject 1 - Trial Length Distribution of (a) Hits, (b) False Negatives, and (c) False Positives, Session 1 (left), Session 2 (right)...46 ix Figure 5.4: Study 1, Able-bodied Subject 2 - Trial Length Distribution of (a) Flits, (b) False Negatives, and (c) False Positives, Session 1 (left), Session 2 (right)...47 Figure 5.5: Study 1, Able-bodied Subject 1 - Distribution of (a) Flits, (b) False Negatives, and (c) False Positives in 3-Minute Bins, Session 1 (left), Session 2 (right)..48 Figure 5.6: Study 1, Able-bodied Subject 2 - Distribution of (a) Hits, (b) False Negatives, and (c) False Positives in 3-Minute Bins Session 1 (left), Session 2 (right). ..48 Figure 5.7: Study 2, SCI Subject 1 - Filtered Averages of FCi-Ci, Ci and EOG for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the O-second marker 51 Figure 5.8: Study 2, SCI Subject 2 - Filtered Averages of FC.-Ci, Ci and EOG for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the O-second marker 52 Figure 5.9: Study 2, SCI Subject 1 - Trial Length Distribution of (a) Hits, (b) False Negatives, and (c) False Positives, Session 1 (left), Session 2 (right) 53 Figure 5.10: Study 2, SCI Subject 2 - Trial Length Distribution of (a) Hits, (b) False Negatives, and (c) False Positives, Session 1 (left), Session 2 (right) 54 Figure 5.11: Study 2, SCI Subject 1 - Distribution of (a) Hits, (b) False Negatives, and (c) False Positives in 3-Minute Bins, Session 1 (left), Session 2 (right) 55 Figure 5.12: Study 2, SCI Subject 2 - Distribution of (a) Hits, (b) False Negatives, and (c) False Positives in 3-Minute Bins, Session 1 (left), Session 2 (right) 55 Figure 5.13: Study 1, Able-bodied Subject 1 - Filtered Averages of Q, Cz and C2 for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the O-second marker 59 Figure 5.14: Study 1, Able-bodied Subject 2 - Filtered Averages of Ci, Cz, and C2 for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the O-second marker 59 Figure 5.15: Study 2, SCI Subject 1 - Filtered Averages of Ci, Cz and C2 for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the O-second marker 60 x Figure 5.16: Study 2, SCI Subject 2— Filtered Averages of d, Cz and C2 for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker 60 Figured: Study 1, Able-bodied Subject 1, Session 1 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution 95 Figure C.2: Study 1, Able-bodied Subject 1, Session 2 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution 96 Figure C.3: Study 1, Able-bodied Subject 2, Session 1 - (a) Smoothed Observed HTT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution 97 Figure C.4: Study 1, Able-bodied Subject 2, Session 2 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution 98 Figure C.5: Study 2, SCI Subject 1, Session 1 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution.... 99 Figure C.6: Study 2, SCI Subject 1, Session 2 - (a) Smoothed Observed HTT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed xi FPDistribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution.. 100 Figure C.7: Study 2, SCI Subject 2, Session 1 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution.. 101 Figure C.8: Study 2, SCI Subject 2, Session 2 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution.. 102 xii ACKNOWLEDGEMENTS The work presented here would not have been possible without the support and contribution of many people. I would like to thank the participants in the pilot and evaluation studies as well as my co-supervisor Dr. Peter Lawrence for his long-term mentorship and guidance. Dan Lisogurski's work provided the base for this thesis project, and I would like to thank him for always responding promptly to my questions. I would like to thank Dr. Robert Hare from the Department of Psychology who made his laboratory available and was generous in loaning the EEG amplifier used in this research, and Michael Young of the British Columbia Institute of Technology for providing clinical support as well as other EEG equipment. I would like to thank Dr. Andrea Townson who assisted in recruiting the spinal cord injured subjects of Study 2 as well as arranging for the set-up of the Neil Squire Foundation lab at the G.F. Strong Rehabilitation Center. I wish to express my gratitude to Dr. Bertrand Clarke from the Department of Statistics who spent time discussing the estimation and probability issues of Section 4.3. I am very grateful to Dr. Steve Mason who gave me tremendous support for understanding the underlying issues and guiding me throughout the experimental tasks as well as defining and formalizing my thesis report. My deepest thanks goes to Dr. Gary Birch for his enthusiastic guidance, constant support and valuable feedback. I would not have been able to complete this work without his patience and understanding. I dedicate this thesis to my family and friends and especially to my mother, Zohreh Nokhasteh, for her unrelenting support and encouragement so that I would complete my studies. ZIBA BOZORGZADEH The University of British Columbia December 2000 xiii In Loving Memory of My Father Dr. Farhad Bozorgzadeh CHAPTER 1 INTRODUCTION The need to understand and improve the quality of interaction between humans and advanced machines has given rise to a relatively new field, namely the Human-Computer Interaction (HCI) [1, 2, 3] . An underlying challenge within the field of HCI deals with establishing new or improved means of communication between users and the computer system. As a result of recent advances in signal processing technologies and increased computing power, novel sensing modalities, such as speech, vision-based gesture recognition, analysis of facial expressions, eye tracking, force-sensing and electroencephalograph (EEG), have been introduced as potential interfaces that can be embodied in a HCI system. Furthermore, integration of more than one modality into an interface would potentially overcome the limitations of unimodal means of communication and allow for a more effective utilization of the information flow. In particular, using a multi-modal HCI would give people with physical or cognitive disabilities better access to computers or computer-controlled devices. In an effort to provide alternative communication channels for people who suffer from severe loss of motor function, much research has been done over the past two decades towards the development of a Brain-Computer Interface (BCI). The term BCI has been formally defined as a "communication system that does not depend on the brain's normal output pathways of peripheral nerves and muscles" [4]. In a BCI system, the brain activity, commonly recorded as EEG data, is received as input and is converted to one or more control signals. The ultimate goal of this research is to develop an improved interface for individuals 1 with a high-level of impairment, such as those with severe stages of amyotropic lateral sclerosis (ALS), multiple sclerosis (MS), or high-level spinal cord injuries. The realization of a BCI would allow them to effectively control devices such as wheelchairs, robotic assistive appliances, computers and neural prostheses. Most BCI technologies developed to date have focused on the problem of distinguishing between a number of commands in a continuous sequence of commands [5]. This class of BCIs meets the requirements of synchronous applications [6] in which the system initiates the period of control and not the user. However, in many practical applications such as turning lights on or off, it is important that the BCI operates in an asynchronous (event-driven) mode where the user issues infrequent commands for monitoring or controlling a process at his or her own pace. A BCI operating in this mode must distinguish between control signals and attentive idle or spontaneous EEG. The Outlier Processing Method (OPM) [7] and the Low-Frequency Asynchronous Switch Detection (LF-ASD) [6] are the only techniques that have been designed for on-line asynchronous control of spontaneous EEG. Mason and Birch's evaluation of OPM indicated relatively high rates of false activations that limited its applicability to the asynchronous control problem [6]. In order to address the need for an effective technique to detect asynchronous control signals from the human operator, they proposed the LF-ASD. The LF-ASD was designed to be driven by imagined voluntary movement-related potentials (rVMRPs). Mason and Birch tested the LF-ASD design with actual, single-trial right-hand index finger flexions and idle EEG. The results of this study demonstrated estimated mean correct classification rates between 66% and 89% by the LF-ASD with the OPM's performance ranging from 64% to 72% [6]. Their method of evaluating the LF-ASD design was based on three assumptions: 1) on-line performance of the LF-ASD will be similar to its off-line performance, 2) imagined movements will have enough similarity to actual movements to drive the LF-ASD, and 3) people with disabilities can produce consistent detectable IVMRP patterns similar to able-bodied people. Lisogurski and Birch [8] tested Mason and Birch's first assumption in an on-line version of the LF-ASD in which finger flexions were monitored and used as feedback for confirming system classification. The effectiveness of the switch design in detecting 2 voluntary movement-related potentials (VMRPs) was validated online giving true positive rates in the range of 50% while maintaining false positives under 10%. Since the long-term goal of this research is to develop an advanced communication interface for people with disabilities, it was essential to prove the effectiveness of the LF-ASD with imaginary movements. It was also important to demonstrate that the LF-ASD can be used with persons with disabilities. Therefore, the goals of this work (see Section 1.1) were chosen to investigate Mason and Birch's last two assumptions: 1) imagined movements will have enough similarity to actual movements to drive the LF-ASD, and 2) people with disabilities can produce consistent detectable IVMRP patterns similar to able-bodied people. A problem that had to be overcome to work with imagined movements was how to measure the user's intent when imaging a movement in an asynchronous environment. To date, the LF-ASD has only been assessed with actual movements since this was a tangible measure of intent and could be obtained immediately from the subject. In order to confirm the detection of self-paced IVMRPs, self-report was used to evaluate system performance. However, self-report could not solve the problem of knowing exactly when the movement was imagined. Therefore, it was quite possible for the subject's intent to overlap with the system's false switch activation. The result is referred to as a "lucky hit" in the context of this research. A method for estimating the lucky hits is proposed in Section 4.3. 1.1 Research Goals The goals of this research were to develop a system for using the LF-ASD with imagined movements and to evaluate the effectiveness of this protocol system with able-bodied subjects and subjects with spinal cord injuries. Another goal of this research was to develop a structured methodology for collecting data from subjects with spinal cord injury since there is a problem with measuring intent with imagined movements. Moreover, a methodology for analysis of the results for estimating the percentage of lucky hits was required. 3 1.2 Overview Chapter 2 presents a summary of relevant research that provided the groundwork for the development of the new system. Chapter 3 provides details of the hardware and software components for the design of the system. In Chapter 4, the methodology for collecting data for the evaluation of the system for the two studies (Study 1 with able-bodied subjects and Study 2 with spinal cord injured subjects) is specified. This chapter also includes a methodology for estimating lucky hits. The results from the two studies are presented and discussed in Chapter 5. Chapter 6 contains conclusions and suggested future work. A list of abbreviations as well as a table of reference for how the EEG was encoded during data collection are available in Appendix A. Appendix B provides a reference for all the procedures, instructions and forms used in this research. Appendix C contains plots that give insight to the lucky hit estimation results given in Chapter 5. 4 CHAPTER 2 BACKGROUND Various studies have revealed that EEG-based communication could potentially provide a new mode of communication for people with disabilities [4, 5]. To date, the methodologies applied can be divided into two major categories: methods using components present in the spontaneous EEG and methods that use EEG components evoked by a specific sensory event or stimulus. BCI techniques can also be categorized depending on the type of application for which they are designed: synchronous versus asynchronous (defined in Chapter 1 and Section 2.1.6). This chapter provides an overview of several existing BCI techniques, with further detail on the development and implementation of the LF-ASD for detecting imagined movements. It then concludes by addressing the issue of measuring user intent in the asynchronous detection of imagined movements. 2.1 Existing BCI Techniques There are currently more than 20 active BCI research groups worldwide [4]. The BCI systems developed to date use either EEG activity recorded from the scalp or the activity of individual cortical neurons recorded from implanted electrodes. Compared to the more expensive and technically demanding magnetoencephalography (MEG), positron emission tomography (PET), and functional magnetic resonance imaging (fMRI) techniques, EEG, which is easily recorded and processed with inexpensive equipment, appears to offer the 5 practical possibility of a new non-muscular and non-invasive communication channel. Another particular advantage of EEG is that at low sampling rates (typically between 64 Hz to 200 Hz), it offers sufficient resolution for feature extraction and pattern recognition. Nevertheless, the EEG signal is an extremely complex signal and the features of interest within the EEG signal are highly variable (inter- and intra-subject) with a low signal-to-noise ratio (SNR). Signals in the EEG not related to the control signals may be considered noise. EEG noise may result from a non-neural source such as eye movement and muscle movement or it may result from a neural source. EEG data contaminated with eye blinks or other strong eye movements are typically rejected. The most common technique in removing noise from a neural source involves averaging multiple time-locked trials. Ensemble averaging can be used to compile templates of the desired EEG features during the training phase or during off-line analysis, but in the on-line implementation of the BCI system, it is more practical and efficient to extract the desired EEG features on a single-trial basis, rather than demanding multiple repetition of a single command and being limited to a trigger for time-locking the trials. Effective single-trial feature extraction is one of the major challenges facing BCI researchers. Current BCIs are relatively low bandwidth devices designed mostly for synchronous applications. Like any communication or control system, a BCI system is comprised of an input, an output and a translation algorithm. The input to a BCI system consists of a particular feature (or features) of brain activity and the methodology used to measure that feature. Through a series of signal processing computations, the translation algorithm converts this input into an output, which can be a cursor movement, letter or icon selection, neuro-prosthesis operation or another form of device control. BCI systems vary significantly from one another based on their inputs, outputs and translation algorithms as well as the type of application for which they are designed. This section provides a brief overview of a number of BCI systems developed to date. 2.1.1 BCI Research at the Wadsworth Center The Wadsworth Center, one of the leading BCI research centers, has conducted several studies towards the development of a BCI system that focuses on using the mu (8-12 Hz) 6 rhythm or beta (13-28 Hz) rhythm frequency bands over the sensory-motor cortex to move a cursor to a target on a computer screen in one or two dimensions [9]. The performance of the system is evaluated in a synchronous control environment based on whether the subject can move the cursor or not during system-directed trials. Cursor movement in each direction is controlled as a linear function of EEG amplitude. Wolpaw and McFarland [10] report single-trial accuracies ranging from 51% to 94% across 16 subjects (three spinal cord injured and two ALS subjects) with various training on one-dimensional (1-D) cursor control. Here, 50% represents chance performance (single-trial accuracy expected in the absence of any EEG control), and accuracy implies the subject's ability to move the cursor towards the target indicated on the monitor by the system. There is no mention of how subjects with motor disabilities performed in comparison to able-bodied subjects. It should be noted that for six of the 16 subjects who had completed at least 20 training sessions, the four to five later sessions from each subject were used to report the results. The other ten completed ten sessions, and sessions nine and ten were used. Subjects varied greatly in their learning rates and accuracies higher than 90% were expected to take several months to develop [11]. In another study [10], three subjects, who had previously mastered 1-D control, were trained on two-dimensional (2-D) cursor control and were able to achieve accuracies of 60% to 65% with four targets, where single-trial accuracy expected in the absence of any EEG control, i.e., chance performance, was 25%. The subjects' five to six later sessions were used in reporting these results. It was not reported how many training sessions in total were conducted for the 2-D control. In a different study, the 1-D cursor control was used to answer simple yes/no questions in a two stage Response Verification system involving 1) item selection, and 2) confirmation of selection [12]. Four individuals, including one with early-stage ALS, were asked an average of 4.0 to 4.6 questions per minute in a synchronous communication application. This contrasts to a rate of about 16 questions per minute when the answers are spoken. With redundancy incorporated in the system (i.e., two stage verification), accuracies in the range of 60% to 86% were achieved, with 50% representing chance performance. 7 It is emphasized that these systems have only been evaluated in a synchronous control environment, and there is no report of the accuracy rate of the system in detecting the subject's idle or do-nothing mental states. 2.1.2 The Graz BCI System At another leading BCI research center, the Graz BCI system, developed by Pfurtscheller and his colleagues [13, 14, 15] also uses spontaneous EEG activity but they monitor the changes in amplitude in the mu rhythm and other frequency bands associated with preparation of specific movements. In the prototypes of this system [13], the subject was asked to press a microswitch with his/her left index finger, right index finger or move the toes of the right foot upwards upon presentation of a stimulus cue on a computer monitor. The result of the detection was then fed back to the subject via a symbol presented on the monitor during system-directed trials. The performance of this EEG-based cursor control system has been examined in detecting left and right hand motor imagery [14]. The EEG was recorded from electrodes overlying the sensory-motor areas, and the signals were analyzed in subject-specific frequency bands and classified on-line using the LVQ [16] neural network. With the output of the network provided as feedback at the end of each trial (delayed feedback), long-term experimental series with four subjects were carried out resulting in minimum on-line error rates ranging from 10.0% to 38.1% over the last 14, 12, 11 and 3 training sessions of the four subjects respectively. Here, the term error rate is used ambiguously in the sense that 100% implies perfect classification. Trials that neither matched left or right hand motor imagery were rejected. The Graz BCI system has also only been evaluated in a synchronous control environment, and there is no report of system performance during subjects' idle mental state. 2.1.3 VEP-Based BCI Sutter [17] discusses an experimental communication system for severely disabled persons that uses the visual evoked potentials (VEPs) recorded from the visual cortex of the brain of 8 the subject in order to determine the subject's selected target (one out of 64 symbols) when gazing at an 8x8 grid on a CRT screen. The symbols (rectangular target fields) would alternate rapidly between red and green in different sequences in order to induce clearly detectable signals on the scalp. The correct symbol was identified based on the correlation between the VEPs and the templates produced by averaging during the training phase. The prototype system was tested with over 70 able-bodied and approximately 20 severely disabled persons. With able-bodied subjects, response times ranging from 1 to 3 seconds were achieved after an initial tuning process of 10 minutes to 1 hour. One ALS patient was able to reach communication rates of 10 to 12 words/minute using implanted electrodes (mean access time of approximately 1.2 s). Sutter's system is designed for synchronous communication applications and is limited in its applicability as a generic interface system. The interface requires a structured environment with the subject's complete attention and a high level of eye-control. 2.1.4 P300-Based BCI The P300 is an event-related brain potential (ERP) elicited by rare, task-relevant events and has a latency of approximately 300 milliseconds. The amplitude of the P300 varies directly with the relevance of the eliciting events and inversely with the probability of the stimuli. Farwell and Donchin [18] describe the development of a system that uses the P300 component of the ERP produced by the subject when he or she would focus on a computer screen to select one of the 26 letters of the alphabet from a 6 by 6 matrix. Four subjects participated in a study that was conducted in two sessions. The EEG was recorded from the parietal (Pz) site (Figure 2.1). The first session served to assess the feasibility of the technique and to familiarize the subjects with the apparatus and procedures. The data obtained in the second session was used to study the characteristics of the proposed system. Based on the results of this study, the developed system could serve as a keyboard emulation system providing a "suggested" mean communication rate of 12.0 bits, or 2.3 characters, per minute with 95% accuracy. This claim was based on an off-line bootstrapping approach that compared speed versus accuracy using four different algorithms and averaging the best response for each subject. 9 A new version of this system [19] was implemented and ten able-bodied and 4 disabled subjects participated in a study where they were instructed to count the number of times the row, or the column, containing the designated letter "P" was intensified (2/12 per trial with each trial defined as the intensification of all 6 rows and 6 columns of the matrix). The P300 component was extracted from EEG recorded from the fronto-centro-parietal sites (Fz, Cz, and Pz - see Figure 2.1). Under the assumption that compressing all trials in an iterative sampling (bootstrapping) approach would give comparable on-line performance, they would determine the minimum number of trials required to obtain 80% or 90% accuracy rates. Estimated mean communication rates of 7.8 characters per minute and 4.8 characters per minute were "suggested" for achieving 80% and 90% accuracy rates respectively. An online study with 5 able-bodied subjects was also conducted [19]. The bootstrapping approach was used to determine the optimum trial length for obtaining 90% accuracy. With the system designed to receive 90% accuracy, the BCI was able to correctly identify the cell selected by the subject on 56% of the trials. Subject specific performance was not available. As in Sutter's system described in 2.1.3 above, the suggested interface is not practical as a generic interface system as it imposes a structured environment requiring the subject's full attention and concentration and assuming a reasonable level of eye-control. 2.1.5 The Thought Translation Device (TTD) Birbaumer and his colleagues [20] have been able to use the subject's slow cortical potentials (SCPs) of their EEG in a 2-second rhythm, producing either cortical negativity or positivity according to the task requirement. SCP differences, primarily measured at the vertex (Cz -see Figure 2.1), between a baseline interval and an active control interval are linearly transformed into vertical or horizontal cursor movements on a computer screen. They demonstrate that ALS patients can use this technique for conducting a binary choice through the alphabet in order to select letters or words from a language support program. In order to achieve their goal, their patients had to fixate and monitor a screen over fairly complex and slow moving conditions. After prolonged training of over hundreds of sessions, three ALS patients were able to achieve self-control. The training period, however, is not specified. 10 Mean percentage accuracies (correct classification rates) for the binary synchronous selection of tasks based on the last 20 sessions for these three patients were reported to be 86.7%, 46.2%, and 66.1% for the selection task with corresponding 51.5%, 74.0%, and 76.2% for the rejection task respectively [20]. Two of these patients can now write their own letters by using the TTD at an average speed of 2 minutes per each letter selection [21]. Since the subject must always maintain control on his or her SCPs to select or reject a letter, there is no report of system performance during the subject's idle mental state. A potential disadvantage of the use of SCPs should also be noted in that there is no available data regarding the maximum accuracy of self-control of the SCP, and it may be that the ability to change SCP "on command" could be limited by some protective mechanism that prevents the cortex from voluntary over-excitation. 2.1.6 BCI Research at the University of British Columbia In most existing BCI systems, including those given as examples in Sections 2.1.1 through 2.1.5, the performance is assessed based on the synchronous detection of control signals in response to cue stimuli presented by the supervisory system. There is no discussion of idle or "no movement" state classification when evaluating these BCI control or communication techniques. However, it is equally important to assess the capability of a BCI system in accurately detecting idle data when the subject is not focusing on activating a particular device, i.e., asynchronous signal detection. Practical applications of this design can be seen through the control of daily activities such as turning lights or any electrical appliance on or off. The BCI research group at the University of British Columbia1 in affiliation with the Neil Squire Foundation2 has focused on developing a direct BCI system that meets the requirements of asynchronous control applications [22]. Mason and Birch [6] compared three Asynchronous Switch Designs (ASDs) in classifying EEG for asynchronous applications. The Outlier Processing Method (OPM) [7] and the mu Event-Related Desynchronization (mu-ERD) [16, 23] are two BCI techniques 1 Department of Electrical and Computer Engineering, University of British Columbia, 2356 Main Mall, Vancouver, B.C. V6T 1Z4 Canada www.ece.ubc.ca 2 Neil Squire Foundation Head Office: Suite 220 - 2250 Boundary Road, Burnaby, B.C., V5M 3Z3 Canada www.neilsquire.ca 11 adapted as signal detectors in Mason and Birch's work. The OPM is based on the premise that EEG activity measured from the scalp can be modeled as the summation of event-related potentials (ERPs) and statistically independent background EEG activity. In this approach, a generalized robust maximum likelihood estimate is utilized to provide a robust estimate of the ongoing, underlying EEG process, which in turn is subtracted from the original EEG activity to yield an estimate of the outlier potential. The time series of outliers produces waveform patterns that provide single-trial event-related information. In the mu-ERD technique, the feature set is based on the desynchronization in neural activity, which starts about 2 seconds before the motor act over the contralateral hemisphere and becomes bilaterally symmetrical with movement execution. The ERE) is measurable by a decrease in signal power for the mu (8-12 Hz) frequency band involved. Mason and Birch also defined a Low-Frequency Asynchronous Switch Design (LF-ASD) that would continuously sample surface electrodes spatially distributed over the motor areas of the human cortex. Since voluntary movement control is an existing, internal control system in humans, which seemed naturally suited to drive a BCI, the LF-ASD feature set was based on voluntary movement-related potentials (VMRPs). The LF-ASD performed the best among the techniques they studied. The mean range of correct classification rates for the three ASD designs (LF-ASD, OPM and mu-ERD) across all five subjects that participated in Mason and Birch's evaluations study are presented in Table 2.1. Table 2.1: Performance of the ASDs [6] ASD Design Mean Range of Correct Classification Rate LF-ASD 66% - 89% OPM 64% - 72% MuERD 49% - 66% Mason and Birch's experimental investigation of the spatiotemporal signal features in the 1 to 4 Hz frequency band demonstrated that the LF-ASD design has the ability to differentiate between right index finger flexions from idle EEG on a single-trial basis with reasonably low error rates (see Section 2.1.6.1 for performance values). 12 The experimental evaluation of the LF-ASD was performed offline. As a result, performance of the LF-ASD in reaction to the operator's other cognitive tasks was not known. Moreover, without an on-line implementation that provides feedback, the effects of subject training could not be studied. Lisogurski and Birch [8] revised and produced an on line implementation of the LF-ASD capable of identifying VMRPs from a continuous sampling of EEG. The revised ASD was further validated on two able-bodied subjects performing actual index finger flexions and gave promising results (see Section 2.1.6.2 for performance values). The works of Birch, Mason and Lisogurski provided the foundation for this thesis project. The following sub-sections describe their contributions in more detail. 2.1.6.1 The LF-ASD The Low Frequency Asynchronous Switch Design (LF-ASD) [24] is comprised of a Feature Extractor, a Feature Classifier and a Decision Module. The Feature Extractor continuously extracts the signal characteristic feature set from the filtered EEG data, and the Feature Classifier classifies the feature set as either active or idle depending on whether a control signal is detected or not. The Decision Module modifies the output of the Feature Classifier as described below. The LF-ASD design was based on bipolar electrode pairs, which were sampled at 128 F£z and band-pass filtered from 1-4 Hz. The feature values were defined by Equation 1.1. / \E,(n)-E,(n) Ei(n)>0and £,(«)> 0 n gij(n) = \ . (El) [ 0 otherwise where the elemental features, E* and Ej, were defined by El(n) = eki(n + ai)-eki(n + 0i), i = l,2,...,M (1.2) EAn) = ek.{n + a.)-ekXn+B,-),j = 1, 2,...,M (1.3) where ek(n) is the kth observed, bipolar EEG signal, n indicates discrete samples of time, ai, (3j, oij, and Pj are system delay parameters, and M is the number of features evaluated. The sub-subscript i used in ek (n) associates the pair of delay parameters a; and Pi to the bipolar signal ek(n) and similarly for subscript j. The notation does not imply that different bipolar signals were used in obtaining the pair of E;(n) and Ej(n). 13 In order to increase the robustness of the signal detection to trial-by-trial latency variation, these feature values were collapsed over l/8th of a second into the aggregate features defined by GiJ(n) = max(giJ(n-&),gij(n-7),...,gij(n+l)). (1.4) where max() represents the maximum. The output of the LF-ASD was an equally weighted, six dimensional vector (i.e., M = 6 in Equations 1.2 and 1.3), with each dimension reflecting the value of an aggregate feature. This feature vector was generated from six symmetrical electrode pairs, Fi-FCi, FZ-FCZ, F2-FC2, FCj-Ci, FCZ-CZ, and FC2-C2, positioned over the Supplementary Motor Area (SMA) and Primary Motor Area (MI). The electrode pairs over the SMA and MI are indicated in Figure 2.1. Figure 2.1: EEG Electrode Placement. Feature set electrodes shaded. The subject's nose is at the top. The LF-ASD was initially evaluated on five able-bodied subjects. The optimal feature dimensions (the delay parameters a;, pi, Oj, and fy), as shown in Table 2.2, were selected based on one subject and good results where obtained when applied to all five subjects. Performance improved when the delays were customized for each individual. The 14 feature delay values for common electrode pairs were constrained to be equal over each region in order to generalize the switch detection operation to all types of movements. Table 2.2: Optimal Feature Dimensions [6] Feature Fin) EM ek.(n) Pi * ek.(n) CCj* Pr fi F,-FC, -1 +25 FrFCi 0 +50 h FZ-FCZ -1 +25 FZ-FC2 0 +50 h F2-FC2 -1 +25 F2-FC2 0 +50 U FC1-C1 -1 +15 FQ-C, -12 +30 h FCZ~CZ -1 +15 FCZ~CZ -12 +30 h FC2-C2 -1 +15 FC2-C2 -12 +30 Units for delay parameters are (discrete time) samples, with the sampling rate of 128 Hz, n=l/128th second The LF-ASD performed a sample-by-sample classification of each feature vector every l/16th of a second using Learning Vector Quantization (LVQ3) [16] with three vectors per class (there being two classes: active and idle). LVQ3 is basically a nearest-neighbour classifier that, through an iterative learning procedure, generates a labeled codebook in such a way that the nearest codebook vector for each training example is of the same category as the given example. Since the optimal classification rate was not known, it was believed that the classifier output contained redundant temporal information. Therefore, the classifier output was modified by a Decision Module, which would take the moving average of the input over a window of size five that was experimentally determined. The LF-ASD was evaluated on five able-bodied subjects who were asked to change the direction of the center ball in the "pong" style display (shown in Figure 3.2) by executing a non-standard right-hand index finger flexion. Off-line evaluations demonstrated hit (true positive) rates in the range of 38% to 81% with corresponding false positive error rates in the range of 0.3% to 11.6% on this asynchronous signal detection task. False positive errors are defined as the identification of motor activity in idle EEG, and a hit is a correctly classified motor potential. Estimated mean correct classification rates for this technique were reported in the range of 66% to 89% [6]. 15 2.1.6.2 On-Line Evaluation of the LF-ASD Lisogurski and Birch [8] revised the LF-ASD by lowering the overall system delay and implemented an on-line version of the LF-ASD, where EEG was continually classified for the same control task defined above, except during periods where ocular artifact was detected. The on-line system was tested on two right-handed male subjects who each participated in three sessions. During each session, the LF-ASD was trained on 25 (artifact-free) movements and tested on 75 movements during the operating phase. The 25 training movements were used to compile a codebook for the LVQ3 classifier that determined the decision boundaries between the active and idle classifications for the operating phase. The LF-ASD continuously monitored and classified the EEG, and the subject received visual feedback one second after a detected movement or if a false positive occurred. The on-line performance with the two new subjects demonstrated hit rates in the range of 50% while maintaining false positives to a very low level (for Subject 1: between 1-2% and for Subject 2 between 3-6% for the first and third sessions and 10% for the second session). These results were obtained using the feature delays of the same subject that Mason and Birch [6] had used across five subjects, and it is believed that through on-line customization of the feature delays for each subject the performance of the LF-ASD can be further improved. The on-line implementation would time-lock the self-paced movements made by the subject via a custom-made glove, which was built for Mason and Birch's experiment, to measure finger flexions using piezoelectric sensors located over the knuckles [25]. Through the glove feedback, the system could assess the accuracy of the on-line classification. For the detection of imagined movements, this input was no longer necessary. However, as there is no physical indication of the onset of the mental activation causing an imagined or stimulated movement, the system must obtain feedback through another means. 2.2 Use of Imagined Movements in the LF-ASD The results obtained to date show that the LF-ASD is capable of detecting motor potentials in single trial EEG and thus provide the necessary ground for advancing towards the next stage 16 of the research, which is to develop a system to test the ability of the LF-ASD in detecting imagined motor potentials (see Chapter 3). It was interesting to note that in Lisogurski and Birch's on-line study described in Section 2.1.6.2, the subjects reported that many of their false positives were detected when they thought about making a movement but did not execute one. This provided some very preliminary evidence that it may be possible to activate the LF-ASD simply by planning a movement. This observation is also consistent with the recent work by Pfurtscheller [14, 26] where their subjects used only imagined movements. Through quantifying the ERD, which is characteristic of a voluntary movement, Pfurtscheller and his colleagues have been able to discriminate between various types of electrical brain activity patterns related to movement planning. The results of this study support the general hypothesis that predictable EEG patterns can be extracted over the contralateral sensorimotor hand area when unilateral hand movements are imagined and that the spatiotemporal pattern related to the imagination of the movement is similar to the spatiotemporal pattern during the planning or preparation of a voluntary movement. Babiloni [27] also shows the similarity between the distributions of the spatial filters, computed by means of the Signal Space Projection, between imagined and actual hand movements. Cunnington et al. [28] have also observed that the early component of movement-related potentials (MRPs) does not differ in amplitude, onset time, or topography when imagining a movement compared to actual movement, indicating that similar pre-movement preparatory processes generated in the same cortical area is involved in both movement execution and motor imagery. 2.3 Need for Self-Report of Subject Intent As stated in Section 2.1, most BCI systems have been designed for synchronous control applications and are system-directed (versus self-paced). In system directed environments, the system's functionality can be easily assessed since the success or failure of the system is defined based on the subject's response within a controlled window. Such a setting eliminates the need for subject self-report and alleviates psychological pressure on the subject who is not only compelled to perform well but must also report accurately. 17 The focus of this work is towards developing a system that can effectively operate in an asynchronous control environment. However, signal analysis of self-paced imagined movements in this type of environment is difficult because one needs an indication of intent in order to process the data. This appears to be the underlying reason that the design of BCIs for asynchronous control applications has received very little attention so far. Synchronous communication applications do not have this difficulty since user intent is assumed during the communication periods defined by the system. In self-paced asynchronous applications, human intentions are not directly observable, and the subject's intent can only be established through the human's explicit expression of his intentions, or through inferences made from his observable actions [3, 6]. The participant's self-report is thus required to differentiate between true switch activations (hits) and false switch activations (false positives) as well as to report those intended movements that were not detected by the system (false negatives). Consequently, the evaluation results of Chapter 5 are entirely dependent on the subject's accurate judgment and consistent report as well as his or her level of attention and focus throughout the evaluation sessions. It is important that the task of self-reporting be chosen such that it does not interfere with the task under study. However, it is difficult to choose a feedback mechanism for a self-paced setting that does not involve some form of motor function in order to avoid conflict with the activation mechanism of a BCI system that is based on motor potential patterns. Chapter 3 provides details of what type of self-report methodology was chosen and how it was incorporated in the design of the system. 18 CHAPTER 3 SYSTEM DESIGN As described in Chapter 1, a new system was needed for detecting imagined movements made by subjects with spinal cord injuries. The structure of the system that was developed for this purpose using the LP-ASD design (Figure 3.1) is introduced in this chapter, with specific detail on the hardware and software components. In the proposed system, the user's amplified brain activity is converted by the LF-ASD into a control signal that bears information regarding the user's Mental Process pattern (e.g., IVMRP versus non-IVMRP). The System Operation Algorithm uses the output of the LF-ASD as well as an additional user input for reporting intent (via the Sip & Puff Switch) to control the Experimental Display. The Experimental Display provides visual feedback (GUI Feedback) to the user regarding system behaviour and prompts for user input when necessary. Two versions of this system were developed. The first version was tested in the first study with able-bodied subjects. Based on the outcome of this study, the System Operation Algorithm was slightly modified and improved, resulting in a second version. The second version was tested in a study with subjects with spinal cord injuries. The issue of codebook selection is addressed under a separate subsection. 19 Experimental Display EEG Amplifiers Feature Extractor Feature Classifier Decision Module System Operation Algorithm BCI Software (PC) Figure 3.1: BCI System Components 3.1 System Hardware Description The block diagram of the new system is shown in Figure 3.2. The User's EEG and EOG signals were amplified with a Grass Model 8-18C EEG amplifier. The amplifiers have a differential input impedance of 20 Md and a CMRR of greater than 10,000:1 at 10 Hz. A Grass Electrode Impedance Meter, Model EZM1E, was used to check all scalp electrode impedances (see Section 4.1.1 for actual settings). The analog filters in the amplifier were set to a passband of 0.1-35 Hz, except for the EOG channel, which was filtered between 1.0-35 Hz. All signals were sampled at 128 Hz by a PC equipped with a 12 bit Data Translation DT 2801-A analog to digital converter [29]. A consistent sampling rate was achieved using the hardware clock on the A/D board, which transferred samples to the PC over the ISA bus using DMA. This had been verified by sampling a 16 Hz 100 uV peak-to-peak sine wave while the software was in full operation [25]. The same sine wave was used to calibrate the offset for each channel and scale the signals to uV. 20 X-Windows Workstation User EEG Electrodes EOG Electrodes Sip & Puff Switch EEG Amplifiers & Filters Amplifier to ADC Interface Ethernet PC DT2801-A A/D Converter BCI Software Hard Disk Researcher V , J f A Monitor V . J Figure 3.2: System Block Diagram An AMD-K6 233 MHz personal computer running the Linux 2.0.30 operating system was used to acquire and process the data simultaneously. The raw EEG data was stored on disk for post-processing. The software (written in C) controlled a diagnostic display for the researcher and the Experimental Display (Figure 3.3), which appeared on a second monitor on the X-Windows workstation, for the user. Figure 3.3: LF-ASD Experimental Display 21 The Experimental Display was similar to that used by Lisogurski and Mason, who had developed a simple pong-style video game in order to keep the user (participant) attentive throughout the data collection process. Both balls were contained within the double-border box and moved at a constant and moderate speed of approximately 4 cm/s. The box occupies a visual angle of 3.5° from top to bottom in order to minimize eye movements. One ball moves freely within the rectangle bouncing off the walls and the other ball (center ball). The controlled center ball is constrained to move along either a horizontal or vertical path passing through the center of the box. In the design of the new system, many components were modeled after Lisogurski's system [25] in order to minimize the changes between Lisogurski's study of LF-ASD on VMRPs and the study of LF-ASD on IVMRPs in this research project. The main hardware difference required replacing the glove feedback, which monitored index finger flexions, with a custom-made sip and puff switch, built by Neil Squire Foundation Research and Development Group. This feedback mechanism was used to obtain self-report from the user for self-paced imagined movements, thus confirming user intent. The sip & puff switch design was based on a simple voltage divider producing 2.5 V for idle, 1.66 V for a sip action and 3.33 V for a puff action. In order to simplify the system operation, as described in the next section, either a sip or a puff was used to reflect a negative report on behalf of the user. 3.2 System Version 1 Operation The flow diagram of the system operation developed for detecting imagined movements for this research is shown in Figure 3.4. There are basically three cases within the system operation that must be recognized and managed: 1) true positives (TPs or FUTs), 2) false positives (FPs) and 3) false negatives (FNs). True positives or hits refer to desired switch activations corresponding to an imagined movement. False positives (FPs) are false switch activations that the system reports but are not intended by the operator. False negatives (FNs) occur when the system does not respond to the user's attempt to imagine moving his/her index finger. 22 Legend: fb_cnt = Feedback mode counter reject_cnt = Reject mode counter sp = Sip & Puff Switch flag switch_det = Switch detected flag Obtain EEG sample Obtain EOG sample Filter EEG Obtain Sip & Puff sample Check with respect to threshold Switch to REJECT Mode Count HIT Unfreeze balls reject_cnt = 1.0 s i yes-Check with respect to threshold If sip or puff, sp = ON; otherwise, sp = OFF OPERATE REJECT Check EEG for acceptable EOG: 6.0 s back in time Count FP fb_cnt = 0 switch_det = OFF reject_cnt = 1.0 s Flash borders reject_cnt + 1.0 s Switch to REJECT Mode switch_det = ON Flash balls reject_cnt = -5.5 s fb_cnt = 4.0 s Switch to OPERATE Mode Switch to FEEDBACK Mode Count FN with pre-contamination reject_cnt = 1.0 s Switch to REJECT Mode Count IDLE Figure 3.4: System Version 1 Operation Flow Diagram In the process of handling the three cases of HTTs, FPs and FNs, the system functions in one of these three modes: Operate, Reject and Feedback. A better depiction of how the 23 modes interact in each case is presented in form of a timeline diagram in Figure 3.5. Each instance of HTT, FP and FN was defined as a trial. The start and end of trials are indicated in Figure 3.5. Start of Trial Hit End of Trial Switch Activated No Report by Operator During Feedback Reject Mode Operate Mode 1 sec -5.5 sec -xx Delay Reject Mode GUI Feedback 4 sec 1 sec -xx-Feedback Mode Reject Operate Mode Mode Start of Trial FP End of Trial Switch Activated Sip or Puff Report by Operator During Feedback x-Reject Mode 1 sec -5.5 sec Operate Mode Delay Reject Mode GUI Feedback -x-1 sec XX Feedback Reject Operate Mode Mode Mode Start of Trial End of Trial Sip or Puff Report by Operator FN X-Reject Mode Operate Mode 1 sec X-X^ Reject Mode Operate Mode Time Figure 3.5: Timeline Diagram of the System Version 1 Operation for Managing the Three Cases of Hit, FP and FN 24 In the Experimental Display, the LF-ASD switch activations are indicated by the flashing of the center ball and its change of direction from vertical to horizontal and vice versa, which is referred to as GUI Feedback in Figures 3.1 and 3.5. The GUI Feedback Period is approximately 5.5 seconds long. This length was chosen so that there would be enough time for the center ball to change its direction after the flashing period. The Reject Periods after the Feedback Period and the sip and puff switch activations were arbitrarily chosen to be one second each. Due to the self-paced nature of the system, the user's feedback in confirming the accuracy of the system is required. Therefore, after the GUI Feedback, the system provides a four second Feedback Period in which the balls stop moving, and the subject either reports a false positive by activating the sip and puff switch or do nothing indicating a true classification, i.e., imagined movement. False negatives are reported by activating the sip and puff switch while the balls are moving. The design put forward did not use subject-specific customized parameters, and the feature set design was generic and applicable to various VMRPs. As a result, some switch activations would occur within one second prior to a sip or puff action when the subject would report a false negative. As the subject was instructed to report false positives only during the Feedback Mode, where the balls are stationary, these false positives were classified as self-report anomalies and were not followed by a Feedback Mode. Since the onset of the Mental Activity leading to a switch activation was not known and to maintain an unconstrained environment, the EEG was classified despite the presence of ocular artifacts. The EOG state was marked during system classification so that the effect of ocular artifact in the operation of the Feature Classifier could be studied in post-processing as well. As an extra option, if the switch activations resulting in false positive reports contained ocular artifact, the double border of the Experimental Display (Figure 3.3) would flash at the end of the Feedback Period so that the user would become aware of the possible external factors leading to false activations. During data recording, the state of system operation was encoded and saved with the raw data as an extra channel of information for post-processing the data. Appendix A.2 contains the codes used to encode the system state. 25 3.3 System Version 2 Operation From the results of Study 1 (see Chapter 5 and refer to Chapter 4 for a description of the Study 1 methodology), the classifier decision boundary setting resulted in a relatively high false positive rate, which in return caused undue user frustration. Also, lengthy delay and Reject Periods exaggerated this frustration. Therefore, in order to reduce user frustration, the System Operation for Study 2 was a modification of System Version 1 Operation described in Section 3.2. These changes, which are explained below in further detail, included modifying the LVQ classifier, removing a delay period, correcting the resetting of the display and adjusting the length of the Reject Periods. The LVQ classifier modification involved making the Feature Classifier decision boundary variable into an operator controllable variable so that it could be adjusted to an appropriate setting for each subject. This allowed for achieving a reasonably low false positive rate, which was arbitrarily chosen as 1% over 30 seconds. Lowering the false positive rate implies lowering the overall switch activations, so consequently, the hit rate is reduced. However, lowering the system's false activations, reduces the subject's frustration, which is more important than achieving a high hit rate. The classifier was modified so that the distance between the extracted features and the codebook features was scaled by a factor determined by the decision boundary scale (DB Scale) set by the researcher (see Figure 3.6). The default setting (Version 1) was chosen at half of the maximum DB Scale (Max DB Scale). Default Increasing FPs Idle Decreasing FPs Active 0 Max DB Scale / 2 Max DB Scale Figure 3.6: Decision Boundary Scaling Diagram Increasing the DB Scale biased the classifier towards lower false positives by increasing the distance of the active codebook vectors. Decreasing the DB Scale biased 26 towards higher false positives by increasing the distance of the idle codebook vectors. Since it is expected that hit rates will increase with subject training, selecting a DB Scale setting that minimizes the false positives could provide a potentially better starting point for a BCI system. For each case of idle and active codebook vectors the scaling factor was determined as follows: • Idle Codebook Vector (Idle): _ ,. r Max DB Scale 12 , ,~ 1N Scaling factor = , ana J . 1; DB Scale Active Codebook Vector (Active): _ ,. . MaxDBScale/2 n^ Scaling factor • Max DB Scale - DB Scale Originally, the intent was to keep the structure of the experiment as close as possible to that devised by Mason and followed up by Lisogurski, and both had allowed a one second delay between the movement and subject display feedback (flashing of the balls - GUI Feedback) to prevent contamination of the signals with visual evoked potentials (VEPs) caused by the change in the display. As a result, for post processing the signals, clean VMRPs could be studied without the effect of artifacts. Therefore, in Version 1 used in the first study, the balls would start flashing one second after a switch activation. However, it was realized that for identifying pre-movement planning or imagining a movement, the delay was not only unnecessary, but it seemed to confuse the subject who would try to define a strategy for identification of those activations corresponding to his/her intent to move. The system already had a built in signal processing delay of 640.5 milliseconds [25], and since in an online system the goal is to minimize the system delay as much as possible, the removal of the one second delay period after the switch activation (see Figure 3.5) was justified for Study 2. From results obtained from running able-bodied subjects (Section 5.1) and the SCI pilot study subject, it was found that the majority of the false positives would occur within the one to two seconds after the Feedback Period was complete. In examining the cause, it was found that there was a mistake in the system code in resetting the screen after the 27 Feedback Period. The screen was to be reset so that the balls would start moving at the end of the Feedback Period and then followed up with a one second Reject Period so that the effects of the VEPs are disregarded. However, in Version 1, the resetting of the screen had been done at the end of the Reject Period. Therefore, the LF-ASD was activated at the onset of the change in display. Again, as the mistake had been realized and in order to help reduce the false positives possibly related to the VEPs, the correction was justified for Study 2. Additionally, to remove any effect of VEPs due to the change in display and to further account for the shift in the subject's attention, the Reject Periods following the Feedback Period, which originally had been arbitrarily set at one-second periods, were increased to two seconds. 3.4 Codebook Selection The Feature Classifier component of the LF-ASD (Figure 3.1) uses the active and idle feature vector information in an LVQ codebook to classify active versus idle EEG data. For the detection of VMRPs, Lisogurski's system [8, 25] would train an LVQ codebook based on 25 actual movements and then test the performance of the system using the codebook obtained per subject and per session. However, as the onset of mental activation for imagining a movement is unknown, and IVMRPs cannot be time-locked for training the LVQ algorithm, the system had to be initialized and operated with an existing LVQ codebook. The existing LVQ codebooks for this research were those obtained from Lisogurski's on-line results and were tested during pilot studies with able-bodied subjects. In comparison, the codebook obtained from Subject 2 Session 2 performed better and was thus chosen as the base codebook for this work. This particular codebook resulted in a hit rate of 49.3% with 90.4% of the idle EEG classified correctly for the subject for whom it was trained. The software currently has the option of storing features of uncontaminated hits as well as idle points, which can be used to train a new codebook customized for the user based on the session's results. However, training a new codebook based on IVMRPs and codebook customization are issues that require extensive investigation and are recommended for future studies of the LF-ASD. 28 CHAPTER 4 EVALUATION METHODOLOGY To verify the performance of the system designed in Chapter 3, two studies were conducted. The first study involved testing Version 1 of the system (described in Section 3.1) on two able-bodied subjects using the evaluation methodology of Section 4.1.3. This study was a logical extension from Lisogurski's experiment with actual index finger flexions with able-bodied subjects. In the second study, along with a slightly modified system operation, Version 2 of the system (described in Section 3.3), a more structured evaluation methodology was devised (Section 4.2.3) so that the performance of the system with subjects with spinal cord injury could be assessed. These studies were approved by the Department of Research Services at the University of British Columbia (B89-279). 4.1 Study 1 Methodology: Able-bodied Subjects The details of the experimental setup, subject selection criteria and data collection methodology for Study 1 are described in this section. 4.1.1 Study 1 Experimental Setup The participant was seated, in a darkened room, with eyes approximately 100 cm from a black and white monitor with the Experimental Display as shown in Figure 3.3. The participant was asked to remain motionless during the experiment except during the 29 Feedback Periods. The participant wore an ElectroCap™ [30] electrode cap connected to the EEG amplifier referred to in Chapter 3. The fourteen (14) monopolar electrodes, which conform to the International Ten-Twenty System [31], are shown in Figure 2.1. Oz was used as reference, and POz was used as a grounding node. Each electrode was encased in a plastic holder. The cap was placed on the subject's head with reference to the nasion (in the front -the indented region above the nose) and the inion (in the back - the lowest point of the skull) and stretched to properly position the electrodes. Individual scalp sites were slightly abraded through the hole in the top of each electrode and conducting gel was injected. To ensure good electrical contact, all electrode impedances between the 14 monopolar electrodes (A/D channels 3-16 in Table B.l in Appendix B.2) and the reference EEG (Oz) were below 5 kfi. The scalp electrode impedances were generally all below 3 kd. Appendix B.2 contains detailed instructions used to set-up the BCI system. Electro-oculographic (EOG) activity was measured as the bipolar potential difference from an electrode placed to the right corner of the right eye to a second electrode positioned below the right eye. The criteria used to determine ocular contamination was chosen when the difference between the EOG electrodes exceeded ±25 u.V as in Lisogurski's system [25] and for half a second after, allowing the EOG signal to settle after an eye blink or a strong eye muscle movement. Prior to each experiment, the setting was verified by ensuring that the software would detect the subject's blinks. 4.1.2 Study 1 Subject Selection The two right-handed able-bodied subjects who participated in Study 1 met the following inclusion/exclusion criteria: 1. no evidence of brain injury (self-reported), 2. no physical impairments, 3. medication free, 4. right handed, 5. 18-40 years of age, and 6. interested with a keen and positive attitude. 30 Both participants signed a copy of Appendix B.5 giving their informed consent to participate in this study. Specific information for each participant is given in Table 4.1. The LAT 24-R [32] Handedness Inventory indicated that both subjects were strongly right handed (based on Questions 1-12). A sample of the forms and questionnaires completed by each subject is shown in Appendix B.7. Table 4.1: Study 1 Participant Information Subject Sex Age LAT24-Rf 1 Male 27 35 2 Female 27 36 t 12-15=Strongly Left Handed, 33-36=Strongly Right Handed 4.1.3 Study 1 Method for Data Collection The two subjects participated in two sessions. Each session consisted of up to 30 minutes for set-up and an explanation of the task followed by about an hour of self-paced imagination of index finger flexions. In the first session, the subjects were given an acquaintance period in order to get familiarized with all the operating modes of the system. An example of the non standard right-hand index finger flexion that had been used in Lisogurski's [25] experiments was demonstrated to the subjects, and they were asked to practice making the same movement. They were then asked to imagine making 75 such movements for the actual test. The instructions in Appendix B.3 were followed. Since training and customizing the codebook with imagined movements was considered out of the scope of this work, the experimental sessions involved only testing the LF-ASD for detecting imagined movements using an already existing codebook. As described in Section 3.4, the LVQ codebook obtained from Subject 2 Session 2 in Lisogurski's [25] experiments was used. In each test session, the subject was required to make 75 imagined movements (sum of false negatives and true positives (hits)). This value was chosen to parallel the studies in Lisogurski's experiments with actual movements in which 25 movements were collected for training the classifier and then 75 movements for the test phase. 31 Future studies can also focus on defining the optimum number for testing as the issue of subject fatigue is raised in Chapter 5 where the results of the studies are presented and discussed. 4.2 Study 2 Methodology: SCI Subjects The subject selection criteria and data collection method for Study 2 were more involved compared to Study 1 and are detailed in the following subsections. 4.2.1 Study 2 Experimental Setup The experimental setup for Study 2 was the same as that described in Section 4.1.1 for Study 1. 4.2.2 Study 2 Subject Selection The criteria for choosing subjects with spinal cord injury (SCI) was chosen as follows: 1. no residual sensation or motor function in the hands (generally C5 level injuries and higher), 2. no evidence of brain injury, 3. ideally medication free, 4. no other neurological condition that may interfere with otherwise normal motor cortex function, e.g., epilepsy, 5. within three years of injury3, 6. right handed, 7. 18-40 years of age, 8. visual system intact, 9. no ventilator dependency, 3 This criterion was originally set to recruit subjects who were within one year from injury and close to release from the in-patient rehabilitation with the reasoning that they would be more familiarized with imagining a motor function than those subjects who have been injured for a longer period. However, due to the unavailability of subjects meeting all specified criteria within the time restraint of this thesis project, the date of the injury was extended, and so the SCI subjects who participated in Study 2, had been injured within the last three years. 32 10. no chin or jaw or head (preferably) support that would interfere with the Electro-Cap and chin strap, 11. ability to operate a sip and puff switch and to hold attention for an hour and a half period and understand instructions, 12. emotionally stable, and 13. interested with a keen and positive attitude. Two SCI subjects (out-patients) were recruited from the G.F. Strong Rehabilitation Center4, and their diagnosis was confirmed by the medical investigators5. Both participants signed a copy of Appendix B.5 giving their informed consent to participate in this study. Table 4.2 contains specific information for each SCI participant. ASIA A refers to complete injury with no sensory or motor function preserved in the sacral segments S4-S5 (the Perianal area), and ASIA B refers to incomplete injury with sensory but no motor function preserved below the neurological level and includes sacral segments S4-S5 [33]. Ta )le 4.2: Study 2 Participant Information Subject Sex Age Discharge Diagnosis 1 2 Male Male 27 32 C5 - ASIA B quadriplegia C5 - ASIA A quadriplegia 4.2.3 Study 2 Method for Data Collection The methodology for data collection in Study 2 was the same as in Study 1 except for an additional step that involved determining a suitable decision boundary scale (DB Scale) parameter to control the false positive rate to the desired value of 1% (see Section 3.3). This FP rate was chosen arbitrarily as a reasonable level of FP performance for the purpose of this initial investigation. With the addition of this stage, the first session seemed to have become overwhelming for the subject who was just introduced to the research and was asked to become familiarized with all the system operation modes. Therefore, a more structured evaluation methodology was realized in which the first session was dedicated to familiarizing 4 G.F. Strong Centre, 4255 Laurel Street, Vancouver, B.C., V5Z 2G9 Canada 5 Medical doctors assigned to this research project from the G.F. Strong Centre. 33 the subject to the BCI system, customizing the system parameters, and focusing on providing the necessary guidance and instructions so that the subject develops an approach for activating the system. The second and third sessions were conducted to each collect 75 imagined or attempts to move the right-hand index finger flexion. Appendix B.4 contains the instructions that were followed for conducting the sessions in this study. For setting the DB Scale parameter in the first session, the subject was instructed to watch the two balls in the Experimental Display (Figure 3.3) and monitor their interaction in order to maintain attentiveness while avoiding planning or preparing a motor activity. In this Decision Boundary Adjustment Mode, the false positive rate was monitored, and the subject was not given any feedback for any false switch activation. The DB Scale was then set so that the false positive rate (ratio of false positives to true negatives (idle points)) would average 1% over a period of 30 seconds. 4.3 Method for Lucky Hit Estimation The LF-ASD was designed to recognize a mental process associated with an imagined voluntary movement (IVM). However, due to the generic design of the LF-ASD and lack of subject-specific customized parameterization, patterns produced by mental processes other than an F/M could also trigger the LF-ASD. This was confirmed by the presence of false switch activations reported by the user. Therefore, it was possible for a false switch activation (FP) to occur concurrently with a false negative (FN) and to get identified as a hit by the subject. These classifications, which are referred to as "lucky hits" (LHs) in the context of this research, were recognized as system errors where a LH was defined as a FN masked by a FP but reported as a hit. In order to account for this class of errors, a methodology for LH estimation was developed. The next subsections provide an overview of the underlying problem of estimating the LHs and the details of a suggested estimation methodology. The estimation results for both studies are presented in Chapter 5. 34 4.3.1 Problem Overview As depicted in Figure 4.1, a lucky hit (LH) classification would occur in the presence of an IVM mental process that is not recognized by the switch (FN) concurrently with a non-IVM mental process that triggers the switch (FP). True hits (THs) are depicted in the top path of Figure 4.1 where the IVM mental process results in a Switch Activation Pattern classified by the LF-ASD as active control classification. The term Switch Activation Pattern refers to those patterns that the LF-ASD would identify as elicited by an IVM, based on the information stored in the LVQ codebook. Mental Process Input Output IVM p—H Switch Activation Pattern Non-IVM Other Patterns w LF-ASD * » • Active ° f«— Idle Active classification True Hit (TH) ^ W w Active classification Lucky Hit (LH^ W w Active classification W False Positive (FP) W w Idle classification False Negative (FN) W w Idle classification w True Negative (TN^ W w NOTE: Only one of these inputs enters the LF-ASD at each classification epoch, resulting in one of the corresponding outputs Figure 4.1: Classification Process The possible LF-ASD outputs of Figure 4.1 are decoded in Table 4.3 in terms of User Intent, Switch Activation Pattern and Activation Source. User Intent implies user's intended motor imagery, and Activation Source refers to the mental process (IVM versus non-IVM) 35 that had activated the LF-ASD, applying only to cases with a positive (Yes) Switch Activation Pattern. Table 4.3: BCI System Outputs Output User Switch Activation Activation Intent Pattern Source True Hit TH Yes Yes IVM Lucky Hit LH Yes Yes Non-IVM False Positive FP - No Yes Non-rVM False Negative FN Yes No Not Applicable True Negative TN No No Not Applicable The sum of THs and LHs yielded the total hits (FfTTs), or LH = HIT - TH . (4.1) Table 4.3 shows that the distinguishing element between a TH and a LH was the Activation Source. As there was no physical measure for determining the Activation Source, Equation 4.1 contained two unknowns (LH and TH), but their sum (HIT) was known. Therefore, either the LHs or the THs needed to be identified to solve the problem. The definition of a LH was used as an approach to solve for the unknown LHs, and the problem was thus reduced to estimating the LHs. The method to estimate the LHs, as described in the next section, was based on the observed experimental data of the trials ending with a HIT, FP and FN. The trial lengths were not equal as the experimental paradigm was self-paced, and the subject attempted to activate the switch at random intervals. Due to this randomness, the degree to which the system was prone to error could be analyzed probabilistically. Ideally one must repeat an experiment a great many times in order to calculate accurate statistical measures that describe and characterize randomness in a physical system [34]. However, it was impractical to schedule many sessions with each subject to model the switch activation and user intent probabilities under identical conditions for each subject. Thus, the error analysis was only theoretical. Here, it was assumed that the relative frequencies of HTTs, FPs, and FNs are good approximations for their repetitive physical probabilities, given that as more and more trials are carried out, relative frequencies converge to probabilities. The estimated probabilities of HTTs, FPs and FNs could then be used to estimate the probability of LHs. 36 4.3.2 Detailed Estimation Methodology Following the last statement of Section 4.3.1, if the probability of a trial ending with a LH after n samples, P(LH(n)), could be estimated, the total number of lucky hits, N(LH), would be estimated by N(LH) = round(£ P(LH(n))HHIT (n)) (4.2) n in which the summation is over the discrete time index n, representing trial lengths. A trial length is defined as the count of consecutive true negative (TN) classifications (attentive idle points) before a trial was terminated. Here, HHrr(n) is the histogram of trial lengths of the trials ending with a HTT, and the function round(), rounds its argument to the nearest integer. Table 4.4 provides a reference guide for the variables used in computing Equation 4.2. Table 4.4: Section 4.3.2 Reference of Notations Class Term Definition Reference Equation N(x) N(T) total number of trials -N(FP) total number of trials ending with a FP -N(FN) total number of trials ending with a FN -N(HIT) total number of trials ending with a HIT -N(LH) estimated total number of trials ending with a LH (4.2) HJn) Hrin) histogram of trial lengths of all the observed trials -Hppfn) histogram of trial lengths of all observed trials ending with aFP -HFN(n) histogram of trial lengths of all observed trials ending with a FN HHn{n) histogram of trial lengths of all observed trials ending with a HIT -HLH(n) estimated histogram of trial lengths of trials ending with a LH -P(x(n)) P(FP(n)) probability of a trial ending with a FP after n samples (4.7) P(FN(n)) probability of a trial ending with a FN after n samples (4.4) P(HIT(n)) probability of a trial ending with a HTT after n samples -P(LH(n)) probability of a trial ending with a LH after n samples (4.3) As stated in the introduction of Section 4.3, a LH is a FN masked by a switch activation that was generated by a FP. The underlying assumption is that LHs were 37 generated by the same mechanism(s) that generated the reported FPs. Table 4.3 shows that a LH and a FP have the same type of Activation Source, i.e., a non-IVM. Since the necessary condition for a LH was that the FN and FP would occur concurrently or within a very close window that was indistinguishable by the subject, it was assumed that the probability of LHs could be represented by P(LH)~P(FNnFP), where the FNs provide insight to the distribution of the user's undetected intents, HpN(n), and the FPs provide insight to the distribution of the false switch activations, Hpp(n). On this basis, a high probability of LHs would be expected if a subject reported many FNs and many FPs. It was reasonable to model the FNs and FPs as statistically independent since there was no obvious metric available to account for the factors (e.g., mental, psychological) that would result in dependency amongst trials. In practice, P(LH(n)) was estimated by P(LH(n)) = P(FN(n))P(FP(n)), (4.3) for all trials of length n. Here, P(FN(n)) is the estimated probability of a trial ending with a FN after n samples, and P(FP(n)) is the estimated probability of a trial ending with a FP after n samples. The rest of this section explains how each of these estimates were computed and the reason for introducing the extra complexity of estimating P(FN(n)). Recalling that the subject would report the FNs with an approximate delay6 of one or two seconds after he or she expected the system to indicate a switch activation (via GUI Feedback), the reported HpN(n) distribution was not an accurate representation of the true HFN(n) distribution. The key problem in estimating P(FN(n)) was the difficulty in accounting for this delay in the estimation of the HFN(H) distribution. In this work, the 6 There is an interpretive time window centering the subject's IVM mental process, in which the user must define his or her own strategy in distinguishing a switch activation that stemmed from an IVM in self-paced environments. It is expected that the probability of a LH occurring within this time window would increase in proportion to the length of the window. The built-in signal processing delay of the system (640.5 milliseconds [25]) contributes to the length of this window. In Study 1, there was an additional 1-second delay (see Figure 3.5 - cases of HIT and FP) before the subject would receive feedback (GUI Feedback) from the system regarding an active switch classification. This 1-second delay was removed as part of the System Version 2 Operation improvements for Study 2 (Section 3.3) to reduce user frustration. It was possible, therefore, for LHs to occur within the period of 1640.5 milliseconds for Study 1 and 640.5 milliseconds for Study 2, after the subject's intent to make a movement. 38 probability of P(FN(n)) in Equation 4.3 was calculated based on the assumption that the distribution of the user's undetected intents, HpN(n), was similar to the distribution of the user's detected intents, HHrT(n). The final equation used to compute P(FN(n)) is given by P(FN(n)) = H™(n'a) , where (4.4) HT(n,a) HFN{n,a) = aHHIT(n), (4.5) for all trials of length n. Here, HFN(n,a) is the estimated histogram of trial lengths of all trials ending with a FN dependant on the scalar factor a and HHrr(n). The constant a was computed as a ratio of the total number of FNs to the total number of HTTs: N(FN) a = —-—-. N(HIT) Hr(n,a) in Equation 4.4 is the estimated histogram of trial lengths of all trials and is dependent on the scalar factor a as it contains HpN(n,a) as shown in Equation 4.6: HT (n,a) = HHIT (n) + HFP(n) + HFN (n,a), (4.6) for all trials of length n. The next task to complete Equation 4.3 was to obtain P(FP(n)), which was computed by P(FP(n)) = -^f^-, (4.7) HT (n, a) for all trials of length n. In summary, P(FN(n)), as computed by Equation 4.4, and P(FP(n)), as computed by Equation 4.7, were used in the estimation of P(LH(n))of Equation 4.3, which is understood as the probability of trials of length n ending with a LH, given the total number of trials of length n. Consequently, it was multiplied by HT(n,or) for the final estimation of N(LH) in Equation 4.2. Alternative options for computing P(FN(n)) of Equation 4.3 were considered, and the reasons for their rejection are described in the remainder of this section. 39 In order to account for the delay in reporting the FNs, it was possible to shift the histogram of the observed FN trials, HFNOI), back in time by x, resulting in HFN(n-x). This method would estimate P(FN(n), T) , which is dependent on the factor x. The time-shift x could be considered to be 1.5 seconds as the average of 1 or 2 seconds expected delays. However, the disadvantage of this method was the use of a constant time-shift x across all observed FN trials. The appropriate value of x was also questionable. Moreover, HFN(U-X) lacked information about the distribution of User Intent for trial lengths where relatively more FPs were detected. Naturally, if the switch was falsely activated, the subject did not have the opportunity to report a FN, resulting in a potential LH; however, HpN(n-x) would not reflect this. Another option for estimating P(FN(n)) was considered that incorporated the distribution of the user's detected intents, HHrT(n) and the shifted distribution of the user's undetected intents, HRMOI-X). It was possible to estimate P(FN(n)) by scaling the probability of a trial in which the user intended a finger movement, P(I(n)) (with the scaling ratio P). Here, P(I(n)) could be estimated by summing P(HIT(n)) andP(FN(n),r). The scaling factor P could be computed as the ratio of the total number of undetected intents, N(FN), to the total number of intents (N(TN)+N(HTT)). This estimation attempted to overcome the shortcoming of P(FN(n), T) in encompassing the full range of the distribution of User Intent; however, it still relied on the "fuzzy" approach of time-shifting the HpN(n). It should be noted that this P(FN(n)) estimation method gave comparable LH estimation results as the estimation of Equation 4.4. However, since the time-shift approach introduced a degree of uncertainty, Equation 4.4 was chosen for the final computation of P(FN(n)). 4.3.3 Computation of Lucky Hits It should be noted that the distributions, Hpp(n) and HHrT(n), which were used to compute the estimated probability function of P(LH(n)) of Equation 4.3, were discrete functions and discontinuous for trial lengths where no switch activation had been detected. In other words, if there were no trials of length n0 that ended with a FP or a HIT, then Hpp((n0)) and HHrT(n0) 40 were zero, resulting in their corresponding probabilities to be zero and discontinuous for this particular trial length. This was the disadvantage in using relative frequencies as approximations of probability distributions when data from a limited number of trials (between 78 and 188) were used. Since it is expected that continuous functions will describe the behaviour of the system better, a 5-tap uniform smoothing filter was applied to smooth the above mentioned histogram distributions. The total number of FPs and HTTs, i.e., N(FP) and N(FflT), were conserved. The LH estimation results are presented in Chapter 5. As there was no data that could be used to determine the number of LHs directly, it was hard to distinguish the accuracy of various estimation methods. Indeed, methods different from what is presented in Section 4.3.2 may give better results over repeated usage. It should be noted that the introduced methodology has been based on intuition and modeling assumptions that are difficult to verify. Appendix C contains a reference of plots that were produced in the process of estimating the LHs. For each study and each subject, the set of plots display (a) the smoothed observed FflT distribution HHrr(n), (b) the smoothed estimated FN distribution HpN(n,a), (c) the smoothed observed FP distribution Hpp(n), (d) the estimated probability of a trial ending with a FN after n samples, P(FN(n)), (e) the estimated probability of a trial ending with a FP after n samples, P(FP(n)), (f) the estimated probability of a trial ending with a LH after n samples, P(LH(n)), and (g) the estimated LH distribution, HLHOI). 41 CHAPTER 5 RESULTS & DISCUSSION 5.1 Results of Study 1 The performance of the two able-bodied subjects who participated in Study 1 is shown in Table 5.1 [35]. Samples collected during the Feedback and Reject Periods were not included in the results. The imagined movements were self-paced so a different number of idle data points were collected from each subject. The first session for Subject 2 was shortened before the target of 75 imagined movements due to a report of fatigue by the subject. Table 5.1: Study 1 Modified Online LF-ASD Performance with Imagined Movements by Al )le-Bodied Subjects Session/ Actual Class Subject 1 Reported Class Subject 2 Reported Class Idle Act. Perf. Idle Act. Perf. 1/ldle 1/Act. 3514 21 66 54 98.2% 72.0% 1885 21 19 48 99.0%* 69.6%* 2/Idle 2/Act. 3617 14 114 61 96.9% 81.3% 1758 17 23 58 98.7% 77.3% Idle = Attentive spontaneous E Act. = Activations 3G Perf. = Performance (% correctly classified) * With only 69 imagined movements 42 Both subjects reported that they followed a consistent strategy in confirming the hits: A switch activation occurring within 1 to 1.5 seconds following an imagined right hand index finger flexion was considered a hit (true positive). In both subjects, the LF-ASD would occasionally detect VMRPs associated with the sip or puff activations when subjects would report false negatives. The contributions of this class of false positives were computed to be 0.4% and 0.3% for Subject 1, and 0.9% and 0.8% for Subject 2, Sessions 1 and 2 respectively. These were believed to be anomalies in the methodology due to the self-reporting system and were not included in the performance values reported in Table 5.1. It was observed that many of the false positives would occur immediately after the Feedback Period was over and the balls in the display would start moving again. Therefore, it is possible that the LF-ASD was picking up changes in the EEG associated with the changes in the display. This observation inspired the modifications made to the algorithm for Study 2 (refer to Section 3.3). As a result, a programming mistake was realized, where the display was not reset immediately after the Feedback Period, but after the Reject Period leading to the Operate Mode. This may have led to a slightly higher false positive rate. In order to simulate a more natural interface, the LF-ASD was allowed to classify even if EOG artifact was detected. Any hits or false positives that were detected in the presence of EOG artifact (see Section 4.1) were marked separately as "contaminated hits" or "contaminated false positives" so that the significance of EOG contamination on the EEG data and its effect on the classifications could be studied. The effect of EOG contamination on the LF-ASD performance was examined through determination of the following ratios. The ratio of contaminated hits to total hits ranged from 10% to 20%, and the ratio of contaminated false positives to total false positives varied from 20% to 30%. These ratios support the notion that the LF-ASD was not triggered by eye blinks or eye movements. Figures 5.1 and 5.2 display the filtered (0.1-4 FIz) averages of the bipolar difference of FCi-Ci electrode signals (left column), monopolar electrode signal Ci (middle column) and the bipolar EOG signal (right column) for uncontaminated hits for both sessions for subjects 1 and 2 respectively. The 2-second averages center at the onset of the hit detection, which occurs at the 0-second marker. 43 Avg. of FC1-C1 Avg. of C1 Avg. of EOG o > <u 0 CO CO CO -1 0 1 Time (seconds) -1 0 1 Time (seconds) -1 0 1 Time (seconds) Figure 5.1: Study 1, Able-bodied Subject 1 - Filtered Averages ofFCi-Ci, C] and EOG for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the O-second marker. Time (seconds) Time (seconds) Time (seconds) Figure 5.2: Study 1, Able-bodied Subject 2 - Filtered Averages ofFCi-Cj, C\ and EOG for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the O-second marker. 44 The FC1-C1 averages for Subject 1 clearly display a movement-related potential (MRP), in form of a bi-level wavelet, activated at the O-second marker (trigger point). For Subject 2, the average of FC1-C1 for Session 1 seems to be blurred under other brain activity. This subject had gotten fatigued before all 75 imagined movements were carried out, and the session was terminated early. It is possible that this fatigue affected the FQ-Ci average. For both subjects the average of the monopolar signal Q seem to show a distinct pattern with a strong dip occurring at the trigger point, thus providing evidence of MI activity during imagined movements. This shape is in the form of a classic averaged MRP during voluntary flexion of the right hand forefinger [36, 37, 38]. The EOG averages in the rightmost column confirm that there was no consistent ocular activity affecting the detection of uncontaminated hits. As defined in Section 3.2, each trial was terminated by either a switch activation, leading to a hit or a false positive, or a false negative report made by the user. The histograms in Figures 5.3 and 5.4 display the frequency and variation of trial lengths for each subject for true positives (hits), false negatives and false positives for Subjects 1 and 2 respectively. Basically a trial length is defined as the count of consecutive true negative classifications (attentive idle points) before a trial was terminated. It can be observed that for both subjects the frequency distribution of hits (Figure 5.3(a)) and false positives (Figure 5.3(c)) are comparable. This observation raised the discussion for an estimation of "lucky hits" as it was quite possible for the system's false activations to overlap with intended mental activations. According to the suggested method of Section 4.3, the "lucky hit" (LH) estimates were computed as shown in Table 5.2. Table 5.2: Study 1 Lucky Hit Estimation Session Subject 1 Subject 2 Lucky Hits Hits Perc. LH/Hit A Per/. Lucky Hits Hits Perc. LH/Hit A Per/. 1 11 54 20.4% -14.7% 4 48 8.3% -5.8%* 2 14 61 23.0% -18.7% 7 58 12.1% -9.3% Perc. LH/ Hit = Percentage of Lucky Hits/Total Hits A Perf. = Change in Performance (% correctly classified) *With only 69 imagined movements 45 The change in performance of the system (A Perf.) is also demonstrated in Table 5.2. The overall performance of the system is consequently lowered as expected. The percentage of LHs was estimated to be higher for Subject 1 compared to Subject 2. This result confirmed the intuitive expectation that when there was a larger overlap among the distributions of false positives and hits, there is an increased chance that a hit might have been classified by a faulty mechanism. Appendix C contains a reference of plots (Figures C.l through C.4) that were produced for the lucky hit estimation of Study 1. S1S1S1 S1S1S2 Time (seconds) 10 5 0 III III I , L 10 5 0 LLlllil Time (seconds) 10 10 Figure 5.3: Study 1, Able-bodied Subject 1 - Trial Length Distribution of (a) Hits, (b) False Negatives, and (c) False Positives, Session 1 (left), Session 2 (right). Subject 1 reported that it was difficult for him to differentiate between planning a movement and not planning a movement as both required mental preparation. He also mentioned that he would try to pay attention to the motion of the balls at the beginning but it was easy for his mind to wonder off at times. It seemed that the false positive rate decreased as he got used to the system and had better concentration. 46 S1S2S1 S1S2S2 >,10 (Li a) b _l_L Time (seconds) 10 Time (seconds) Figure 5.4: Study 1, Able-bodied Subject 2 - Trial Length Distribution of (a) Hits, (b) False Negatives, and(c) False Positives, Session 1 (left), Session 2 (right). Subject 2 commented on how her tension would cause more false positives; however, if she relaxed completely then she would get fewer hits. She indicated that it was essential to maintain absolute concentration for the mental activation and such attention would require a lot of energy. Both subjects, in fact, reported that the sessions were very tiring. In order to investigate the effect of fatigue during the duration of each session, Figures 5.5 and 5.6 were produced. These figures display the distribution of (a) hits, (b) false negatives and (c) false positives against time throughout the sessions 1 (left column) and 2 (right column) for subjects 1 and 2 respectively. Each bin represents 3 minute time periods. 47 S1S1S1 S1S1S2 (a) 10 123456789 15 (b)10 5 0 •llll-l-l 123456789 123456789 Time (3 Minute Bins) 1 2 3 4 5 6 7 8 9101112 Time (3 Minute Bins) Figure 5.5: Study 1, Able-bodied Subject 1 - Distribution of (a) Hits, (b) False Negatives, and(c) False Positives in 3-Minute Bins, Session 1 (left), Session 2 (right). S1S2S1 S1S2S2 1 2 3 4 5 6 1 2 3 4 5 6 Time (3 Minute Bins) 1 2 3 4 5 6 1 2 3 4 5 6 Time (3 Minute Bins) Figure 5.6: Study 1, Able-bodied Subject 2 - Distribution of (a) Hits, (b) False Negatives, and (c) False Positives in 3-Minute Bins Session 1 (left), Session 2 (right). Subject 2 became fatigued before completing all 75 imagined movements in Session 1. Therefore, this session was terminated after she performed 69 imagined movements. Towards the end of this session, there is an increase of false positives as shown in Figure 5.6(c). Such a pattern can also be observed for Subject 1 (Figure 5.5(c) Session 1). For both subjects, however, the false positives did not increase towards the end of Session 2, which may indicate, inline with their self-report, that the subjects were exerting concentration throughout the session. As this work on detecting imagined movements was new, other potential approaches to verify the results of Table 5.1 were considered. In the second session, an additional experiment was conducted where Subject 1 was asked to repeat the experiment by actually making 75 right-handed index finger flexions. The result of this experiment is presented in Table 5.3. Interestingly, Subject 1 reported that the majority of the hits correlated with the planning of his actual finger movements. Table 5.3: Study 1 Modified Online LF-ASD Performance with Actual Movements Subject 1 Actual Reported Class Class Idle Act. Perf. Idle 3000 85 97.5% Act. 9 66 88.0% Idle = Attentive spontaneous EEG Act. = Activations Perf. = Performance (% correctly classified) 5.2 Results of Study 2 Two spinal cord injured subjects each participated in two sessions. The results are presented in Table 5.4 [39]. The DB Scale parameter was set to 110 and 108 on a Max DB Scale parameter of 200 for Subjects 1 and 2 respectively. As in Study 1, samples collected during the Feedback and Reject Periods were not included in the results. The imagined movements were self-paced so a different number of idle data points (or true negatives) were collected from each subject. 49 Table 5.4: Study 2 Modified Online LF-ASD Performance wit 1 Imagined Movements by SCI Subjects Session/ Actual Class Subject 1 Reported Class Subject 2 Reported Class Idle Act. Perf. Idle Act. Perf. 1/ldle 1/Act. 5970 42 99.3% 42 33 44.0% 3578 3 99.9% 48 27 36.0% 2/ldle 2/Act. 7265 70 99.0% 42 33 44.0% 2927 14 99.5% 39 36 48.0% Idle = Attentive spontaneous EEG Act. = Activations Perf. = Performance (% correctly classified) Both subjects reported that they followed a consistent strategy in confirming the hits: A switch activation occurring within 1 to 1.5 seconds following their intention or attempt to move their right hand index finger was considered a hit (true positive). Similarly as reported in Table 5.1, the false positives associated with the sip or puff activations were not included in Table 5.4. The contributions of this class of false positives were computed to be 0.2% and 0.1% for Subject 1, and 0.1% and 0.4% for Subject 2, Sessions 1 and 2 respectively. The ratio of contaminated hits to total hits ranged from 0% to 4%, and the ratio of contaminated false positives to total false positives varied from 0% to 14%. Again, these ratios support the notion that the LF-ASD was not triggered by eye blinks or eye movements. Figures 5.7 and 5.8 display the filtered (0.1-4 FIz) averages of the bipolar difference of FCi-Ci electrode signals (left column), monopolar electrode signal Ci (middle column) and the bipolar EOG signal (right column) for uncontaminated hits for both sessions for Subjects 1 and 2 respectively. The averages center at the onset of a hit detection (at the 0-second marker). 50 Avg. ofFC1-C1 Avg. ofC1 Avg. of EOG Time (seconds) Time (seconds) Time (seconds) Figure 5.7: Study 2, SCI Subject 1 - Filtered Averages ofFCi-Cj, Cj and EOG for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker. The FC1-C1 averages for Subjects 1 and 2 display a weaker motor potential activated at the 0-second marker (trigger point) compared to those in Study 1. However, for both subjects the average of the monopolar signal Q seem to show a distinct pattern with a strong dip occurring at the trigger point, similar to those patterns observed for able-bodied subjects (see Figures 5.1 and 5.2). Again, this shape is in the form of a classic MRP during voluntary flexion of the right hand forefinger [36, 37, 38]. Although the EOG averages (the rightmost column) are not completely flat for the 2 second-period centering the trigger point, no strong consistent ocular activity appears to be disturbing the trigger point. 51 Avg. of FC1-C1 Avg. of C1 Avg. of EOG > CO CM CO CM CO O > 2 -1 0 1 Time (seconds) -1 0 1 Time (seconds) -1 0 1 Time (seconds) Figure 5.8: Study 2, SCI Subject 2 - Filtered Averages ofFCi-Cj, Ci and EOG for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker. Figures 5.9 and 5.10 display trial length histograms for true positives (hits), false negatives and false positives for Subjects 1 and 2 respectively. A trial length is defined as the count of true negative classifications (attentive idle points) before a switch activation or a false negative report by the subject. Subject 1 had the tendency to test the system by waiting for a long time before attempting a movement resulting in long trials with several ending with a false positive switch activation (Figure 5.9(c) Right). 52 S2S1S1 S2S1S2 10 5 0 Jil II nun i • mi i i i 10 Time (seconds) Time (seconds) Figure 5.9: Study 2, SCI Subject 1 - Trial Length Distribution of (a) Hits, (b) False Negatives, and(c) False Positives, Session 1 (left), Session 2 (right). From the histograms of Figures 5.9 and 5.10, it can observed that there is less overlap among the distributions of hits, false positives and false negatives for Study 2 subjects compared to Study 1 subjects (Figures 5.3 and 5.4). Consequently, the percentage of lucky hits estimated for Study 2 subjects were lower in comparison as can be seen in Table 5.5. The overall performance is slightly lowered of course. Appendix C contains a reference of plots (Figures C.5 through C.8) that were produced for the lucky hit estimation of Study 2. Table 5.5: Study 2 Lucky Hit Estimation Session Subject 1 Subject 2 Lucky Hits Hits Perc. A Perf. Lucky Hits Hits Perc. A Perf. 1 3 33 9.1% -4.0% 1 27 3.7% -1.3% 2 5 33 15.2% -6.7% 5 36 13.9% -6.7% Perc. = Percentage of Lucky Hits/Total Hits A Perf. = Change in Performance (% correctly classified) 53 S2S2S1 S2S2S2 Time (seconds) Time (seconds) Figure 5.10: Study 2, SCI Subject 2 - Trial Length Distribution of (a) Hits, (b) False Negatives, and(c) False Positives, Session 1 (left), Session 2 (right). The histograms of Figures 5.11 and 5.12 display the distribution of (a) hits, (b) false negatives and (c) false positives against time throughout the session for Subjects 1 and 2 respectively. Each bin represents 3 minute time periods. Similar to Study 1, both subjects display an increase in false positives at the end of Session 1 (Figures 5.11(c) Left and 5.12 (c) Left) and decrease in hits towards the end of both sessions, which most probably is fatigue related. Subject 1 in Session 1 (Figure 5.11(b) Right) and Subject 2 in both sessions (Figure 5.12(c)) reported increasingly more false negatives towards the end of the sessions. 54 S2S1S1 S2S1S2 1 2 3 4 5 6 7 Time (3 Minute Bins) 123456789 123456789 Time (3 Minute Bins) Figure 5.11: Study 2, SCI Subject 1 - Distribution of (a) Hits, (b) False Negatives, and (c) False Positives in 3-Minute Bins, Session 1 (left), Session 2 (right). S2S2S1 S2S2S2 12 3 4 Time (3 Minute Bins) 12 3 4 Time (3 Minute Bins) Figure 5.12: Study 2, SCI Subject 2 - Distribution of (a) Hits, (b) False Negatives, and (c) False Positives in 3-Minute Bins, Session 1 (left), Session 2 (right). A future study can also investigate the appropriate number of imagined movements to be executed by the subject in each session. The value 75 was chosen similarly to Lisogurski's data collection methodology where he believed that having the subject perform 25 movements for training and then 75 movements would not tire the subject and would allow for a suitable evaluation of the system. Even though the effects of training cannot be studied through conducting only two sessions, one cannot escape from observing that in Table 5.4 Subject 1 does not display any improvement in Session 2. However, if the sessions were terminated after 50 movements to account for the effect of fatigue, as subjects would generally tire towards the end of the session, then the results of Table 5.6 may possibly indicate some preliminary training effects. For the sake of consistency, the false positives associated with the sip or puff activations were not included in Table 5.6. The contributions of this class of false positives were computed to be 0.2% and 0.1% for Subject 1, and 0.2% and 0.3% for Subject 2, Sessions 1 and 2 respectively. Table 5.6: Study 2 Modified Online LF-ASD Performance with Only 50 Imagined Movements by SCI Subjects Session/ Actual Class Subject 1 Reported Class Subject 2 Reported Class Idle Act. Perf. Idle Act. Perf. 1/ldle 1/Act. 3667 14 99.6% 29 21 42.0% 2412 2 99.9% 31 19 38.0% 2/ldle 2/Act. 5655 63 98.9% 23 27 54.0% 1614 12 99.6% 26 24 48.0% Idle = Attentive spontaneous EEG Act. = Activations Perf. = Performance (% correctly classified) 5.3 Overall Discussion It should be noted again that the results presented in Sections 5.1 and 5.2 were obtained using an existing codebook, and it is believed that through customization of the system parameters, the hit rates and the false positive rates can be improved. The on-line performance displayed very promising hit rates while keeping false positive rates at a low level for Study 1. The hit rates were lowered and the false positive rates were improved as a result of tuning the DB 56 Scale setting (see Section 3.3) in the system for Study 2 in order to mitigate subject frustration. It is interesting to note that in Study 2, the DB Scale setting as determined through the methodology of Section 4.2.3 and Appendix B.4 (where the subject was instructed to pay attention to the movement of the balls without getting any feedback) would result in a FP rate artificially high and in actual test setting the FP rate would lower. As a result, one can aim for achieving FP values of up to 1.5 % or 1.7% during the DB Scale Adjustment period in order to actually obtain a false positive rate of approximately 1% for the testing part of the session. This in effect implies the possibility that the LF-ASD is sensitive to other patterns as well as the motor potential and confirms the necessity of subject parameter customization as well as pattern customization specific to motor imagery. A dominant occurrence of false positives within 1 to 2 seconds right after the Feedback Period (see Figures 5.3(c), 5.4(c), 5.9(c) and 5.10(c)) was commonly observed in both studies. It was thus hypothesized that perhaps the change in the GUI display, resulting in the shift in the subject's attention was triggering the LF-ASD switch. Pfurtscheller et al. [40] provide evidence that a perceived visual cue stimulus might lead to early specific activation of the primary sensorimotor cortex after cue-onset. The start of the balls moving after the Feedback Period could be thought of as a pseudo cue-onset in this study. Extending the Reject Period after the Feedback Period from one-second to two seconds on a trial by error basis was introduced; however, as the sample subject size was small, no significant effect was realized. This can be further examined in a future study in order to realize the optimum length of the Reject Period. The monopolar averages of the Ci signal, which is near the left-side, right-hand primary motor area (Figure 2.1), display a strong pattern associated with the detection of hits for both able-bodied (Figures 5.1 and 5.2 Middle Column) and spinal cord injured subjects (Figure 5.7 and 5.8 Middle Column). The FCi-Ci averages for SCI subjects (Figures 5.7 and 5.8 Left Column) are relatively weaker compared to those obtained for the able-bodied subjects (Figures 5.1 and 5.2 Left Column). Perhaps these factors can be incorporated in order to fine-tune the Feature Classifier of the LF-ASD, which is dependant on the signal amplitude, to further improve the detection of imagined movements. 57 The LF-ASD in its current state has been designed to detect generic motor activity based on the bipolar response of the MI and the SMA. The role that the MI plays during motor imagery has not been clearly defined. Green et al. [41] use high-resolution EEG to record and map movement-related cortical potentials associated with visually cued actual and imagined finger and toe movements in normal subjects. This study concludes that the motor networks underlying the generation of actual and imagined movements are different and that imagined or simulated movements are generated in or near midline structures such as the SMA and do not involve the MI. Moreover, the latencies are similar for actual and imagined motor potentials, but the amplitudes of imagined potentials are lower and more variable, reflecting weaker signals. Cunnington et al. [28] support the above findings that both movement execution and motor imagery involve similar pre-movement preparatory processes generated most likely by the supplementary motor area. It indicates that there is a bilateral symmetrical distribution of the late component of the MRP as opposed to a strong contralateral pattern produced when actually performing a right hand movement. However, they do not dismiss the role of the MI when imagining movements, and they only state that considering that the late component of the MRP reflects mainly the activity of the primary motor cortex associated with movement execution, the MI "may" only operate during actual movement performance. Figures 5.13 through 5.16 display the filtered (0.1-4 Hz) averages of the monopolar signals Ci (left column), Cz (middle column) and C2 (right column) uncontaminated hits for the subjects in both studies. As highlighted in Figure 2.1, the Ci, Cz, and C2 electrode positions are located over the primary motor cortex (MI). The averages center at the onset of a hit detection (at the O-second marker). 58 Avg. of C1 Avg. of Cz Avg. of C2 Time (seconds) -1 0 1 Time (seconds) 1 0 1 Time (seconds) Figure 5.13: Study 1, Able-bodied Subject 1 - Filtered Averages ofCi, Cz and C2 for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker. -1 0 1-1 0 1-1 0 1 Time (seconds) Time (seconds) Time (seconds) Figure 5.14: Study 1, Able-bodied Subject 2 - Filtered Averages ofCi, Cv and C2 for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker. 59 Time (seconds) -1 0 1 Time (seconds) 1 0 1 Time (seconds) Figure 5.15: Study 2, SCI Subject 1 - Filtered Averages ofCj, Cz and C2 for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker. -1 0 1-1 0 1-1 0 1 Time (seconds) Time (seconds) Time (seconds) Figure 5.16: Study 2, SCI Subject 2— Filtered Averages of C\, Cz and C2 for All Uncontaminated Hits, Session 1 (top), Session 2 (bottom). Hit detected at the 0-second marker. 60 As a measure of comparison, the difference between maximum (most positive) and minimum (most negative) values within the 1-second period centering on the detected hit for both studies is presented in Table 5.8. These results display significant activity over the MI for able-bodied subjects and spinal cord injured subjects. The latter observation is most important as there has been no on-line data collected to date to indicate activity of the MI during movement attempts by SCI subjects. The MRPs of actual finger and hand movements show maximum negativity not at the vertex but over the contralateral precentral hand area [36]. It is also reported in [36] that the total amplitude of the pre-movement potentials did not differ significantly between the left precentral and vertex electrodes, but over the right precentral region the potential was significantly different compared to both the left precentral and vertex regions. Except for able-bodied Subject 1 Session 1 and SCI Subject 1 (both sessions), which display a stronger contralateral activity of the MI, Table 5.8 indicates that the greatest potential difference was recorded at position Cz (vertex). These results are in agreement with Cunnington's findings comparing actual and imagined movement patterns [28]. Table 5.7: Max-Min Differences in Amplitude of Averaged Monopolar Signals for all Uncontaminated Hits StudyXSubjectXSessionX c, Max-Min (ftV) cz Max-Min (JXV) c2 Max-Min (/xV) S1S1S1 15.2185* 14.3282 13.1495 S1S1S2 18.5268 21.0180* 19.3717 S1S2S1 11.2075 12.8403* 12.1432 S1S2S2 22.8138 25.1062* 20.0474 S2S1S1 21.3365* 20.6647 17.8899 S2S1S2 18.8621* 17.4364 15.7387 S2S2S1 13.6601 14.6633* 13.0943 S2S2S2 11.2504 11.5269* 11.3235 * The greatest value across Q, Cz and C2 for eac h subject Furthermore, it was realized after conducting the two studies that the codebook selected from Lisogurski's study should have been adopted with consideration of the difference between the scaling and offset coefficients between the data collected for this research and that collected by Lisogurski mainly due to the difference in the hardware setup 61 (e.g., EEG amplifier gain settings). The average input peak amplitude of FCZ was computed as shown in Table 5.8. The significant variation of these values from Lisogurski's Subject 2 Session 2 average from which the codebook was selected may be one of the potential sources of false switch activations. Proper normalization of collected EEG data with respect to "off-the-shelf codebook templates is left as part of the future development of this BCI system. Table 5.8 : Average Input (FCZ) Peak Amplitude Study Subject Session Average FCZ Peak Amplitude (JUV) Lisogurski 2 2 8.7 1 1 1 13.5 1 "1 2 22.7 1 2 1 24.1 1 2 2 22.0 2 1 1 30.3 2 1 2 28.3 2 2 1 25.6 2 2 2 31.1 The decision to let the classifier be active at all times even during EOG activity was made so that the system imposes less control upon the subject. The studies conducted for this research show that the LF-ASD was not triggered by the subject's EOG activity. Although the proportion of contaminated false switch activations (FPs) to total FPs was low (below 30% in Study 1 and below 14% in Study 2), for future studies of the LF-ASD, it is suggested that classifications coinciding with any strong ocular activity be warned to the subject (e.g., by flashing the border of the display box of Figure 3.3 for 1 second) instead of having the subject wait for the completion of the GUI Feedback Period (where the balls flash and the center ball changes direction for the duration of approximately 5.5 seconds) and then the Feedback Period (where the balls stop for 4 seconds) in order to report a false positive. Consequently, the subject is given immediate feedback from the system and becomes less frustrated by realizing a potential cause of error (presence of EOG artifact). This approach could more effectively induce the user to reduce the production of such noise while employing motor imagery. 62 CHAPTER 6 CONCLUSIONS This work has presented a BCI system and evaluation methods to study how well the LF-ASD responds to self-paced imagined voluntary movements in able-bodied subjects and subjects with spinal cord injuries. The findings were as follows: 1. Two studies were conducted as part of the evaluation methodology. In the first study, two able-bodied subjects participated in two sessions. The estimated hit rates were 57.3%, 62.6%, 63.8%, and 68.0% and the estimated false positive rates were 2.1%, 3.4%, 1.2% and 1.7% for Subject 1 and Subject 2 across the two sessions respectively. These results include the self-reported errors (false positives and false negatives) as well as the estimated lucky hits (a lucky hit (LH) was defined as a false positive occurring concurrent with an undetected imagined movement - refer to Section 4.3 for more details). In the absence of the LH estimation, the system reported hit rates of 72.0%, 81.3%, 69.6% and 77.3% and false positive rates of 1.8%, 3.1%, 1.0% and 1.3% for Subject 1 and Subject 2 across the two sessions respectively. The results of the LH estimation lowered the overall performance (the corresponding numbers can be compared). The result of this study provided strong evidence that able-bodied subjects can activate the on line LF-ASD with imagined right-hand index finger flexions. In the second study, the system was modified based on findings from the first study. An option of biasing the classifier decision boundary was introduced in order to minimize the false positive rate and lower the subjects' frustration 63 level. Two subjects with spinal cord injury participated in two sessions in the second study. The estimated hit rates were 40.0%, 37.3%, 34.7% and 41.3% and the estimated false positive rates were 0.7%, 1.0%, 0.1% and 0.6% for Subject 1 and Subject 2 across the two sessions respectively. Again, these results include the estimated lucky hits. In the absence of the LH estimation, the system reported hit rates of 44.0%, 44.0%, 36.0% and 48.0% and false positive rates of 0.7%, 1.0%, 0.1% and 0.5% for Subject 1 and Subject 2 across the two sessions respectively. The LH estimations of the second study lowered the performance to a lesser degree compared to Study 1. The results of this study provided strong evidence that SCI subjects can activate the on-line LF-ASD with imagined right-hand index finger flexions. 2. The averages of the bipolar difference of FCi-Ci electrode signals, as well as the monopolar electrode signals Ci, Cz and C2 for uncontaminated hits for both studies provided further evidence that the LF-ASD feature set can be used to detect imagined voluntary movements by subjects with spinal cord injuries. The detected imagined movements displayed patterns very similar to the classic shape of the Movement Related Potentials during actual index finger flexions [36, 37, 38]. The results from the two studies were obtained using an LVQ codebook that was prepared in the on-line study of the LF-ASD with able-bodied subjects actually making voluntary movements [25]. It is believed that through customization of the codebook and other system parameters, tuned for each subject, the performance of the LF-ASD can be further enhanced while maintaining low false positive rates. There are many outstanding issues that remain to be studied. These include exploring methods to improve the activation accuracy of the switch, developing adaptive methods to automatically customize the switch to a specific individual, exploring ways to extend the switch functionality, and exploring the effects of user training and other operating factors such as fatigue and attention. See Section 6.2 for suggested future studies. 64 6.1 Summary of Contributions The main contributions were as follows: 1. Developed a system, including software and hardware, to study the LF-ASD with imaginary movements in subjects with spinal cord injuries (see Chapter 3). 2. Introduced an evaluation methodology for testing the BCI system with imagined movements (see Section 4.2 and Appendix B.4). 3. Evaluated the new system in two studies in which able-bodied as well as spinal cord injured subjects were asked to control the LF-ASD through movement imagination. This was required before any additional modifications of the system were to be implemented. The results of these two studies (presented in Chapter 5) are positive initial indications that the LF-ASD can be activated with self-paced imagined movements. 4. Introduced a method for estimating lucky hits (see Section 4.3). 5. Showed the first ensemble averaged motor-related potentials from spinal cord injured subjects using self-paced attempted movements from paralyzed parts of their body (see Figures 5.7, 5.8, 5.15 and 5.16). These plots reveal that the LF-ASD was able to detect a consistent waveform pattern, illustrating the involvement of the MI region of the motor cortex in SCI subjects attempting index finger flexion. 6. The work of this thesis provides further evidence that a BCI system based on the LF-ASD technique may be possible to assist people with high level of motor impairment to control another device through the preparation or imagination of motor-related tasks. 65 6.2 Suggested Future Work Suggested items for using the proposed system and evaluation methodologies are listed below: 1. Evaluate the performance of the LF-ASD with a larger pool of subjects with spinal cord injuries. 2. Monitor and study the potential influence of the surface EMG of the extensor muscles in the performance of the LF-ASD (see [28, page 431] for suggested methodology). 3. Explore the impacts of subject training as well as motivation, fatigue, frustration and other aspects of mental states. Suggested items for modifying the proposed system are listed below: 1. Study the effect of codebook customization based on data collected for the subject in each session, as well as other system parameter customization 2. Explore alternative classification techniques, for example Distinction Sensitive Learning Vector Quantization (DSLVQ) [42], instead of LVQ in order to update the classifier continuously with recent training data and to account for the variability of EEG patterns throughout the session. 3. Study the effect of placing a more significant weight on the electrode pairs in the SMA compared to those in the Mt for the classification of imagined movements. 4. Evaluate the performance of the LF-ASD in the detection of imagined movements of other than a right hand index finger flexion. 66 REFERENCES [1] Shneiderman, B., Designing the User Interface - Strategies for Effective Human-Computer Interaction. Massachusetts: Addison-Wesley (Second Ed.), 1992. [2] Sharma, R., Pavlovic, V. I, and Huang, T. S., "Toward Multimodal Human-Computer Interface," Proceedings of the IEEE, vol. 86, no. 5, pp. 853-869, May 1998. [3] Haasbroek, L. J. Advanced Human-Computer Interfaces and Intent Support: A Survey and Perspective," IEEE International Conference on Systems, Man and Cybernetics: System Engineering in the Service of Humans, vol. 4, pp. 350-355, 1993. [4] Wolpaw, J. R., Birbaumer, N., Heetderks, W. I., McFarland, D. J., Peckham, P. H., Schalk, G., Donchin, E., Quatrano, L. A., Robinson, C. J., and Vaughan, T. M., "Brain-Computer Interface Technology: A Review of the First International Meeting," IEEE Transactions on Rehabilitation Engineering, vol. 8, no. 2, pp. 164-173, June 2000. [5] Vaughan, T. M., Wolpaw, J. R., and Donchin, E., "EEG-Based Communication: Prospects and Problems," IEEE Transactions on Rehabilitation Engineering, vol. 4, no. 4, pp. 425-430, December 1996. [6] Mason, S. G. and Birch, G. E., "A Brain-Controlled Switch for Asynchronous Control Applications," IEEE Transactions on Biomedical Engineering, submitted September 2000. [7] Birch, G. E., Lawrence, P. D., and Hare, R. D., "Single-Trial Processing of Event-Related Potentials Using Outlier Information," IEEE Transactions on Biomedical Engineering, vol. 40, no. 1, pp. 59-73, January 1993. 67 [8] Lisogurski, D. and Birch, G. E. "Identification of Finger Flexions from Continuous EEG as a Brain Computer Interface," Proceedings of the 20th Annual International Conference of IEEE Engineering in Medicine and Biology Society, vol. 20, pp. 2004-2008, Hong Kong, 1998. [9] Wolpaw, J. R., McFarland, D. J., and Vaughan, T. M., "Brain-Computer Interface Research at the Wadsworth Center," IEEE Transactions on Rehabilitation Engineering, vol. 8, no. 2, pp. 222-226, June 2000. [10] Wolpaw, J. R., Ramoser, H., McFarland, D., and Pfurtscheller, G., "EEG-Based Communication: Improved Accuracy by Response Verification," IEEE Transactions on Rehabilitation Engineering, vol. 6, no. 3, pp. 326-333, September 1998. [11] McFarland, D., McCane, L. M., and Wolpaw, J. R., "EEG-Based Communication and Control: Short-term Role of Feedback," IEEE Transactions on Rehabilitation Engineering, vol. 6, no. 1, pp. 7-11, 1998. [12] Miner, L. A., McFarland, D. J., and Wolpaw, J. R., "Answering Questions With an Electroencephalogram-Based Brain-Computer Interface," Archives of Physical Medicine and Rehabilitation, vol. 79 pp. 1029-1033, March 1998. [13] Kalcher, J., Flotzinger, D., Pfurtscheller, G., Golly, S., and Neuper, C. "Graz Brain-Computer Interface II: Studies for an EEG-Based Communication Device," Proceedings of the RESNA'95 Annual Conference, Vancouver, Canada, 1995. [14] Pfurtscheller, G., Neuper, C, Schlogl, A., and Lugger, K., "Separability of EEG Signals Recorded During Right and Left Motor Imagery Using Adaptive Autoregressive Parameters," IEEE Transactions on Rehabilitation Engineering, vol. 6, no. 3, pp. 316-325, September 1998. [15] Pfurtscheller, G., Neuper, C, Guger, C, Harkam, W., Ramoser, H., Schlogl, B., Obermaier, B., and Pregenzer, M., "Current Trends in Graz Brain-Computer Interface (BCI) Research," IEEE Transactions on Rehabilitation Engineering, vol. 8, no. 2, pp. 216-219, June 2000. [16] Flotzinger, D., Kalcher, J., and Pfurtscheller, G., "EEG Classification by Learning Vector Quantization," Biomedizinische Technik, vol. 37, no. 12, pp. 1464-1480, 1992. [17] Sutter, E. E., "The Brain Response Interface: Communication Through Visually-Induced Electrical Brain Responses," Journal of Microcomputer Applications, vol. 15, pp. 31-45, 1992. 68 [18] Farwell, L. A. and Donchin, E., "Talking Off the Top of Your Head: Towards a Mental Prosthesis Utilizing Event-Related Brain Potentials," Electroencephalography and Clinical Neurophysiology, vol. 70, pp. 510-523, April 1988. [19] Donchin, E., Spencer, K. M., and Wijesinghe, R., "The Mental Prosthesis: Assessing the Speed of a P300-Based Brain-Computer Interface," IEEE Transactions on Rehabilitation Engineering, vol. 8, no. 2, pp. 174-179, June 2000. [20] Kiibler, A., Kotchoubey, B., Hinterberger, T., Ghanayim, N., Perelmouter, J., Schauer, M., Fritsch, C, Taub, E., and Birbaumer, N., "The Thought Translation Device: A Neurophysiological Approach to Communication in Total Motor Paralysis," Experimental Brain Research, vol. 124, no. 4, pp. 223-232, 1999. [21] Birbaumer, N., Kiibler, A., Ghanayim, N., Hinterberger, T., Perelmouter, J., Kaiser, J., Iversen, I., Kotchoubey, B., Neumann, N., and Flor, H., "The Thought Translation Device (TTD) for Completely Paralyzed Patients," IEEE Transactions on Rehabilitation Engineering, vol. 8, no. 2, pp. 190-193, June 2000. [22] Birch, G. E. and Mason, S. G., "Brain-Computer Interface Research at the Neil Squire Foundation," IEEE Transactions on Rehabilitation Engineering, vol. 8, no. 2, pp. 193-195, June 2000. [23] Wolpaw, J. R. and McFarland, D., "Multichannel EEG-Based Brain-Computer Communication," Electroencephalography and Clinical Neurophysiology, vol. 90 pp. 444.449, 1994. [24] Mason, S. G., "Detection of Single-Trial Index Finger Flexions from Continuous, Spatiotemporal EEG." Ph.D. Thesis, The University of British Columbia, 1997. [25] Lisogurski, D., "On Line Single Trial Detection of Index Finger Flexions from Spatiotemporal EEG." M.A.Sc. Thesis, The University of British Columbia, 1998. [26] Pfurtscheller, G. and Neuper, C, "Motor Imagery Activates Primary Sensorimotor Area in Humans," Neuroscience Letters, no. 239, pp. 65-68, November 1997. [27] Babiloni, F., Cincotti, F., Lazzarini, L., Millan, J., Mourino, J., Varsta, M., Heikkonen, J., Spanedda, F., and Marciani, M. G., "EEG Recognition of Imagined Hand and Foot Movements Through Signal Space Projection." First International Meeting on Brain-Computer Interface Technology: Theory and Practice, pp. 35-39, The Brain-Computer Interface (BCI) Project, The Rensselaerville Institute, Rensselaerville, New York, 1999. 69 [30] Jasper, H. H., "The Ten-Twenty Electrode System of the International Federation," Electroencephalography and Clinical Neurophysiology, vol. 10 pp. 371-375, 1958. [28] Cunnington, R., Iansek, R., Bradshaw, J. L., and Phillips, J. G., "Movement-Related Potentials Associated with Movement Preparation and Motor Imagery," Experimental Brain Research, vol. Ill pp. 429-436, February 1996. [29] Data Translation, DT2801 Series User Manual. 100 Locke Drive, Marlboro, MA, USA, 1992. [30] Electro-Cap International, Instruction Manual for the ECI Electro-Cap Electrode System. Electro-Cap International, Inc., 1011 W. Lexington Road, Eaton, Ohio, USA. [31] Jasper, H. EC., "The Ten-Twenty Electrode System of the International Federation," Electroencephalography and Clinical Neurophysiology, vol. 10 pp. 371-375, 1958. [32] Coren, S., The Left-Hander Syndrome: The Causes and Consequences of Left Handedness. New York: Free Press/Macmillan, 1992. [33] American Spinal Injury Association/International Medical Society of Paraplegia (ASIA/TMSOP), International Standards for Neurological and Functional Classification of Spinal Cord Injury. Supported by the American Paralysis Association. Chicago, Illinois, U.S.A., American Spinal Injury Association, 1996. [34] Helstrom, C. W., Probability and Stochastic Processes for Engineers. New York: Macmillan Publishing Company, 1984. [35] Bozorgzadeh, Z., Birch, G. E., and Mason, S. G. "The LF-ASD BCI: On-line Identification of Imagined Finger Movements in Spontaneous EEG with Abie-Bodied Subjects," Proceedings of the Annual International Conference on Acoustic, Speech and Signal Processing - ICASSP 2000, pp. 2385-2388, Istanbul, Turkey, June 5-9, 2000. [36] Deecke, L., Scheid, P., and Kornhuber, H., "Distribution of Readiness Potential, Pre-motion Positivity and Motor Potential of the Human Cerebral Cortex Preceding Voluntary Finger Movements," Experimental Brain Research, vol. 7, pp. 158-168, 1969. [37] Deecke, L. and Kornhuber, H., "Comparison of Bereitschaftspotential, Pre-motion Positivity and Motor Potential Preceding Voluntary Flexion and Extension Movements in Man," Progress in Brain Research Volume 54 - Motivation, Motor and 70 Sensory Processes of the Brain: Electrical Potentials, Behaviour and Clinical Use, pp. 171-176, Amsterdam: Elsevier/North-Holland Biomedical Press, 1980. [38] Kristeva, R. and Deecke, L., "Cerebral Potentials Proceeding Right and Left Unilateral and Bilateral Finger Movements in Sinistrals," Progress in Brain Research Volume 54 - Motivation, Motor and Sensory Processes of the Brain: Electrical Potentials, Behaviour and Clinical Use, pp. 748-754, Amsterdam: Elsevier/North-Holland Biomedical Press, 1980. [39] Mason, S. G., Bozorgzadeh, Z., and Birch, G. E. "The LF-ASD Brain Computer Interface: On-line Identification of Imagined Finger Flexions in Subjects with Spinal Cord Injuries," The Fourth International ACM SIGCAPH Conference on Assistive Technologies - ASSETS 2000, Washington, D.C., USA, November 13-15, 2000. [40] Pfurtscheller, G., Neuper, C, Hamoser, H., and Miiller-Gerking, J., "Visually Guided Motor Imagery Activates Sensorimotor Areas in Humans," Neuroscience Letters, vol. 269, pp. 153-156, May 1999. [41] Green, J. B., Bialy, Y., Sora, Y., and Thatcher, R. W., "An Electroencephalographic Study of Imagined Movement," Archive of Physical Medical Rehabilitation, vol. 78 pp. 578-581, 1997. [42] Pregenzer, M., Pfurtscheller, and G., Flotzinger, D., "Selection of Electrode Positions for an EEG-Based Brain Computer Interface (BCI)," Biomedizinische Technik, vol. 39, no. 10, pp. 264-269, 1994. 71 APPENDIX A TABLES 72 A.l Glossary of Abbreviations Symbol Expanded Term A/D Analog to Digital ALS Amyotropic Lateral Sclerosis ASD Asynchronous Switch Design BCI Brain-Computer Interface CRT Cathodic Ray Tube DB Scale Decision Boundary Scale DSLVQ Distinction Sensitive Learning Vector Quantization EEG Electroencephalograph EMG Electromyograph EOG Electro-oculograph ERD Event-Related Desynchronization ERP Event-Related Potential fMRI Functional Magnetic Resonance Imaging FN False Negative FP False Positive HCI Human-Computer Interface IVM Imagined Voluntary Movement IVMRP Imagined Voluntary Movement-Related Potential LF-ASD Low-Frequency Asynchronous Switch Design LH Lucky Hit LVQ Learning Vector Quantization Max DB Scale Maximum Decision Boundary Scale MEG Magnetoencephalography MI Primary Motor Area MRP Movement-Related Potential OPM Outlier Processing Method PET Positron Emission Tomography SCP Slow Cortical Potential SMA Supplementary Motor Area SNR Signal-to-Noise Ratio TH True Hit VEP Visual Evoked Potential VMRP Voluntary Movement-Related Potential A.2 Encoding Tables Table A.2.1: EOG Encoding Index Code Description 0 No EOG Artifact 1 EOG Artifact 2 Post EOG Artifact (extends for up to half a second after EOG artifact) Table A.2.2: EEG Encoding Index Code Description Ox * Idle classification lx* Active classification 13 FP due to FN report in the Operate mode 14 FP with EOG artifact due to FN report in the Operate mode 15 FP with post EOG artifact due to FN report in the Operate mode 2x* False Positive 3x * False Negative 39 FN report during the break 4x * Feedback mode 5x* Reject mode 53 FP due to FN report in the Reject mode 54 FP with EOG artifact due to FN report in the Reject mode 55 FP with post EOG artifact due to FN report in the Reject mode 56 FN with EOG artifact within the last 6 seconds of report 6x * HTT (marked at the end of Feedback mode) 90 Breaktime * x can be either 0, 1 or 2 as defined for the three cases of EOG classification of Table A.2.1 APPENDIX B PROCEDURES, INSTRUCTIONS & FORMS 75 B.l BCI System Apparatus B.l.l Equipment & Accessories • Batteries: 4 1.5 C Cells (back-up for the Grass Electrode Impedance Meter, Model EZM1E) • Bipolar electrodes for EOG recording • Custom-made Sip & Puff Switch • Electro-Cap, medium (54-58 cm), red • Electro-Cap, large (58-62 cm), blue • Electro-Cap chin strap • Grass Electroencephalograph EEG Amplifier, Model 8-18 C • Grass Mini Electrode Board (headbox), Model IGMEB-INT36 • Grass Electrode Impedance Meter, Model EZM1E • 6cc syringes with Luer lock tip • Alcohol (Iso-propyl 70%) • Alcohol prep pads (medium) • Blunt needles o Preferred: 16 x 3A" gauge - can be used for applying the ECI Electro-Gel and the Quik Gel o 18x1" gauge - can only be used for applying the ECI Electro-Gel • Container for used needles • Cotton swabs • Disposable latex gloves (medium size) • ECI Electro-Gel or Quik Gel (Electrode Gel) - 16 oz • Electrode washers (4mm ID x 19 mm OD) - adhesive (to be used for EOG electrodes) • Gauze, non-sterile, general-use sponges, 2" x 2" - 4 ply • Medical tape • Omni Prep or Nuprep (ECG & EEG Abrasive Skin Prepping Gel) - 5 oz • Antibacterial disinfecting liquid (for disinfecting the electrodes) • Disposable cups (for disinfecting the straw for the Sip & Puff Switch) • Hand towel • Measuring tape • Reading material (to occupy the subject during Subject Preparation) B.1.2 EEG Supplies B.1.3 Lab Supplies 76 Rubber band (optional - to hold down the Electrode Test switch on the impedance meter) Sip & Puff Switch accessories: straw & long tube extension for connecting the straw and filter to the Sip & Puff Switch (referred to as Sip & Puff Switch filter extension) Soap Soft tissue paper (Kleenex) Stand (to hold the Sip & Puff Switch straw & filter extension) Tape (relatively strong, e.g., electrical tape) Toothbrush (optional - to clean the electrodes) Water bottle 77 B.2 Set-Up Procedure for the BCI System B.2.1 Initial Set-Up 1. Ensure the availability of all items listed in Appendix B.l BCI System Apparatus. 2. Turn the EEG amplifier ON half an hour before the experiment so that the analog circuits warm-up and stabilize. Ensure that the EEG headbox connections and the EEG amplifier settings are set according to those set in Tables B.l and B.2. Table B.l: Electrode Montage for the Grass Electroencepha ograh EEG Amplifier, Model 8-18 C Grass A/D Electrode Headbox Wire 1 EOG 01 EOG#l (right corner of the right eye) 2 Sip & Puff Switch - -3 Af2 P3 White 1 4 Fi T5 Blue 1 5 Fz T3 Black 1 6 F2 c3 Blue 2 7 FC3 F7 Red 1 8 FC, F3 Brown 1 9 FQ FP, Grey 2 10 FC2 Fz Brown 2 11 c3 cz Orange 1 12 Ci Pz Purple 1 13 cz FP2 White 2 14 c2 F4 Purple 2 15 c4 F4 Orange 2 16 Pz F8 Black 2 Ref EEG oz c4 Green 1 ISO/GND POz Green 2 N/A Ref EOG Pd/X EOG #2 (below the right eye) Table B.2: EEG Amplifier Settings for the Grass Electroencephalograh EEG Amplifier, Moc el 8-18 C Channels )/ Amplifier Settings Low Pass Filter (Hz) High Pass Filter (Hz) Sensitivity^ (pV/mm) 1 1 35 50 2t - - -3-16 0.1 35 20 t Channel 2 is used for the Sip & Puff Switch. | Sensitivity: set all knobs to individual Channel Control (instead of the All Channel Control option) 78 Note: 60 Hz Notch Filter (Line Frequency Interference Filter): select IN for all channels (instead of the All Channel option) Power: set to Standby for all channels. 3. Perform the following tests on the EEG impedance meter: 3.1. Do a Battery Test: Press the Battery Test push-button and check that the dial reaches the appropriate setting (indicated by BAT.). 3.2. Do a 10K Cal. test: Push the 10K Cal. and Electrode Test switch UP and check the dial. If the dial is not set at 10 kf>, turn the 10K Cal. Adj. knob accordingly. 4. Place the adhesive electrode washers on the EOG electrode pairs. 5. Have the computer and the subject display monitor ready (see Steps 1- 5 in the Standard Experimental Procedure Per Session Section). 6. Place the Sip and Puff Switch filter extension on the stand. Disinfect the straw by pouring alcohol down the straw and soaking the tip in the alcohol poured into a cup. The straw tip can also be wiped with the alcohol prep pads B.2.2 Subject Preparation 1. Place the subject 100 cm away from the monitor. Confirm approximate distance with a measuring tape. 2. With the measuring tape, measure the head size and choose the appropriate Electro-Cap size: 54-58 cm (medium - red cap) and 58-62 cm (large - blue cap). For further instructions on using the Electro-Cap, refer to the Instruction Manual for the ECI Electro-Cap Electrode System [30]. 3. Place the Electro-Cap™ electrode cap on the subject's head. Ensure chinstrap is tight. 4. With the measuring tape, measure the distance between the nasion (in the front -the indented region above the nose) and the inion (in the back - the lowest point of the skull) and ensure that the Cz electrode is placed in the center. Also, make sure that the Cz is placed in center distance between the subject's ears and aligned with the subject's nose. 5. Adjust the electrodes until all are sitting nicely on the skull. Smooth all over the cap. 6. Fill in all the electrodes with the electrode gel and abrade the skin at each site. Hold down each electrode with the two fingers of one hand and gently rock the blunt needle and syringe, containing the electrode gel, back and forth with the other hand. Mention to the subject that s/he should feel the skin being abraded, but should notify you if the feeling becomes painful. 7. Quickly check the impedances of all electrodes. For easy accessibility, leave the switch on the impedance meter ON (use a tape or an elastic band to hold it down). If some of the impedances are well over 20 k£2, check that the cap is placed tightly and that the electrodes are placed flat on the skull with none of the wires blocking the contact. 79 8. Repeat Step 6 for those electrodes that have an impedance of 5 kQ, or higher. If possible bring the impedance down below 3 k£2; however, do not insist by over-abrading the skull. Note those electrodes for which the impedance could not be reduced lower than 5 kQ. 9. Finally, review the impedances and turn the impedance meter off. 10. For placing the EOG electrodes to the right and below the right eye of the subject, apply abrasive skin prepping gel to a cotton swab or non-sterile gauze and abrade the skin. After abrasion, wipe off excess gel from the skin with a clean gauze. 11. Fill in the electrodes with electrode gel and place the electrodes to the right and below the right eye of the subject via the adhesive electrode washers. Hold the electrodes with medical tape. 12. Use a stronger tape to hold the extending wires of the Electro-Cap as well as the EOG electrodes to the chair of the subject so that they don't pull back. B.2.3 Standard Experimental Procedure Per Session 1. On the khronos (Sun computer monitor for the subject display), login as zibab (no password is required) and at the prompt, choose x for X-Windows. 2. Type xhost + in one of the windows. 3. Close all the windows and place the cursor arrow at the bottom right corner away from view. 4. On the main computer, in the top window, in the ~ / directory, change directory to driver. Log in as root (su, password: Sunny Days) and type ./loadit. This will load the A/D driver. This step is only required at each computer bootup. Exit from root. 5. Type xhost +. 6. If the code has been modified and needs rebuilding, type the following commands, otherwise go to step 7. 6.1. make clean 6.2. make dep 6.3. make 7. In the bottom window, in the ~/ directory, change directory to online. Type these commands: 7.1. script <filename>.log * ** 7.2. date 7.3. cp ~/dan_copy/online/subject2/s2d2 . cod codebook. cod 7.4. go <f ilename> * *** 8. <filename> is suggested to have the following format: The initials of the subject + session date, e.g., if John Doe is tested on January 19, 2000, then <f ilename> is selected as jdl90100. Make sure that <f ilename> has not already been used, as existing files will be overwritten automatically. 9. In the online program, these initial steps are required prior to start of the testing procedure: 80 9.1. Load Calibration files, which loads the scale. cal and offset. cal files in the current directory. If the signals don't look clean, then calibrate online (key in Calibration Online). 9.2. Load Codebook, which loads codebook. cod file in the current directory. 10. Turn off all the lights in the room during the experiment. 11. After the test is completed, use the Train option if you want the feature parameters for this session to be used to compile a new codebook. 12. After quitting the session, in the bottom window, type exit to stop the script from recording. 13. Backup data files. ** script records the session. *** go contains the script that runs online (for offline implementation type goo <filename>.raw) B.2.4 Post-Experimental Clean-Up 1. Turn off the EEG amplifier. 2. Remove the EOG electrodes. 3. Disconnect the Electro-Cap from the headbox and remove the cap from the subject's head. 4. Remove the electrode gel from the subject's hair with a damp Kleenex or paper towel and pat dry with a towel. 5. Wash the Electro-Cap and EOG electrodes with disinfecting antibacterial liquid. A toothbrush can be used in removing the electrode gel off the electrodes. 6. Dispose the blunt needle and syringe in the container for used needles. B.2.5 Post-Analysis Tools statsum runs statsum. c and produces statsum. out (a summary of these classifications: true hits, false positives, false positives associated with a false negative report and false negatives; each with the three cases of 1) artifact free, 2) EOG artifact and 3) post EOG artifact (within 1/2 second after EOG artifact data)) and trialData. out (idle point counts before each possible classifications). averageHits runs averageHits. c and produces an ensemble 5-second period average of the single-trial bipolar difference FC1-C1 signal centering all the true positives (hits). It must be run after statsum since it makes use of the trialData. out file. This file can be modified to produce average of other bipolar or monopolar signals. 81 B.3 Study 1: Instruction Sheet for the Researcher B.3.1 Test Session 1: 1st 75 Trials Max estimated time: 120 minutes Purpose: • To obtain the first series of data based on 75 imagined movements of the right-hand index finger. Instructions: • Ask the subject to read the Subject Consent Form (Appendix B.5) and sign it. Interview the Subject Information Form (Appendix B.6). • Ask the subject to complete the LAT-24 R Handedness Inventory questionnaire (Appendix B.7). • Follow the instructions as laid out in the Set-Up Procedure for the BCI System (Appendix B.2). • Demonstrate the non-standard right-handed index finger flexion that was used in [24] and [25], and ask the subject to make the same movement. Then ask him/her to imagine making the movement. • Lay out the test procedure as follows: o Two balls will appear and start moving. o The display will react to a switch activation by flashing the balls as well as changing the direction of the center ball from vertical to horizontal or vice versa. A feedback period will follow immediately. o During the feedback period, the balls will be stationary, and the subject is required to react as follows: • If the subject intended for the switch activation, do nothing (CASE 1: true positive or a hit). • If the subject did not intend for the switch to be activated, report an error by either sipping or puffing using the Sip & Puff Switch (CASE 2: false positive). o If the subject attempts to activate the switch but there is no change in the display, the subject should report this error while the balls are moving (CASE 3: false negative). There is no need to let the subject know of the special case of a false positive occurring at the time a sip or puff is made for a false negative report. Just let the subject know that there is a special case, which will be best explained when observed by the subject during the experimental acquaintance period. o Ask the subject to describe the three cases. Use the OPERATE and FEEDBACK modes within online to exhibit the different modes of the display if necessary. In any case, the subject should be guided through an initial acquaintance period prior to data collection for this session. o Refrain from using false positive, false negative, etc. terminology in explaining the test procedure. o Tell the subject that it is very important to maintain consistency. 82 o Tell the subject that his/her eye blinks are continuously monitored. It is preferable if the finger flexion attempts are not made while the subject is blinking; however, the subject should not be too concerned about this. o Ask the subject if s/he has any questions. • Go through a preliminary experimental stage to acquaint the subject with the BCI system operation. This stage could take from 15 minutes to 30 minutes. • Once the subject is confident in understanding the BCI system operation, restart the raw EEG file and start recording actual session. Let the subject know that the system will automatically provide breaks during which the experimental display will be covered. However, s/he can request to take a rest any time. • After the session has been completed (a total of 75 movements have been imagined: sum of FNs and HTTs), follow the Post-Experimental Cleanup instructions (Section B.2.4) as laid out in the Apparatus & Set-Up Procedure for the BCI System (Appendix B.2). • Ask the subject to verbalize his/her approach and get feedback and comments. • Schedule next session. B.3.2 Test Session 2: 2na 75 Trials Max estimated time: 120 minutes Purpose: • To obtain the second series of data based on 75 imagined movements of the right-hand index finger. Instructions: • Interview the Subject Information Form (Appendix B.6). • Follow the instructions as laid out in the Set-Up Procedure for the BCI System (Appendix B.2). • Go through a preliminary experimental stage as in Session 1. Review the system operation as well as the movement the subject is to imagine. • Restart the raw EEG file and start recording actual session. Let the subject know that s/he can request to take a rest any time. • After the session has been completed (a total of 75 movements have been imagined: sum of FNs and FfJTs), follow the Post-Experimental Cleanup instructions (Section B.2.4) as laid out in the Apparatus & Set-Up Procedure for the BCI System (Appendix B.2). • Ask the subject to verbalize his/her approach and get feedback and comments. • Pay the subject and ask him/her to sign the receipt booklet. Give the subject a copy of the receipt as well as a photocopy of the first page of the Subject Consent Form. Thank the subject for his/her participation in this study. 83 B.4 Study 2: Instruction Sheet for the Researcher B.4.1 Session 1: Familiarization and Customization Purpose: • To introduce the on-line LF-ASD Brain Computer Interface (BCI) system to the subject. • To set the BCI system parameters customized to the subject. • For the subject to identify an approach to activate the BCI system. This session has been divided into five stages, which are introduced to the subject at the end of Stage 1. The purpose of each stage is explained to the subject only at the beginning of each stage so that s/he is not overwhelmed with too much information. 1. Introduction 2. System Setup & Calibration 3. Laying Out the Test Procedure 4. Developing a Switch Activation Approach 5. Pre-testing Stage 1: Introduction Max estimated time: 20 minutes Purpose: • To introduce the on-line LF-ASD Brain Computer Interface (BCI) system to the subject and the ultimate purpose of the BCI. • Provide history of the current system performance in order to set expectations. • Indicate the importance of the subject's participation to pave the way for future improvement. Instructions: • Lay out the context by answering the question "What are we doing?" o The original objective was to detect EEG patterns on intent to move. o Steve Mason laid out the methodology for a switch, which would be activated by patterns in the brain wave created by an actual index finger movement, o Dan Lisogurski's implemented the online version of the design, o The current system, based off Lisogurski's system, has been designed to detect imagined movements. It has so far only been tested on able-bodied subjects, and this is the initial evaluation of the switch with spinal cord injury subjects. • Set expectations: This is the first design and no online-customization has been incorporated. The results from the able-bodied subjects have shown that the switch false fires and similar behaviour is expected with spinal cord injury subjects. 84 • Define role of the subject: o As this is the first study of the design with spinal cord injury subjects, they are pioneers for leading edge technology. o The subject's contribution is essential for the advancement of the research, and it is necessary that s/he follows instructions so that the controlled data obtained from these sessions are useful in improving the system. • List and describe the purpose of the three sessions: 1. Familiarization and Customization 2. Test Session 1 - make 75 imagined movements 3. Test Session 2 - make 75 imagined movements • List the five stages of the first session. Describe the purpose of each stage only at the beginning of each stage. 1. Introduction 2. System Setup & Calibration 3. General Test Procedure 4. Development of a Switch Activation Approach 5. Pre-testing • Ask the subject to read the Subject Consent Form (Appendix B.5) and sign it. Interview the Subject Information Form (Appendix B.6). • Ask the subject to complete the LAT-24 R Handedness Inventory questionnaire (Appendix B.7) at home and bring it to the next session. If time permitted, this form can also be filled out at the end of the session. However, it is not paramount to have this form filled out for spinal cord injured subjects. • Ask the subject if s/he has any questions. Stage 2: System Set-Up & Calibration (2.a) System Setup Max estimated Time: 30 minutes Purpose: • To establish the connection of the subject to the BCI system in order to collect brain signals from the subject and transfer it to the computer. • Initialize and start up the BCI system. Instructions: • Follow the instructions as laid out in the Set-Up Procedure for the BCI System (Appendix B.2). • Explain the purpose of the equipment and accessories involved during set-up. • With online running, o Ask the subject to make various levels of sips or puffs (e.g., soft, normal, and hard blows) to ensure that the system can detect the sip or puff actions made by the subject. o Ask the subject to make several eye-blinks to ensure that the system can detect EOG artifact. If the EOG signal is not stable and drifts, make sure that 85 the EOG electrodes are not being pulled and are securely placed. Additional electrode gel can also be applied to the electrodes. (2.b) Calibration Max estimated Time: 10 minutes Purpose: • To establish an appropriate Decision Boundary (DB) setting so that a reasonable False Positive rate that does not frustrate the subject is determined. • The reasonable False Positive rate had been arbitrarily chosen as 1% over 30 seconds. Note that the DB setting that gives this FP rate is artificially high and in actual test setting, the FP rate will be lower. As a result, one can aim for achieving FP values of up to 1.5% or 1.7%. Keep in mind that the subject may become highly skeptic of the system if the FP rate is high. If the FP is low, then the subject has to report fewer false-fires (false activations) and will not become too frustrated. Instructions: • Instruct the subject to watch the balls and their collisions in the display that will appear on the monitor in front of them. The center ball only moves vertically or horizontally. The second ball moves throughout the display and collides with the center ball as well as the borders. • Tell the subject that this will be boring but it is important that s/he stays attentive and avoids drifting off or daydreaming. Tell him/her that the maximum estimated time is 10 minutes for this calibration stage. • Ask the subject to request a break should s/he get tired or lose focus. Stage 3: Laying Out the Test Procedure Max estimated time: 5 minutes Purpose: • To give a simple description of the test and tell the subject what s/he will see during the experimental procedure and how s/he should react. Approach: • Describe the test procedure as follows: o As in Stage 2(b), the balls will appear and start moving. o The display will react to a switch activation by flashing the balls as well as changing the direction of the center ball from vertical to horizontal or vice versa. A feedback period will follow immediately, o During the feedback period, the balls will be stationary, and the subject is required to react as follows: • If the subject intended for the switch activation, do nothing (CASE 1: true positive or a hit). 86 • If the subject did not intend for the switch to be activated, report an error by either sipping or puffing using the Sip & Puff Switch (CASE 2: false positive). o If the subject attempts to activate the switch but there is no change in the display, the subject should report this error while the balls are moving (CASE 3: false negative). Advise that the method for activating the switch, the attempt to move the right hand index finger, will be developed in the next stage, Stage (4). o Ask the subject to describe the three cases. Use the OPERATE and FEEDBACK modes within online to exhibit the different modes of the display if necessary. In any case, the subject should be guided through Stage (4) for the initial trials. o There is no need to let the subject know of the special case of a false positive occurring at the time a sip or puff is made for a false negative report. Just let the subject know that there is a special case, which will be best explained when observed by the subject during Stage (4). o Refrain from using false positive, false negative, etc terminology in explaining the test procedure. o Tell the subject that his/her eye blinks are continuously monitored. It is preferable if the finger flexion attempts are not made while the subject is blinking; however, the subject should not be too concerned about this. • Ask the subject if s/he has any questions. Stage 4: Developing a Switch Activation Approach Max estimated time: 30-40 minutes Purpose: • To identify an approach to activate the switch by the subject in a controlled setting. • For the subject to get familiarized with the test procedure. Instructions: • Instruct the subject to attempt a strong right hand index finger flexion and attempt to move the index finger towards the palm. Analogies such as "swatting a fly" or "striking a match" using the index finger can be introduced and if preferred by the subject, used throughout the experiment. • Mention that identifying a method for switch activation is very hard and requires extensive training. It would help to mention that other subjects required several days to acquire a sense or feel for the approach; therefore, it is important that s/he does not get discouraged if the system does not respond immediately. Tell the subject that this is a very important stage as it provides the basis for the next two sessions. • Once the subject imagines or realizes an approach, s/he should maintain consistency. Let the subject know that s/he controls the quality of the data, and the data has no value if the approach is varied from trial to trial. It is human 87 nature to experiment various approaches in the attempt of getting a response. The subject should try to just keep consistent in his/her approach. • This stage should limit the self-paced aspect of the methodology. Keeping the window of activation tight and repeating the action at a fixed rate will assist the subject in realizing and maintaining the approach. Instruct the subject to try to activate the switch after a count of 4 seconds (1-onethousand, 2-onethousand, 3-onethousand and 4-onethousand: NOW). Initially, it may be easier for the subject if you pronounced the time for action by stating NOW out loud. However, after a while, let the subject do the count and control the experiment. • Keep talking to the subject to ensure s/he is following the test procedure properly and provide guidance accordingly. Allow periods of concentration, but do not wait too long before getting the subject's feedback. • Provide more frequent breaks during the experimental stage so that if an approach does not work, it can be changed but only after discussions are made during the breaks. • A good sense of activation can be obtained after getting 50% hit rate in 10 attempts (counting after the first activation). If the system does not respond at all and the subject keeps on reporting false negatives for up to 5 minutes, then ask the subject to choose a different approach. Consider that this is a difficult request to make from subjects with spinal cord injury since their approach in trying to move their index finger is limited in possibilities. Stage 5: Pre-testing Max estimated time: 20 minutes Purpose: • To switch from the controlled mode of Stage 4 to a self-paced mode, which is representative of the settings in Sessions 2 and 3. Instructions: • Ask the subject to make 25 imagined movements (or attempts of) at a self-paced rate. • Indicate that the system will false-fire every 6-8 seconds and that s/he will see more false positives. Therefore, if the subject waits too long, the system will false-fire. It is very important that the subject does not try to test the system. • At this stage, if the false positive rate is too high (over 3%) or the true positive (hit) rate is too low (below 15%), changes in the Decision Boundary setting as well as the switch activation approach may be required and so the procedures must be repeated from Stage 2(b). If the subject is not too fatigued, continue on, otherwise another session must be scheduled to achieve the objectives of this session. Alternatively, Session 2 can be prolonged in order to realize the objectives of this session. • Ask the subject to verbalize his/her approach and get feedback and comments. • Follow the Post-Experimental Cleanup instructions (Section B.2.4) as laid out in Set-Up Procedure for the BCI System (Appendix B.2). 88 • Schedule next session. B.4.2 Session 2: 1st 75 Trials Max estimated time: 120 minutes Purpose: • To obtain the first series of data based on 75 imagined movements of the right-hand index finger. Instructions: • Interview the Subject Information Form (Appendix B.6). • Follow the instructions as laid out in the Set-Up Procedure for the BCI System (Appendix B.2). • Go through a preliminary experimental stage, which can be controlled (as in Session 1 Stage 4) or self-paced (as in Session 1 Stage 5). • Restart the raw EEG file and start recording actual session. Provide non-periodic breaks so that the subject can rest. Let the subject know that s/he can take a rest any time. • After the session has been completed (a total of 75 movements have been imagined/attempted: sum of FNs and HTTs), follow the Post-Experimental Cleanup instructions (Section B.2.4) as laid out in the Apparatus & Set-Up Procedure for the BCI System (Appendix B.2). • Ask the subject to verbalize his/her approach and get feedback and comments. • Schedule next session. B.4.3 Session 3: 2nd 75 Trials Max estimated time: 120 minutes Purpose: • To obtain the second series of data based on 75 imagined movements of the right-hand index finger. Instructions: • Interview the Subject Information Form (Appendix B.6). • Follow the instructions as laid out in the Set-Up Procedure for the BCI System (Appendix B.2). • Go through a preliminary experimental stage, which can be controlled (as in Session 1 Stage 4) or self-paced (as in Session 1 Stage 5). • Restart the raw EEG file and start recording actual session. Provide non-periodic breaks so that the subject can rest. Let the subject know that s/he can take a rest any time. • After the session has been completed (a total of 75 movements have been imagined/attempted: sum of FNs and HTTs), follow the Post-Experimental 89 Cleanup instructions (Section B.2.4) as laid out in the Apparatus & Set-Up Procedure for the BCI System (Appendix B.2). Ask the subject to verbalize his/her approach and get feedback and comments. Pay the subject and ask him/her to sign the receipt booklet. Give the subject a copy of the receipt as well as a photocopy of the first page of the Subject Consent Form. Thank the subject for his/her participation in this study. 90 B.6 Subject Information Form SUBJECT INFORMATION FORM EXTRACTION OF MOTOR-RELATED POTENTIALS FROM SINGLE-TRIAL EEG Name Address Phone Number ( ) Sex M / F Age Neurological Disorders: MS, ALS, Epilepsy, ... Medications: anti-depressant, anti-anxiety, anti-histamine, ... Head Injuries: length of unconsciousness, lasting effects, age at injury Pre-Experiment Consumption: smoking, coffee, meals, alcohol Pre/Post Injury Dexterity/Special Skills: piano, guitar, attitude (perfectionist) Other: sleep, alertness, general anxiety, stress Please do not write below this line. Subject # Day 1 Day 2 Day 3 Experiment Date Start time End time Notes: 92 B.7 LAT-24 R Handedness Inventory •» • «* « W w WW I f~ 3 i i 2 S f "O W Ii Ii 2 : 2 : sI s i 1-would 2 : hkh »ye hrough * Si 5 -o a C u k » <• m ».«:«- « 5S 5V~z «»r „ *> .2 E w k k 2-o -""S3 S • • i. •» - T, • *» W «A l> k C 2- c o u • -52 >»<J » W M k SSf*. gg • « k k • • • I i 3 • mm S3 if u a ? y s 3 5 - -• « s *«• i s k k : 3 i ? ? 2. v o o 8 o - A ^ k k «» m k k C k k O O C WW • 1 ? • • k X 3 s > • • m S I k k o e >. >» e e ? £• S Z 8 s X X • *» • o> •» »» *» «M I I —» *•» «-» •C JC U= o» o» ot 5 ae « I l I l l JC JC JC JC JC 0» O* Ot O* Ok <x « 5 ac oc II i i i I l I l l W I*. «l * I I o> o> a>a> a* o« o> a> "l "l "l "l "l "l "l "l "*! "l "l "l "l "l "l "l k * O k e • • 3 •> 3 0% O k >» m "O «• 3 O k U 3 •* O • >» 3 e c 'mS k 3 « o «» «• 3 « W» 3 V-3 O" 3 —-v» O C 3 >» m* • m m o * r«- JC c "o _ O* • *3 •»» <• 3 O C «t C M o "OC -o "O-^ "OO* * «l « C *- «- C Jrf —• * o «-» o C JC JC JC > C k «• k o» <• ** JC O JC T> C ^k JCM 3 3 o o V c c JC — im * JC o — O* » 3 U — J= 3 JCU JCM JCJC jl O* JC o o s u W ** — «* 3 *-» C -j- jCk £ o c «• !! I! II U is « 3 If II I S *~ ss s: - h •« E; •SS 5S £ 5- ^2 § il k 3 o j| 11 I 11 I I I I I 11 I 11 I if l\ a • I. St.? US s % «l I I !!! * * •«! i I i; • • — k w~ • a. m ~-Ji « c m Am «* u £ O • ma w o o WW & • c a «* e c •o 3 e jrf <— cx U 3 3 3 • — O O U a. B >« a.« w T> w & O r— •» a 3 s S 2& o « k k • » JC JC m* =l =l m+ «<• JC JC o* o* S ae i i •m* m+ W W • • I I m 3 e s w O • w — w 3 2S <« * "O JC * u •» • O JC 3 • C 3 «l ¥» O Ok JBT M >S O o B «J> o *»• w »— mi — 3*- O O « «*^> Jl «-* ^> V 3 * m m%->» JC *» J= it 0» 3 O •C 3 OJC O O >s k « j= jr w « €-» — «A 93 APPENDIX C LUCKY HIT ESTIMATION PLOTS 94 S1S1S1 10 = 5 -0 -10? 5 -0 -10? 5 -0 -1? 0.5 -0 -1? CO 0.5 -0 -1? 0.5 -0 -5? HHn-(n> HpN(n,a) -40 HFp(n) 4 f P(FN(n)) P(FP(n)) P(LH(n)) 'Number of Lucky Hits = 11 Perc. of LH/Hits = 20.37% HLH<n> 4 5 6 Time (seconds) 8 10 Figure C.l: Study 1, Able-bodied Subject 1, Session 1 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution. S1S1S2 Figure C.2: Study 1, Able-bodied Subject 1, Session 2 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution. S2S1S1 10 -"co 5 -0 10 S" 5 10V-2 5 -0 -1 TJ 0.5 »mW HFN(n,a) HFp(n) P(FN(n)) -40 0.5 -0 -1' 0.5 P(FP(n)) P(LH(n)) 5P- 'Number of Lucky Hits = 4 Perc. of LH/Hits = 8.33% HLH<n> 4 5 6 Time (seconds) 8 10 Figure C.3: Study 1, Able-bodied Subject 2, Session 1 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution. 97 S1S2S2 10| 10^ n 5 -10 2 5 -0 -=5" 0.5 -0.5 ==: 0.5 l -4 ?-0 1 ?--4 ?-1? r4 I?—*-5P 1 ?-HLH<n) 0 1 HHrr<n> _l I l_ HpN(n,a) HFp(n) P(FN(n)) _J L. _2 3_ * *~ P(FP(n)) 4 ?-P(LH(n)) _9 U _Q 1 _9 1Q _8 9 1( _9 10 j a 9 1,0 Number of Lucky Hits = 7 Perc. of LH/Hits = 12.07% 4 5 6 Time (seconds) 8 9 10 Figure C.4: Study 1, Able-bodied Subject 2, Session 2-(a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution. S2S1S1 10 nT 5 ice s' 5 -10 0 I 19-g 0.5 -0 -i9-S" 0.5 -HH„(n) HFN(n,a) HFp(n) |-^(FN(n)) P(FP(n)) -4:0 S 0.5 5P T-HLH<n> P(LH(n)) 7- ? 9 ^0 Number of Lucky Hits = 3 Perc. of LH/Hits = 9.09% 4 5 6 Time (seconds) 10 Figure C.5: Study 2, SCI Subject 1, Session 1 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution. 99 10 -"5" 5 -10& S 5 10 5^ •o 0.5 S2S1S2 HHrr<n> HFN(n,a) HLH(n) HFp(n) Number of Lucky Hits = 5 Perc. of LH/Hits = 15.15% 4 5 6 Time (seconds) 8 10 Figure C.6: Study 2, SCI Subject 1, Session 2-(a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution. 100 10 0 10 £" 5 10? 2 5 0 1 v 0.5 1 2 05 0 1* 0.5 5P-S2S2S1 HH^(n) HpN(n,a) HFp(n) n n. P(FN(n)) P(FP(n)) P(LH(n)) _9 10 -40 HLH(n) JR 7_ a 9 1,0 Number of Lucky Hits = 1 Perc. of LH/Hits = 3.70% 4 5 6 Time (seconds) 10 Figure C.7: Study 2, SCI Subject 2, Session 1 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution. 101 S2S2S2 10 = 2 5 -0 -10? 5 -0 -10? 5 -0 -1? 0.5 -0 -1? 2 0.5 -0 -1? 0.5 -0 -50-4 ?-HHrr<n> HFN(n,a) 4=0 HFp(n) P(FN(n)) 40 P(FP(n)) P(LH(n)) HLH(n) 'Number of Lucky Hits = 5 Perc. of LH/Hits = 13.89% 4 5 6 Time (seconds) 8 10 Figure C.8: Study 2, SCI Subject 2, Session 2 - (a) Smoothed Observed HIT Distribution, (b) Smoothed Estimated FN Distribution, (c) Smoothed Observed FP Distribution, (d) Estimated Probability of FNs, (e) Estimated Probability of FPs, (f) Estimated Probability of LHs, and (g) Estimated LH Distribution. 102 

Cite

Citation Scheme:

    

Usage Statistics

Country Views Downloads
United States 5 0
China 4 0
India 1 0
Canada 1 0
Turkey 1 0
Russia 1 0
City Views Downloads
Ashburn 3 0
Beijing 2 0
Guangzhou 1 0
Unknown 1 0
Shenzhen 1 0
San Francisco 1 0
Bangalore 1 0
Windsor 1 0
Redmond 1 0
Kiraz 1 0

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}
Download Stats

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0065267/manifest

Comment

Related Items