Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Towards development of a 3-state self-paced brain computer interface system Bashashati, Ali 2007

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2007-317358.pdf [ 16.2MB ]
Metadata
JSON: 831-1.0100365.json
JSON-LD: 831-1.0100365-ld.json
RDF/XML (Pretty): 831-1.0100365-rdf.xml
RDF/JSON: 831-1.0100365-rdf.json
Turtle: 831-1.0100365-turtle.txt
N-Triples: 831-1.0100365-rdf-ntriples.txt
Original Record: 831-1.0100365-source.json
Full Text
831-1.0100365-fulltext.txt
Citation
831-1.0100365.ris

Full Text

Towards Development of a 3-State Self-Paced Brain Computer Interface System by A L I B A S H A S H A T I B . S c , Sharif University of Technology, 2000 M . S c , Amirkabir University of Technology (Tehran Polytechnic), 2002 A THESIS S U B M I T T E D IN P A R T I A L F U L F I L L M E N T O F T H E R E Q U I R E M E N T S F O R T H E D E G R E E O F D O C T O R OF P H I L O S O P H Y In T H E F A C U L T Y OF G R A D U A T E S T U D I E S (Electrical and Computer Engineering) T H E U N I V E R S I T Y OF B R I T I S H C O L U M B I A September 2007 © Ali Bashashati, 2007 Abstract Brain computer interface (BCI) systems aim at helping individuals with motor disabilities by providing them the ability to control devices such as wheelchairs and computers, using their brain activity only. The majority of B C I research to-date has focused on developing technology for "synchronous" BCIs . These systems allow the user to affect control during specified time periods only. Self-paced (asynchronous) BCIs on the other hand, are designed to respond whenever the user intends to control the system, otherwise they remain in the so called "inactive output state". This dissertation pursues two main objectives: 1) improving the performance of the existing 2-state self-paced B C I system developed at the Ne i l Squire Society, Vancouver, Canada (initial evaluations of this system on eight subjects showed mean true positive (TP) rates of 51.3% and 27.5% at false positive (FP) rates of 2% and 1%, respectively.) and 2) designing the first 3-state self-paced B C I . A t first, a comprehensive survey of signal processing algorithms in B C I systems is conducted. This survey is the first comprehensive review that covers more than 300 B C I published papers and introduces a taxonomy for signal processing in B C I systems. , To achieve the first objective, four separate studies related to the feature extraction and feature classification blocks of the 2-state self-paced B C I are conducted. These studies increase the mean TP rate of the existing system to 73.5% and 47.3% at the FP rates of 2% and 1%, respectively. In these studies, the users were not allowed to control the B C I in 15-34% of the time due to the presence of eye blinks. Thus, another study is also conducted to evaluate the system when the users were allowed to control the output even during eye blinks. Results show slight decrease in TP rates (mean TP rates of 68.0% and 40.6% at the FP rates of 2% and 1%, respectively) with the advantage of providing full control of the system. To achieve the second objective, two new set of movements (right and left hand extension movements) which have not been previously used in the context of B C I systems are used to control the new 3-state self-paced B C I . Results on four able-bodied subjects show significant improvements in detecting the presence of a movement when the system is used in the context of a 2-state self-paced B C I . The mean TP rate is 73.4% at the FP rate of 1%. Initial evaluations of the proposed 3-state self-paced B C I show promise with mean right and left true positive rates of 42.2% and 51.9% at a false positive rate of 1%. i i Table of Contents Chapter 1 Introduction . • 1 1.1. Functional Model of a B C I system 2 1.2. Background 5 1.2.1. Self-Paced and Synchronized B C I Systems 9 1.2.2. B C I Research at the Ne i l Squire Society (NSS) and the University of British Columbia ( U B C ) 13 1.3. Objectives and Contributions 15 1.4. Thesis Scope and Organization 18 1.5. References 21 Chapter 2 A Survey of Signal Processing Algorithms in Brain-Computer Interfaces Based on Electrical Brain Signals 29 2.1. Introduction 29 2.2. General Structure of a B C I System 30 2.2.1. Electrophysiological Sources of Control in Current BCIs 32 2.3. Methods... 34 2.4. Results 37 2.5. Discussion 49 2.5.1. Signal Enhancement 53 2.5.2. Feature Extraction 55 2.5.3. Feature Selection 57 2.5.4. Feature Classification 58 2.5.5. Post-Processing 60 2.6. Conclusions 62 2.7. References 63 2.8. Appendix A . Index of Terms 94 Chapter 3 Brain Computer Interface (BCI) Design for Asynchronous Control Applications: Improvements to the L F - A S D Asynchronous Brain Switch 96 3.1. Introduction 96 3.2. New Asynchronous Brain Switch Designs 99 3.3. Description of New Components and Methods 100 3.4. Evaluation Methods 102 3.5. Results and Discussion 103 3.5.1. Debounce Analysis 103 3.5.2. New vs. Old Transducer Designs 105 3.5.3. Spinal Cord-Injured Subjects vs. Abie-Bodied Subjects 108 3.6. Conclusion 109 3.7. References 110 Chapter 4 A Comparative Study on Generating Training-Data for Self-Paced Brain Interfaces 113 4.1. Introduction 113 4.2. Data Generation Methods 115 4.2.1. Parzen-Based Training-Data Generation ( T R G e n l ) 116 4.2.2. kMeans-Based Training-Data Generation (TR_Gen2) 118 4.2.3. Averaged kMeans-Based Training-Data Generation (TR_Gen3) 118 4.3. Evaluation 119 i i i 4.3.1. Experimental Data 120 4.3.2 Method of Evaluation 121 4.4. Results 123 4.5. Discussion 125 4.6. References 128 Chapter 5 User Customization of the Feature Generator of an Asynchronous Brain Interface 131 5.1. Introduction 131 5.2. Background 133 5.2.1. L F - A S D Feature Generator 134 5.3. Data Recording 135 5.4. Methods 136 5.4.1. Delay Parameter Customization 136 5.4.2. Cross-validation and Stratified Cross-validation 138 5.5. Results 140 5.6. Conclusions 147 5.7. References 150 Chapter 6 A n Improved Asynchronous Brain Interface: Making Use of the Temporal History of the L F - A S D Feature Vectors 152 6.1. Introduction 152 6.2. Proposed design of the L F - A S D - V 5 156 6.2.1. Path Template Generation 157 6.3. Evaluation Method 158 6.4. Results 161 6.5. Conclusions 166 6.6. References 168 Chapter 7 Effects of Eye Blinks on a Self-paced Brain Interface Design 170 7.1. Introduction 170 7.2. Methods '. 173 7.2.1. Brain Interface Designs 173 7.2.2. Experiments 175 7.2.3. Evaluation 177 7.3. Results 179 7.4. Discussion 187 7.5. References 189 Chapter 8 Towards Development of a 3-State Self-Paced Brain Interface Based on Hand Extension Movements 191 8.1. Introduction 191 8.2. Experimental paradigm 195 8.3. Proposed 3-state self-paced Brain computer interface 196 8.3.1. Detector 1 197 8.3.2. Detector 2 200 8.4. Evaluation 201 5. Results 202 8.5. Discussion 204 8.6. References 207 Chapter 9 Improving the Performance of a 3-State Self-Paced Brain Computer Interface ..210 iv 9.1. Introduction 210 9.2. 3-State Self-Paced B C I 213 9.2.1. Detector 1 (DET1) 214 9.2.2. Detector 2 (DET2) 217 9.3. Experimental Paradigm 218 9.4. Evaluation 219 9.5. Results 221 9.6. Conclusions 223 9.7. References 226 Chapter 10 Summary and Conclusions 229 10.1. Summary and Concluding Remarks 229 10.2. Summary of Contributions 232 10.3. Discussions and Future Directions 234 10.4. Applications of 2-State and 3-State Self-Paced BCIs 237 10.5. References 239 Appendix A . U B C Research Ethics Board Certificate 241 Appendix B . Expanded Tables of Chapter 2 242 Appendix C. Details of Methods 257 C . l . Linear Discriminnat Analysis (Lachenbruch 1975) 257 C.2. Principal Components Analysis (Jolliffe 2002) ..258 C.3. K-Means Clustering (Hartigan 1979) 259 C.4. Parzen Probability Density Estimation Method (Parzen 1962) 260 C.5. Description of the L F - A S D Feature Generator (Mason and Birch 2000) 261 C.6. Fuzzy Adaptive Resonance Theory (Carpenter et al 1991) 262 C.7. Learning Vector Quantization (Kohonen 1990) 263 C.8. k-Nearest Neighbor Classification: 264 C.9. References 264 v List of Tables Table 1.1 Description of the B C I transducer components 4 Table 2.1 Taxonomy for B C I Transducer designs 32 Table 2.2 Electrophysiological activities used in B C I designs and their definitions 33 Table 2.3 Pre-processing (signal enhancement) methods in B C I designs 39 Table 2.4 Feature selection/dimensionality reduction methods in B C I designs 40 Table 2.5 Feature extraction methods in B C I designs, refer to appendix 2 for A more detailed version of this table 41 Table 2.6 Feature classification methods in B C I designs that use E E G and E C o G recording technology 45 Table 2.7 Feature classification methods in B C I designs that are based on neural cortical recordings 48 Table 2.8 Post processing methods in B C I designs 49 Table 2.9 Index of Terms 94 Table 3.1 Configurations of the New Asynchronous Brain Switch Designs, where n L F - A S D = normalized-Low-Frequency Asynchronous Switch Design, E N T = Energy Normalization Transform, K L T = Karhunen-Loeve Transform, fuzzy A R T = fuzzy Adaptive Resonance Theory, L V Q 3 = Learning Vector Quantization 100 Table 3.2 Mean True Positive Rates For A l l Subjects With Different Debounce Values, where n L F - A S D = normalized-Low-Frequency Asynchronous Switch Design, S T D = standard deviation of True Positive Rates, s = seconds, Tdb.= debounce period. Note: all False Positive rates = 2% 105 Table 3.3 True Positive Rates for Individual Subjects. Note: all False Positive rates = 2%. Debounce values of 16 are shown 107 Table 4.1 Area under R O C curve (for FP<2%) averaged over 10 runs of the algorithms for each subject for different training-data generation methods 123 Table 4.2 Overall significance level (p-value) of improvements and percentage of improvements (in parentheses) of the area under R O C curve (for FP<2%) when using the proposed training-data generation method. Items that has ' - ' indicate performance decrease and thus no values is reported 124 Table 4.3 Average true positive rates (across eight subjects) at fixed false positive rates of 1 % and 2% for the training-data generation methods 124 Table 5.1 Delay parameters (measured in seconds) estimated from the ensemble averages of the bipolar E E G signals (F2-FC2) generated from the training data set. Note that the same delay parameters are used for all six bipolar channels 138 Table 5.2 True positive (TP) rates (%) at false positives (FP) of 2% for different training and test sets. TSi represents the case in which the data of the ith session were used for training and the other sessions for testing 139 Table 5.3 Performance of the L F - A S D together with standard deviations across 10 runs of stratified cross-validation with original and new delay parameters 143 Table 5.4. Average standard deviation (STD) o f T P rates (at fixed FP=2%) across each cross-validation set 144 Table 6.1. Details of the eeg data of the evaluation set that were blocked by the test system due to eye-blink 161 Table 6.2 True positive (tp) rates at fixed False Positive (fp) rates of 1% and 2% for able-bodied subjects 164 v i Table 6.3 True Positive (tp) rates at fixed False Positive (fp) rates of 1% and 2% for SCI subjects 164 Table 7.1 Details of the E E G data of the evaluation set that were blocked by the test system due to eye-blinks 172 Table 7.2 True positive (TP) rates at fixed False Positive (FP) rates of 1% and 2% for both able-bodied and spinal cord injured subjects for L F - A S D - V 4 and L F - A S D - V 5 designs using Case 1 and Case 3 evaluation methods 180 Table 7.3 True positive (TP) rates at fixed False Positive (FP) rates of 1% and 2% for both able-bodied and spinal cord injured subjects for L F - A S D - V 4 and L F - A S D - V 5 designs using Case 2 and Case 3 evaluation methods 181 Table 7.4 True positive (TP) rate (%) change of the L F - A S D - V 4 when eye-blinks are present (LF-ASD-V4-Case3) versus the original L F - A S D - V 4 when the output is blocked due to eye-blinks (LF-ASD-V4-Case2) 181 Table 7.5 True positive (TP) rate (%) change of the L F - A S D - V 5 when eye-blinks are present (LF-ASD-V5-Case3) versus the original L F - A S D - V 5 when the output is blocked due to eye-blinks (LF-ASD-V5-Case2) 181 Table 7.6 True positive rate (%) change of the L F - A S D - V 5 when eye-blinks are present ( L F -ASD-V5-Case3) versus the original L F - A S D when the output is blocked due to eye-blinks (LF-ASD-V4-Case2) 182 Table 8.1 Estimated aj and aj parameters for each subject. Note all values are in milliseconds. .' 198 Table 8.2 Percentage of true positives (TPic) at fixed false positive rate of 1% for the two designs of DET1 202 Table 8.3 Percentage of right and left true positives ( T P R and T P L ) of the two proposed 3-state brain computer interfaces (when false positive rate is set at 1%). The T P R and T P L values of the best design combination for each subject is highlighted 203 Table 8.4 Best design combination for each subject together with the performances of the 2-state and 3-state systems (at false positive of 1%), where A = < D E T 1 - L F - 1 N N + D E T 2 -P S D - L D A > and B = < D E T 1 - P S D - L D A + D E T 2 - P S D - L D A > 203 Table 9.1 Percentage of true positives (TPic) at fixed false positive rate of 1% for the two designs of DET1 222 Table 9.2 Percentage of right and left true positives (TPR and TPL) of the two proposed 3-state brain computer interfaces (when false positive rate is set at 1%). The TP R and TPL values of the best design combination for each subject is highlighted 222 Table 10.1 comparison between 2-state self-paced B C I systems 231 Table 10.2 Comparison between multi-state bei systems 232 Table B.0.1 Feature extraction methods in B C I designs 242 Table B.0.2 Feature classification methods in B C I designs that use E E G and E C o G recording technology 248 Table B.0.3 Feature classification methods in B C I designs that are based on neural cortical recordings 255 v i i List of Figures Figure 1.1. Simple block diagram of a B C I system 2 Figure 1.2 Functional model of a B C I system depicting the principle functional components, where amp = amplifier. Note the 'Control Display' is optional 4 Figure 1.3 (a) Self-paced brain computer interface operating paradigm, (b) synchronized brain computer interface paradigm 11 Figure 1.4 High-level design of L F - A S D brain computer interface. The 'Feature Classifier' continually monitors the 6-dimensional feature vectors and when a feature vector enters a pre-defined sub-space " IC" , the output of the classifier is activated (IC state). Otherwise, the output state remains inactive (NC state) 14 Figure 1.5 Sample outputs of the 2-state B C I (the L F - A S D ) and proposed 3-state B C I 17 Figure 1.6 Organization of chapters 21 Figure 2.1 Functional model of a B C I system (Mason and Birch 2003). Note the control display is optional. This review focuses on the shaded components of B C I systems 31 Figure 2.2 Signal-enhancement, feature selection and post-processing methods in B C I designs 50 Figure 2.3 Feature extraction methods in B C I designs based on Sensorimotor Activity, V E P , P300, SCP, Response to Mental Tasks, Activi ty of Neural Cells, and Multiple Neuromechanisms 51 Figure 2.4 Feature classification methods in B C I designs. 'In/Out Model ing ' refers to those ANC-based BCIs that directly map the input neural recordings to the output without using a feature extraction technique 52 Figure 3.1 Functional model of a B C I System depicting the principle functional components (Mason and Birch 2003, Mason et al 2003). Note the Control Display is optional 97 Figure 3.2 General asynchronous system control sequence 98 Figure 3.3 Components of n L F - A S D transducers with new components shown in gray, where E N T = Energy Normalization Transform, K L T = Karhunen-Loeve Transform, and 1 -N N = 1-Nearest Neighbor 99 Figure 3.4 Distribution of false switch activations. A . Schematic diagram showing the metrics used to characterize false switch activations: L i = length of the ith switch activation block (i.e. the number of activations that occur in consecutive sample times); and Pj = the ith inter-actiyation period (i.e. the distance from start of the ith activation block to the end of the ith+l block). B . Histogram of switch activation block length (Li values). C. Histogram of inter-activation period ( P j values). The arrows indicate the primary debounce values used in evaluation of the new A B S designs 101 Figure 3.5 R O C curves for Subject SC2 using system design n L F - A S D - A . A . Analysis of the 8 different values of debounce (Tdb) used in this study with corresponding Areas under the R O C curve to indicate overall system performance. B . A n expanded view of A . Only those values of false positives below 8% are shown for clarity 104 Figure 3.6 R O C curves for Subject SC2 with debounce set to 16 samples. Analysis of the five different processing methods used in this study. Only those values of false positives below 8% are shown for clarity 106 Figure 3.7 R O C curve of the true positive rate means for all 8 subjects (depicted with standard error of the mean error bars) with debounce set to 16 samples. Solid line = L F -A S D . Dashed line = n L F - A S D - A . The difference is significant (p < 0.05, 2-way A N O V A ) . Only those values of false positives below 8% are shown for clarity 107 v i i i Figure 3.8 R O C curve of the mean true positive rates for A B subjects vs. SCI subjects (depicted with standard error of the mean error bars) with debounce set to 16 samples. Solid line = able-bodied subjects. Dashed line = spinal cord-injured subjects. A . System design n L F - A S D - A . B . System design L F - A S D 109 Figure 4.1 Structure of an IC trial, where T E I C : time of the expected IC, E R W : expected response window, IC: intended control, N C : no control, Tcue- time of the cue appearance 116 Figure 4.2 IC state training-data generation using T R _ G e n l method, IC: intended control, N C : no control, F V : feature vector, /NC(X)'- probability density function of N C features. '. 117 Figure 4.3 IC state training-data generation using TR_Gen2 method, IC: intended control, N C : no control, /NC(X)'. probability density function of N C features 118 Figure 4.4 Components o f the Low Frequency Asynchronous Switch Design (the L F - A S D ) (from (Borisoff et al 2004)). E N T : energy normalization transform, K L T : Karhounen-Loeve transform, 1-NN: one nearest neighbour 120 Figure 4.5 Average R O C curves (for false positives less than 2%, as we are interested in false positives less than 2%) across eight subjects for the seven training-data generation methods (TR_Genl (dashed line with rectangles), TR_Gen2 (dashed line with plus signs), TR_Gen3 (solid line with circles), T R T E I C (solid line with stars), T R 3 T E I C (solid line with lozenges), TR_A11 (solid line with crosses), T R R a n (dashed line with triangles)) 125 Figure 5.1 Components of the L o w Frequency Asynchronous Switch Design (the L F - A S D ) (from (Borisoff et al 2004)).ENT: energy normalization transform, K L T : Karhounen-Loeve transform, 1-NN: one nearest neighbor 134 Figure 5.2 Desired pattern of the bipolar E E G during movement. Ei(n): amplitude difference between local maximum before the movement and local minimum after the movement, E/n): amplitude difference between local maximum before the movement and local minimum before the movement 134 Figure 5.3 The ensemble averages of the normalized E E G in bipolar channel F2-FC2 for subject A B 2 . Time of the expected movement ( T E M ) is at t=1.8 s 137 Figure 5.4 2-dimensional output of the feature generator for normalized bipolar E E G channel F2-FC2 for subject A B 2 for the customized delay parameters (solid line: feature values of the first dimension, dashed line: feature values of the second dimension, and solid line with cross: bipolar E E G ) . Note that the vertical line at t=l .8 s corresponds to the time of the attempted movement ( T E M ) 141 Figure 5.5 2-dimensional output of the feature generator for normalized bipolar E E G channel F2-FC2 for subject A B 2 for the original delay parameters (solid line: feature values of the first dimension, dashed line: feature values of the second dimension, and solid line with cross: bipolar E E G ) . Note that the vertical line at t=l .8 s corresponds to the time of the attempted movement ( T E M ) 142 Figure 5.6 R O C curve for subject A B I (solid line with circles: R O C curve for the customized B I system, solid line: R O C curve for the original B I system). Note that, for clarity, only false positives below 4% are shown 144 Figure 5.7 R O C curve for subject A B 2 (solid line with circles: R O C curve for the customized BI system, solid line: R O C curve for the original B I system). Note that, for clarity, only false positives below 4% are shown 145 ix Figure 5.8 R O C curve for subject A B 3 (solid line with circles: R O C curve for the customized B I system, solid line: R O C curve for the original BI system). Note that, for clarity, only false positives below 4% are shown 146 Figure 5.9 R O C curve for subject SCI1 (solid line with circles: R O C curve for the customized BI system, solid line: R O C curve for the original BI system). Note that, for clarity, only false positives below 4% are shown 147 Figure 6.1 Functional model of a BI system depicting the principle functional components. More details can be found in (Mason and Birch 2003) 153 Figure 6.2 (a) Ensemble averages of the E E G of subject ID and (b) the corresponding feature vectors over time. The attempted movement is at t=l .875s 155 Figure 6.3 Components of the L F - A S D - V 5 transducer, where E N T = Energy Normalization Transform, K L T = Karhunen-Loeve Transform, and 1-NN = 1-Nearest Neighbour.... 158 Figure 6.4 Path template generation procedure (also known as classifier training) 158 Figure 6.5 Structure of each 2-minute sub-session of E E G recording, where TEM=time of the expected (attempted) movement 160 Figure 6.6 Path templates for subject L B for the two cases 1) movement (IC) state (lines with circles) and 2) no movement (NC) state (lines with crosses). Time=0 corresponds to the time of the expected (attempted) movement (TEM) . Note that the algorithm estimates three path templates per class. However, as can be seen in this figure, the three path templates of the N C state 162 Figure 6.7 Path templates for subject ID for the two cases 1) movement (IC) state (lines with circles) and 2) no movement (NC) state (lines with crosses). Time=0 corresponds to the time of the expected (attempted) movement (TEM) . Note that the algorithm estimates three path templates per class. However, as can be seen in this figure, two of the three path templates of the IC state were very similar and are seen as one template. This is also can be seen for N C state path templates 163 Figure 6.8 Truncated R O C curves for SCI subject B K for L F - A S D - V 4 system (Dashed line) and our proposed L F - A S D - V 5 design (Solid line with circles) 165 Figure 6.9 Truncated R O C curves for SCI subject R T for L F - A S D - V 4 system (Dashed line) and our proposed L F - A S D - V 5 design (Solid line with circles) , 166 Figure 7.1 Components of the L F - A S D - V 4 transducer (Borisoff et al., 2004), where E N T = Energy Normalization Transform, K L T = Karhunen-Loeve Transform, 1-NN = 1-Nearest Neighbour, amp = amplifier, and FVv4 = extracted feature vector 174 Figure 7.2 Desired pattern of the bipolar E E G during movement, where e(n) = amplitude of the signal, Ei(n) = amplitude difference between local maximum before the movement and local minimum after the movement, E/n) = amplitude difference between local maximum before the movement and local minimum before the movement, t=n is the time of the expected movement attempt, and a; and aj are the delay parameters 174 Figure 7.3 Components of the L F - A S D - V 5 transducer (Bashashati et al., 2006 a), where E N T = Energy Normalization Transform, K L T = Karhunen-Loeve Transform, 1 - N N = 1 -Nearest Neighbour, amp = amplifier, F V V 4 = extracted feature vector of L F - A S D - V 4 , and F V V 5 = extracted feature vector of L F - A S D - V 4 175 Figure 7.4 Structure of each 2-minute sub-session of E E G recording, where T E M = time of the expected (attempted) movement, N C = no-control, and IC = intentional control. ..176 Figure 7.5 (a) Periods of intended control. Total length of experiment, in seconds, is T. (b) Timing of L F - A S D activations, (c) Timing of ocular artifacts detected using E O G electrodes, where IC = intentional control, O A i ^ = ocular artifact periods 178 x Figure 7.6 True positive rate change (at false positive rate of 2%) of the L F - A S D - V 4 and L F -A S D - V 5 designs between Case 2 and Case 3 evaluation methods 182 Figure 7.7 Truncated receiver operation characteristic (ROC) curves for subject ID (line with lozenges: LF-ASD-V5-Case2 , line with circles: LF-ASD-V4-Case2 , line with stars: L F -ASD-V5-Case3 , line with triangles: LF-ASD-V4-Case3) . A s we are interested in lower FP rate levels, only those values of FPs below 3% are shown in the R O C curves. The numbers in the parentheses show the false positive and true positive percentages, respectively, for identical classification parameters for LF-ASD-V5-Case2 and L F -ASD-V5-Case3 183 Figure 7.8 Truncated receiver operation characteristic (ROC) curves for subject C B (line with lozenges: LF-ASD-V5-Case2 , line with circles: LF-ASD-V4-Case2 , line with stars: L F -ASD-V5-Case3 , line with triangles: LF-ASD-V4-Case3) . A s we are interested in lower FP rate levels, only those values of FPs below 3% are shown in the R O C curves. The numbers in the parentheses show the false positive and true positive percentages, respectively, for identical classification parameters for LF-ASD-V5-Case2 and L F -ASD-V5-Case3 183 Figure 7.9 Truncated receiver operation characteristic (ROC) curves for subject CS (line with lozenges: LF-ASD-V5-Case2 , line with circles: LF-ASD-V4-Case2 , line with stars: L F -ASD-V5-Case3 , line with triangles: LF-ASD-V4-Case3) . A s we are interested in lower FP rate levels, only those values of FPs below 3% are shown in the R O C curves. The numbers in the parentheses show the false positive and true positive percentages, respectively, for identical classification parameters for LF-ASD-V5-Case2 and L F -ASD-V5-Case3 184 Figure 7.10 Truncated receiver operation characteristic (ROC) curves for subject K T (line with lozenges: LF-ASD-V5-Case2 , line with circles: LF-ASD-V4-Case2 , line with stars: LF-ASD-V5-Case3 , line with triangles: LF-ASD-V4-Case3) . A s we are interested in lower FP rate levels, only those values of FPs below 3% are shown in the R O C curves. The numbers in the parentheses show the false positive and true positive percentages, respectively, for identical classification parameters for LF-ASD-V5-Case2 and L F -ASD-V5-Case3 184 Figure 7.11 Truncated receiver operation characteristic (ROC) curves for subject A J (line with lozenges: LF-ASD-V5-Case2 , line with circles: LF-ASD-V4-Case2 , line with stars: LF-ASD-V5-Case3 , line with triangles: LF-ASD-V4-Case3) . A s we are interested in lower FP rate levels, only those values of FPs below 3% are shown in the R O C curves. The numbers in the parentheses show the false positive and true positive percentages, respectively, for identical classification parameters for LF-ASD-V5-Case2 and L F -ASD-V5-Case3 185 Figure 7.12 Truncated receiver operation characteristic (ROC) curves for subject L B (line with lozenges: LF-ASD-V5-Case2 , line with circles: LF-ASD-V4-Case2 , line with stars: LF-ASD-V5-Case3 , line with triangles: LF-ASD-V4-Case3) . A s we are interested in lower FP rate levels, only those values of FPs below 3% are shown in the R O C curves. The numbers in the parentheses show the false positive and true positive percentages, respectively, for identical classification parameters for LF-ASD-V5-Case2 and L F -ASD-V5-Case3 185 Figure 7.13 Truncated receiver operation characteristic (ROC) curves for subject B K (line with lozenges: LF-ASD-V5-Case2 , line with circles: LF-ASD-V4-Case2 , line with stars: LF-ASD-V5-Case3 , line with triangles: LF-ASD-V4-Case3) . A s we are interested in x i lower FP rate levels, only those values of FPs below 3% are shown in the R O C curves. The numbers in the parentheses show the false positive and true positive percentages, respectively, for identical classification parameters for LF-ASD-V5-Case2 and L F -ASD-V5-Case3 186 Figure 7.14 Truncated receiver operation characteristic (ROC) curves for subject R T ((line with lozenges: LF-ASD-V5-Case2 , line with circles: LF-ASD-V4-Case2 , line with stars: LF-ASD-V5-Case3 , line with triangles: LF-ASD-V4-Case3) . A s we are interested in lower FP rate levels, only those values of FPs below 3% are shown in the R O C curves. The numbers in the parentheses show the false positive and true positive percentages, respectively, for identical classification parameters for LF-ASD-V5-Case2 and L F -ASD-V5-Case3 186 Figure 8.1 Samples of outputs of 2-state and 3-state self-paced BCIs , where N C = N o Control state, IC = Intended Control state 193 Figure 8.2 Screen contents for each of the right hand (a) and left hand (b) extension movement trials, t=0 is the time of movement execution 196 Figure 8.3 Structure of the 3-state self-paced brain computer interface design 197 Figure 8.4 Structure of the two designs of D E T 1 , where K L T = Karhunen-Loeve Transform, and 1-NN = 1-Nearest Neighbour, P S D = power spectral density, and L D A : linear discriminant analysis 197 Figure 8.5 Description of delay terms (a t , a j ) , where e(n) is the amplitude of the bipolar signal 198 Figure 8.6 Structure of D E T 2 - P S D - L D A , where P S D = power spectral density, and L D A = linear discriminant analysis 200 Figure 9.1 Sample of output of a 3-state self-paced BCIs , where N C = No Control state, IC = Intended Control state.. 212 Figure 9.2 Overall structure of the 3-state self-paced B C I design 214 Figure 9.3 Structure of the two designs of D E T 1 , where K L T = Karhunen-Loeve Transform, and 1-NN = 1-Nearest Neighbour, P S D = power spectral density, and L D A : linear discriminant analysis. Note, shaded designs are new designs of DET1 implemented in this study 217 Figure 9.4 Structure of D E T 2 , where L D A = Linear Discriminant Analysis, P S D = Power Spectral Density 218 Figure 10.1 Example of a B C I user interface 239 Fig. C . l . Desired pattern of the bipolar E E G during movement 261 x i i Acknowledgements I would like to express my deepest gratitude to my supervisors, Drs. Rabab Ward and Gary Birch, for providing the opportunity to work under their supervision. Their constant guidance, patience and encouragements have been of great value to me. M y sincere thanks to my committee member, Dr. Peter Lawrence, for his constructive criticism and advice during this thesis. I wish to thank Drs. Steve Mason and Jaimie Borisoff for their help, useful comments and criticisms during this work. I also wish to thank my friend and colleague Borna Noureddin and other lab-mates in the Brain Interface Lab at the G F Strong Hospital for their useful discussions throughout this work. Finally, my loving thanks to my wife, my Mehrnoush, for her support during this work and for being with me in the good and bad days. Without her, it would be impossible to get to this point. I dedicate this thesis to my parents who believed in me and the countless sacrifices they made to make my dreams come true. x i i i Co-Authorship Statement 1- A l i Bashashati, Mehrdad Fatourechi, Rabab K . Ward, and Gary E . Birch, " A Survey of Signal Processing Algorithms in Brain-Computer Interfaces Based on Electrical Brain Signals," Journal of Neural engineering, vol . 4, no. 2, pp. R35-55, Jun 2007. A B reviewed all papers, interpreted the results, wrote manuscript and acted as the corresponding author. M F helped in manuscript writing, results interpretation and manuscript evaluation. R W and G B supervised development of work, helped in manuscript writing and manuscript evaluation. 2- J. Borisoff, S. Mason, A . Bashashati, and G . Birch, "Brain Computer Interface (BCI) Design for Asynchronous Control Applications: Improvements to the L F - A S D Asynchronous Brain Switch," IEEE Trans. Biomed. Eng., vol . 51, no. 6, pp. 985-92, June 2004. A B developed codebook generation mechanisms and dimensionality reduction algorithms, helped in writing and evaluation of the manuscript. JB developed other signal processing components, analyzed the data, interpreted the results, wrote manuscript and acted as corresponding author. S M contributed to data interpretation and manuscript evaluation. G B supervised development of work, helped in results interpretation and manuscript evaluation. 3- A l i Bashashati, S.G. Mason, Jaimie Borisoff, Rabab K . Ward, and G . Birch, " A Comparative Study on Generating Training-Data for Self-Paced Brain Interfaces," IEEE Trans. Neural Systems and Rehabilitation, vol . 15, no. 1, Mar 2007. A B developed the methods, analyzed the data, interpreted the results, wrote manuscript and acted as the corresponding author. S M and JB contributed to results interpretation and manuscript evaluation. R W and G B supervised development of work, helped in data interpretation and manuscript evaluation. 4- A . Bashashati, M . Fatourechi, R. Ward, and G . Birch, "User Customization of the Feature Generator of an asynchronous Brain Interface," Annals of Biomedical Engineering, vol . 34, no. 6, pp. 1051 -60, June 2006. xiv A B developed the methods, analyzed the data, interpreted the results, wrote manuscript and acted as the corresponding author. M F contributed to study design, helped in shaping the manuscript, results interpretation and manuscript evaluation. R W and G B supervised development of work, helped in manuscript writing and evaluation. 5- A . Bashashati, S.G. Mason, Rabab K . Ward, and G . Birch, " A n Improved Asynchronous Brain Interface: Making Use of the Temporal History of the L F - A S D Feature Vectors," Journal of Neural Engineering, vol . 3, no. 2, pp. 87-94, June 2006 (Invited paper). A B developed the methods, analyzed the data, interpreted the results, wrote manuscript and acted as the corresponding author. S M contributed to the development of the initial concept, results interpretation and manuscript evaluation. R W and G B supervised development of work, helped in results interpretation and manuscript evaluation. 6- A l i Bashashati, Borna Noureddin, Rabab Ward, Peter Lawrence, and Gary Birch, "Effects of Eye Blinks on the Performance of a Self-paced Brain Interface," Clinical Neurophysiology, vol . 118, no. 7, pp. 1639-47, July 2007. A B developed the methods, analyzed the results, interpreted the results, wrote manuscript and acted as the corresponding author. B N contributed to development of evaluation methods, results interpretation and manuscript evaluation. P L , R W and G B supervised development of work, helped in manuscript writing, results interpretation and manuscript evaluation. 7- A l i Bashashati, Rabab Ward, and Gary Birch, "Towards Development of a 3-State Self-Paced Brain Interface Based on Hand Extension Movements," J Computational Intelligence & Neuroscience, In press (Accepted in A u g 2007). A B developed the 3-state B C I system, collected and analyzed the E E G data, interpreted the results, wrote manuscript and acts as the corresponding author. R W and G B supervised development of work, helped in manuscript writing, results interpretation and manuscript evaluation. xv 8- A l i Bashashati, Rabab Ward, and Gary Birch, "Improving the Performance of a 3-State Self-Paced Brain Computer Interface," to be submitted. A B developed the system, collected and analyzed the E E G data, interpreted the results, wrote manuscript and w i l l act as the corresponding author. R W and G B supervised development of work, helped in results interpretation and manuscript evaluation. xv i List of Abbreviations Abbreviation Description 1-NN One nearest neibour AAR Adaptive auto-regressive AB Able bodied AEP Auditory evoked potential AGR Adaptive Gaussian representation ALN Adaptive logic network ANC Activity of neural cells ANN Artificial neural networks AR Auto-regressive ARTMAP Adaptive resonance theory MAP ARX Autoregressive with exogenous input BCI Brain computer interface BPF Band-pass filter CAR Common average referencing CBR Changes in brain rhythms C C T M Cross-Correlation based template matching CER Coarse-grained entropy rate CIIMM Coupled hidden markov model CSP Common Spatial Patterns CSSD Common spatial subspace decomposition CSSP Common spatio-spectral patterns CTFR Correlative time-frequency representation CTFSR Correlative time-frequency-space representation DFT Discrete Fourier transform DSLVQ Distinctive sensitive learning vector quantization E E G Electro-encephalogram E O G Electro-oculogram ENT Energy normalization transform ERD Event related desynchronization ERN Event related negativity ERS Event related synchronization ERW Expected response window FLD Fisher's linear discriminat FFT Fast Fourier transform FP False positive Freq-Norm Frequency normalization GA Genetic algorithm G A M Generalized additive models GLA Generalized linear models GPER Gaussian process entropy rates II M M Hidden markov model IC Intentional control ICA Independent component analysis IFFT Inverse fast Fourier transform K L T Karhounen-Loeve transform k-NN k-nearest neighbour LDA Linear discriminant analysis LDS Linear dynamical system L G M Linear Gaussian models implemented by Kalman filter LMS Least mean square XVll Abbreviation Description LPC Linear predictive coding LPF Low pass filter LRP Lateralized readiness potential L V Q Learning vector quantization MD Mahalanobis distance MLP Multi-Layer perceptron neural networks MN Multiple neuro-mechanisms MNF Maximum noise fraction MR A Movement related activity MRP Movement related potential NC No control NMF Non-negative matrix factorization NN Neural networks OLS1 Orthogonal least square OPM Outlier processing method PCA Principal Component Analysis (a.k.a. Karhounen Loeve Transform) PLV Phase locking values PPM Piecewise Prony method PSD Power spectral density RBF Radial basis function RFE Recursive feature/channel elimination RNN Recurrent neural network ROC Receiver operating characteristic ROCC Receiver operating characteristic curve SA-UK Successive averaging and/or considering choice ofunknown SCI Spinal cord injury SCP Slow cortical potentials SE spectral entropy SFFS Sequential forward feature selection SL Surface Laplacian SOFNN Self organizing feature neural network SOM Self organizing map SSEP Somatosensory evoked potential SSP Signal space projection SSVEP Steady state visual evoked potential STD Standard Deviation SVD Singular value decomposition SVM Support vector machine SVR Support vector machine regression SWDA Stepwise discriminant analysis TBNN Tree-based neural network TEIC Time of expected intentional control T E M Time of expected attempted movement TFR Time-frequency representation TP True positive VEFD Variable epoch frequency decomposition VEP Visual evoked potential WE Wavelet entropy WK Wiener-Khinchine ZDA Z-scale based discriminant analysis xv i i i Chapter 1 Introduction Many different disorders and injuries can disrupt the neuromuscular channels through which the brain communicates and controls the body. These disorders can impair the neural pathways that control muscles or impair the muscles themselves. Those who are severely affected may lose all voluntary muscle control, including eye movement and respiration, and may be completely 'locked i n ' to their brain. In the absence of methods for repairing the damage done by these disorders, various assistive devices have been developed to liberate these individuals, but the effectiveness of devices in assisting severe disabilities is often limited. A n option for restoring the function to those with motor impairment is to provide the brain with a new, non-muscular communication and control channel; a direct brain computer interface (BCI 1 ) for conveying messages and commands to the external world. The ultimate purpose of direct brain computer interface (BCI) research is to allow an individual with severe motor disabilities to have effective control over devices such as computers, speech synthesizers, assistive appliances, and neural prostheses. Such an interface would thus increase an individual's independence, leading to an improved quality of life and reduced social costs. Over the past two decades, a variety o f studies have evaluated the possibility that brain signals recorded from the scalp or from within the brain could provide new augmentative technology that does not require muscle control. Using various signal processing algorithms, brain computer interface (BCI) systems detect the presence of specific patterns in a person's ongoing brain activity that relates to the person's intention to initiate control. The B C I system translates these patterns into meaningful control commands. These control signals are then used as input to intelligent devices to assist these individuals. Fig. 1.1 shows a simple block diagram of a B C I system, in which the B C I transducer translates the person's brain activity into useful control signals that w i l l drive the assistive device. Studies show that non-muscular communication and control are possible (for example see (Mason et al 2007, Vaughan et al 2003, Wolpaw et al 2002)) and might serve useful purposes for those who cannot use conventional technologies. To people who are ' locked-in' 1 Brain computer interfaces may be referred to as brain machine interfaces (BMI), brain interfaces (BI), or direct brain interfaces (DBI) in some studies. 1 (e.g. by end-stage amyotrophic lateral sclerosis ( A L S ) , brainstem stroke, or severe polyneuropathy) or lack any useful muscle control (e.g. due to severe cerebral palsy), a B C I system could give the ability to answer simple questions quickly, control the environment, perform slow word-processing, or even operate a neuro-prosthesis. A t the same time, the performance of this new technology, measured in communication speed and accuracy is modest. Communication speed refers to the speed at which a person can send commands to the system or answer yes/no questions. Thus, they have limited practical value and are only for those with most severe disabilities. A s a result, the ultimate value of this new technology w i l l depend largely on the degree to which its accuracy and communication rate can be increased and the extent to which B C I systems can be applied to the communication and control needs of many people with motor disabilities of different origins and varying severity. 1.1. Functional Model of a BCI system Fig. 1.2 shows detailed functional model of a B C I system (Mason and Birch 2003, Mason et al 2003, Mason et al 2007). The figure depicts a generic B C I system in which a person controls a device in an operating environment (e.g., a powered wheelchair in a house) through a series of functional components. In this context, the user's brain activity is used to generate the control signals that operate the B C I system. The user monitors the state of the device to determine the result of his/her control efforts. In some systems, the user may also be presented with a control display, which displays the control signals generated by the B C I system from his/her brain activity. The electrodes placed on the head of the user record the brain signal from the scalp (or the surface of the brain, or the neural activity within the brain) and convert this brain activity to electrical signals. The 'artifact processor' block shown in Fig. 1.2, removes the artefacts from Figure 1.1. Simple block diagram of a BCI system. 2 the electrical signal after it has been amplified. Note that many transducer designs do not include artifact processing. The 'feature generator' block transforms the resultant signals into feature values that correspond to the underlying neurological mechanism (neuromechanism2) employed by the user for control. For example, i f the user is to control the power of his/her M u (8-12Hz) and Beta (18-25Hz) rhythms, the feature generator would continually generate features relating to the power-spectral estimates of the user's M u and Beta rhythms. The feature generator generally can be a concatenation of three components, the 'signal-enhancement', the 'feature extraction', and the 'feature selection' components, as shown in Fig. 1.2. In some B C I designs, pre-processing (signal enhancement) is performed on the brain signal prior to the extraction of features so as to increase the signal-to-noise ratio of the signal. A feature selection component is sometimes added to the B C I system after the feature extraction stage. The aim of feature selection is to reduce the number of features and/or channels used so that very high dimensional and noisy data are excluded. Ideally, the features that are meaningful or useful in the classification stage are identified and chosen, while others (including outliers and artefacts) are omitted. The 'feature translator' translates the features into logical (device-independent) control signals, such as a two-state discrete output. The translation algorithm uses linear classification methods (e.g., classical statistical analyses) or nonlinear ones (e.g., neural networks). According to the definition in (Mason and Birch 2003), the resultant logical output states are independent of any semantic knowledge about the device or how it is controlled. A s shown in Fig . 1.2, a feature translator may consist of two components: 'feature classification' and 'post-processing'. The main aim of the feature classification component is to classify the features into logical control signals. Post-processing methods such as a moving average block may be used after feature classification to reduce the number of error activations of the system. The control interface translates the logical control signals from the feature translator into semantic control signals that are appropriate for the particular type of device used. Finally, the device controller translates the semantic control signals into physical control signals that 2 According to the Merriam-Webster Medical dictionary, a bodily regulatory mechanism based in the structure and functioning of the nervous system is called a neuromechanism. 3 are used by the device. The device controller also controls the overall behavior of the device. For more detail refer to (Mason and Birch 2003). Table 1.1 provides a simplified description of the B C I transducer components. electrodes Signal Feature Feature Enhancement Extraction Selection Feature Post-Classification processing Figure 1.2 Functional model of a BCI system depicting the principle functional components, where amp = amplifier. Note the 'Control Display' is optional. T A B L E 1.1 DESCRIPTION OF THE BCI T R A N S D U C E R COMPONENTS Item Terms Definition BCI Artifact Processor Transducer Feature Generator Signal Enhancement Feature Extraction Feature Selection Removes artifact from the input signal (1) Enhance signal-to-noise ratio of brain signals (2) The output of this block is a signal with the same nature as the input (i.e. the output like the input is in the temporal domain). Generates the feature vectors Selects a subset of features Feature Translator Feature Classification Classifies the features into logical control signals Post-Processing Increases the performance after feature classification e.g., by blocking the activations with low certainty 4 1.2. Background 3 Present-day BCIs use a variety of electrophysiological mechanisms such as visual evoked potentials (VEP) , slow cortical potentials (SCP), P300 evoked potentials, M u and Beta rhythms associated with movement execution or imagination, movement related potentials (MRP) , cortical neuronal action potentials, and BCIs that are based on the response to mental tasks. In B C I systems, electrophysiological sources refer to the neurological mechanisms or processes employed by a B C I user to generate control signals. In the 1970s, Vida l developed a B C I system based on V E P recorded from the scalp over the visual cortex to determine the direction of gaze, and thus to determine the direction in which the user wished to move a cursor (Vidal 1977). Sutter (Sutter 1992) described a similar B C I system called the brain response interface (BRI). This system uses the V E P s produced by brief visual stimuli and recorded from the scalp over the visual cortex. The user faces a video screen displaying 64 symbols (e.g. letters) in an 8*8 grid and looks at the symbol he or she wants to select. Subgroups of these 64 symbols undergo an equiluminant red/green alterations 40-70 times/s. each symbol is included in several subgroups, and the entire set of subgroups is presented several times. Each subgroup's V E P amplitude about 100 ms after the stimulus is computed and compared to a V E P template already established for the user. From these comparisons the system determines with high accuracy the symbol that the user is looking at (Sutter 1984, Sutter 1992). Middendorf et al. reported another method for using V E P s to determine gaze direction. Several virtual buttons appear on a screen and flash at different rates. The user looks at a button and the system determines the frequency of the response over the visual cortex. When this frequency matches that of a button, the system concludes that the user wants to select it (Middendorf et al 2000). Among the lowest frequency features of the scalp-recorded E E G are the slow voltage changes generated in the cortex. These potential shifts occur over 0.5-10.0 seconds and are called slow cortical potentials (SCP). Negative SCPs are typically associated with movement and other functions involving cortical activation, while positive SCPs are usually associated with reduced cortical activation (Rockstroh et al 1984). Birbaumer and his colleagues have 3 A comprehensive survey of brain computer interface designs has been included in Chapter 2 of this dissertation, therefore, this part only provides a high-level literature review of the field. 5 shown that people can learn to control their SCPs (using SCP-based biofeedback) and thereby control the movement of an object on a computer screen (Birbaumer et al 1999, Birbaumer et al 2000). This is done with the help of visual, auditory or tactile feedbacks (Birbaumer et al 2000). This demonstration is the basis for a B C I referred to as 'thought translation device' (TTD). They used a language support program (Perelmouter and Birbaumer 2000) that enables the user to choose a letter or a letter combination by a series o f two-choice selections. This system has been tested extensively on people with late-stage A L S and has proved to be able to supply basic communication capability (Birbaumer et al 2000). Whi le the idea of the language support program has proved useful, an additional predictive algorithm that uses the first two letters of a word to select the most l ikely word from a lexicon that encompasses the user's vocabulary can markedly increase the communication speed. Infrequent or particularly significant auditory, visual, or somatosensory stimuli, when interspersed with frequent or routine stimuli, typically evoke in the E E G over parietal cortex a positive peak at about 300 ms (after the presentation of the stimuli) named the P300 evoked potential (Donchin and Smith 1970). Donchin and his colleagues have used this 'P300' , or 'oddball ' response to build a brain computer interface (Donchin et al 2000, Farwell and Donchin 1988). In this system, the user faces a 6*6 matrix of letters, numbers, and/or other symbols or commands. Every 125 ms, a single row or column flashes; and, in a complete trial of 12 flashes, each row or column flashes twice. The user makes a selection by counting how many times the row or column containing the desired choice flashes. P300 is prominent only in the responses elicited by the desired choice, and the B C I uses this effect to determine the user's intent. In people with visual impairments, auditory or tactile stimuli might be used (Glover et al 1986, Roder et al 1996). In two recent studies Sellers et al. and Piccione et al. have shown that a P300-based brain computer interface could function as an alternative method of communication for paralyzed patients including spinal cord injured and A L S patients (Piccione et al 2006, Sellers and Donchin 2006). In related work, Bayliss and Ballard (Bayliss and Ballard 2000) recorded P300s in a virtual environment. Offline analyses suggested that single-trial P300 amplitudes could be used for environmental control. Unlike a SCP-based brain computer interface, a P300-based B C I has an apparent advantage in that it requires no initial user training: P300 is a typical, or naive, response to a desired 6 choice. A P300 used in a brain computer interface is also likely to change over time. Studies to date have only reported short-term evaluations of the P300-based B C I systems. Thus, appropriate adaptation by the translation algorithm is likely to be important for this B C I in long-term application, as it is for other types of B C I systems. In awake people, primary sensory or motor cortical areas often display 8-12 H z ( M u rhythm) E E G activity when they are not engaged in processing sensory input or producing motor output (Niedermeyer and Lopes da Silya 1998). Analyses also show that the Mu-rhythm activity comprises a variety of different 8-12 H z rhythms, distinguished from each other by location, frequency, and/or relationship to concurrent sensory input or motor output. These M u rhythms are usually associated with 18-25 H z Beta rhythms. While some Beta rhythms are harmonics of the M u rhythms, some are separable from them by topography and/or timing, and thus are independent E E G features (McFarland et al 2000, Pfurtscheller and Berghold 1989, Pfurtscheller and Lopes da Silva 1999). Several factors suggest that the M u and/or Beta rhythms could be good signal features for EEG-based communication. They are associated with those cortical areas most directly connected to the brain's normal motor output channels. Movement execution, preparation or even imagination (McFarland et al 2000, Pfurtscheller et al 1997) is typically accompanied by a decrease in the M u and Beta rhythms, particularly contralateral to the movement. This decrease has been labeled 'event-related desynchronization' or E R D (Pfurtscheller and Lopes da Silva 1999). Its opposite, rhythm increase, or 'event-related synchronization' (ERS) occurs after movement and with relaxation (Pfurtscheller and Lopes da Silva 1999). Since the mid-1980s, several Mu/Beta rhythm based brain computer interfaces have been developed. With the B C I system in (McFarland et al 1997, Wolpaw et al 1991, Wolpaw et al 2000), people with or without motor disabilities learn to control their M u or Beta rhythm amplitude and use that control to move a cursor in one or two dimensions to targets on a computer screen. They participate in 2-3 40-minute sessions per week, and most (i.e. about 80%) acquire significant control within 2-3 weeks. With this control, users can move the cursor to answer spoken yes/no questions with accuracies of about 95% (Miner et al 1998, Wolpaw et al 1998). Research with this system has focused on defining the topographical, spectral, and temporal features of the M u and Beta rhythm control, optimizing the mutually adaptive interactions between the user and the B C I system, and incorporating other E E G features into 7 this B C I (McFarland et al 1997, Ramoser et al 1997, Schalk et al 2000). Recent studies of this research group on A L S patients show usefulness of this brain computer interface for these patients (Kubler et al 2005). The Graz B C I group has also developed a brain computer interface system based on the E R D and E R S of the M u and Beta rhythms in the E E G over the sensorimotor cortex. Research to date has focused on distinguishing between the E E G associated with imagination of different simple motor actions, such as right or left hand or foot movement, and thereby enabling the user to control a cursor or an orthotic device that opens and closes a paralyzed hand (Neuper et al 1999, Pfurtscheller et al 1993, Pfurtscheller et al 2000). Over a period of 6-7 sessions and with two-choice trials (i.e. left hand vs. right hand imagery), users can reach accuracies over 90%. About 90% of people can use this system successfully. Current studies are seeking modifications that improve classification. Recently, this group has also started developing the so called self-paced brain computer interfaces (Scherer et al 2004, Townsend et al 2004). Unlike synchronized brain computer interfaces, self-paced brain computer interfaces enable users to control the system at any time they wish. Self-paced and synchronized brain computer interfaces wi l l be discussed later in this section. Penny et al. describe a B C I that also uses the E E G over sensorimotor cortex to control cursor movement (Penny et al 2000). They concentrate on detecting the E E G associated with imagery of actions like right or left hand movements, and/or tasks like simple calculations. Their translation algorithm uses Autoregressive parameters and a logistic regression model trained with a Bayesian evidence framework. They report user success in controlling one-dimensional cursor movement (Roberts and Penny 2003). Babiloni et al. (Babiloni et al 2000) are developing a Laplacian E E G analysis and a signal-space projection algorithm to detect imagined movements in the E E G over the sensorimotor cortex. Anderson et al. have also designed a multi-class brain computer interface that detects mental tasks such as solving a multiplication problem, imagining a 3D object, and mental counting using Autoregressive parameters of the E E G (Anderson et al 1998, Garrett et al 2003). Levine et al. (Levine et al 2000) recorded electrocorticogram (EcoG) activity from 17 patients temporarily implanted with 16-126 subdural electrodes prior to epilepsy surgery. They found topographically focused potentials associated with specific movements and 8 vocalizations. Results of their analyses show that these potentials can provide the basis for a B C I with multiple control channels (Graimann et al 2004, Levine et al 2000). Two other studies have also reported the use of EcoG activity in a brain computer interface system for epileptic patients (Lai et al 2005, Leuthardt et al 2004). Wi th these patients, it was possible in just one session to differentiate without any training imagination of hand-, tongue-, and mouth movement. Several laboratories have used multielectrode arrays to record signals from single neurons in the motor cortex of monkeys or rats during learned movements (Chapin et al 1999, Donoghue 2002, Isaacs et al 2000, L i u et al 1999, Nicolelis 2001, Nicolelis 2003). These studies are aimed at reconstructing a movement from multielectrode recorded spike trains or synaptic field potentials. After extensive training and implementation of learning algorithms (for an exception, where animals learned rapidly, see (Serruya et al 2002)), monkeys move cursors on screens toward targets or move an artificial hand in four directions, clearly demonstrating the possibility of translating cellular activity into simple movements online. After extensive training, even complex movement patterns can be reconstructed from an astonishingly small number of cells located in the motor or parietal areas (Nicolelis 2001). Plasticity of cortical circuits allows learned control of movements directly from the cellular activity even outside the primary and secondary homuncular representations of the motor cortex (Schwartz et al 2001, Taylor et al 2002). The first multielectrode array was implanted in a quadriplegic patient's motor hand area in 2004 by Donoghue's group (Hochberg et al 2006). Within a few training sessions, the patient learnt to use his neuronal activity to move a computer cursor in several directions. While training has been limited due to recurring illness and medication effects, the results have been encouraging and suggest that more rapid and accurate control are possible in the future. 1.2.1. Self-Paced and Synchronized BCI Systems A s previously mentioned, many brain computer interface transducer designs have been proposed in the literature (for a review of the field, see (Mason et al 2007, Nicolelis 2003, Vaughan et al 2003, Wolpaw et al 2002). Few of them, however, have been designed specifically for self-paced control. The concept of self-paced control of brain computer interface systems was introduced in (Birch 1988, Mason and Birch 2000) and denoted as 9 "asynchronous control". In a self-paced brain computer interface the users affect the B C I transducer output whenever they want by intentionally changing their brain states (see Fig. 1.3(a)). Between periods of intentional control (IC), users are said to be in a no-control (NC) state - they may be idle, daydreaming, thinking about a problem or lunch, or performing any other action other than trying to control the B C I transducer. These B C I transducers are thus designed to respond only when there is an intentional user control. The appropriate B C I response to no-control (NC) would be a neutral or inactive output. We refer to this ability as NC support. N C support is necessary for most types of machine or device interactions where frequent intentional controls (IC) are spaced by periods of inaction. N C support must handle the diversity of activity and thoughts that make up the N C state and must operate effectively during such periods that range from a few seconds to a few hours depending on the type of application. It is unlikely that one can model all possible NC states related to a target application. In general, this problem can be constrained by including some mechanisms to turn the interface off when it is not in use and turn it on when a control action is desired. While this mechanism is a useful mode for avoiding false responses during long periods of N C states, it is not practical during periods of intended interaction which contain frequent short pauses containing other thoughts, composition, or response monitoring. Therefore, having a mechanism that turns off the system is not sufficient for NC support. The BCI system should still be able to handle NC state while the system in on. In comparison to the self-paced BCIs , in most brain computer interface system evaluations reported in the literature, the allowable time periods for intentional user control are restricted to periods defined by a computer (as shown in Fig. 1.3(b)). In other words, these systems operate only during specific periods determined by the system (not the user). Thus, these systems have not been tested for general intermittent or self-paced operating paradigms. Since the user's input is synchronized with the external computer, this type of control has been termed "synchronized control" (Mason and Birch 2000). In these experimental systems, the B C I technology is tested only during those periods with intentional user control and the response of the BCI transducer during the NC state is not tested. 10 1 1 1 1 I I I I > time Figure 1.3 (a) Self-paced brain computer interface operating paradigm, (b) synchronized brain computer interface paradigm The performance o f a self-paced brain computer interface system that differentiates between N C and IC classes is evaluated by 1) the percentage of correct activations during IC states (true positives, TPs) and 2) the percentage of false output activations during N C states (false positives, FPs). A state is identified as true positive i f the brain computer interface system is activated at least once in a time window around the actual intentional control state (Birch et al 1993, Graimann et al 2004, Scherer et al 2004, Townsend et al 2004, Yom-Tov and Inbar 2003). A n y activation outside this window is considered a false positive. Ideally, one needs to design a system that has 100% true positive rate and 0% false positive rate. Although self-paced control is the most natural mode of interaction, it has received less attention. Only a few B C I systems (Birch et al 1993, Graimann et al 2004, Mason and Birch 2000, Mi l l an et al 2004, Scherer et al 2004, Townsend et al 2004, Yom-Tov and Inbar 2003) have been specifically designed and tested for self-paced control. A s recognized by many researchers in brain computer interface field (e.g. Wolpaw et al 2002), self-paced B C I systems address a problem that is important for practical applications, i.e. detection of user commands without the timing cues provided by structured trials. The self-paced B C I systems introduced in (Birch et al 1993, Graimann et al 2004, Mason and Birch 2000, Townsend et al 2004, Yom-Tov and Inbar 2003) attempt to detect a specific 11 intentional control state, e.g. imagined right hand movement, from the ongoing brain signal. A s the output of these B C I systems can have two possible states, i.e. N C and IC states, these self-paced B C I systems are considered a 2-state self-paced BCIs . Among all the self-paced B C I systems introduced in the literature, only two have attempted to increase the number of output states (Mil lan et al 2004, Scherer et al 2004). The 3-state self-paced B C I implemented in (Scherer et al 2004) attempts to differentiate between right hand, left hand and foot movements in order to operate a virtual keyboard. Although this system is operated in a self-paced paradigm, its response during N C periods has not yet been studied and reported. Therefore, the ability of this system to support NC state is not known. In the experimental paradigm for which this system is tested, the user should continuously engage in operating the virtual keyboard without having the option to stay in a no-control state. In the study of (Mil lan et al 2004) the subjects were asked to perform one of the following three actions: (1) imagine right hand movement, (2) imagine left hand movement, and (3) relax. A 3-state self-paced B C I was designed to navigate a mobile robot in an 80cm*60cm house-like environment by differentiating amongst these three states. The system generates 'unknown state output' when there is not enough confidence in choosing one of the three above mentioned mental tasks. The classifier of this system was not explicitly trained to recognize idle (NC) state (Mil lan et al 2004). According to the authors, it could process them adequately by responding 'unknown'. It was also reported that the task of steering the robot between rooms was so engaging that the two tested subjects preferred to emit continuously mental commands rather than to go through NC (idle) state. Therefore, the response of this system on N C (idle) state was evaluated on a dataset with limited amount of N C (idle) state. Moreover, having the choice of 'unknown state output' may represent some neutral output but it is not clear whether the unknown state output was caused by the actual idle (NC) state or by lack of confidence in detecting one of the three commands. Additionally, there is no evidence that the N C state w i l l fall into the unknown state in these designs. Although, the two 3-state self-paced B C I systems mentioned above (Mil lan et al 2004, Scherer et al 2004) can be used in some practical applications, ideally a self-paced B C I system should have three important capabilities as following: 12 (1) The system should be continuously available for control, (2) The system should not require the user to constantly engage in control, and (3) The system should support N C state as defined above. The 3 r d criterion actually implies the 2 n d criterion, however, to emphasize its importance we have considered them as two separate criteria. 1.2.2. BCI Research at the Neil Squire Society (NSS) and the University of British Columbia (UBC) B C I research at the N S S and U B C began 15 years ago with the development of the Outlier Processing Method (OPM) (Birch 1988, Mason et al 1991). Results from this work on the O P M were promising as hit rates (true positive rates) of greater than 90% were achieved on a thumb movement task. However, its relatively poor performance on spontaneous, idle E E G (yielding false positive rates ranging from 10% to 30%) restricted its use as a self-paced B C I system. During the past years, the L o w Frequency-Asynchronous Switch Design ( L F - A S D ) was introduced as a brain computer interface for self-paced control applications (Mason and Birch, 2000). Figure 1.4 shows the high-level design of the L F - A S D . The original design of this brain computer interface (Mason and Birch 2000) was modified to reduce the processing delays (Lisogurski and Birch 1998) and improve the classification accuracies (Mason et al 2000). The most recent version of this brain computer interface seeks to recognize the movement related potentials (MRPs) in the E E G signal. The inputs to the L F - A S D are six bipolar E E G signals recorded over the supplementary motor area ( S M A ) and sensory-motor cortex (defined with reference to the International 10-20 System at F1 -FC1 , F z - F C z , F2-FC2, F C 1 - C 1 , F C z - C z , and FC2-C2) sampled at 128Hz. The switch (i.e. the B C I system) produces a single output every 1716th o f a second; this output is equivalent to a 2-position switch. Internally, features are extracted from the amplified, bipolar E E G signals and then classified as a class ' 1 ' (IC state) or class ' 0 ' (NC state). The feature extraction algorithm low-pass filters the signal to below 4Hz and then calculates a 6-dimensional feature vector using a feature extraction scheme designed to recognize the temporal patterns in a single-trial bipolar E E G related to voluntary movement. The feature vector is classified as either an IC or an N C state class using a one-nearest neighbor (1-NN) classifier. Reference vectors for 13 both N C and IC classes in the feature classifier are selected from the training data using learning vector quantization ( L V Q ) algorithm (Kohonen 1990). EEG data Feature space 2-state output Figure 1.4 High-level design of LF-ASD brain computer interface. The 'Feature Classifier' continually monitors the 6-dimensional feature vectors and when a feature vector enters a pre-defined sub-space "IC", the output of the classifier is activated (IC state). Otherwise, the output state remains inactive (NC state). In the latest study, before the start of this project on both able-bodied and disabled subjects, the subjects activated the brain computer interface with an imagined finger flexion to control a simple pong style video game at their own pace. Users would themselves report the system classification errors with a pneumatic sip and puff switch whenever an error occurred. During the game, i f ocular artifact (OA) was detected (by using a customized threshold on the E O G signal), the input to the brain switch was ignored during the persistence of O A til l for 0.5 seconds afterward. Evaluation results showed an average hit (true positive (TP)) rate of 51.3% and false positive (FP) rate of 2% (Borisoff et al 2002). Among the proposed self-paced B C I transducers whose offline performances have been reported in terms of true positives and false positives (Graimann et al 2004, Levine et al 2000, Townsend et al 2004, Yom-Tov and Inbar 2003), the L F - A S D has the advantage of being capable o f generating low false positive rates. A l l these offline studies have reported average false positive rates in the range of 6% to 28% while the average true positive rates were between 73% and 94% (Graimann et al 2004, Levine et al 2000, Townsend et al 2004, Yom-Tov and Inbar 2003). It is difficult, however, to directly compare the results of different studies, as the recording equipment, recording and classification protocols, decision rate and time intervals during 14 which false positives could occur and mental tasks considered are different. In addition, the amount of data involved and the degree of training the subjects received before participating in the B C I experiments also vary between studies. Although the evaluation of the L F - A S D had shown the potential capability of the system, the error rates of the system was still not suitable for real-world applications, therefore further improvements in the performance of the system was needed. On the other hand, the L F - A S D provided a 2-state (binary) output, i.e., N C and IC states. This output can be used as a single push button for example to select an object on the monitor or turn something on and off. Although this single function switch is very helpful for people with extreme disabilities, providing more options in controlling the output of the system is still desirable. 1.3. Objectives and Contributions The goals of this thesis are to: (a) Improve the performance of the Ne i l Squire Society's existing 2-state self-paced brain computer interface (BCI) system (the L F - A S D ) , and (b) Design and investigate the feasibility of a 3-state self-paced B C I . To our best knowledge, this is the first 3-state self-paced B C I system 4 that best satisfies the criteria mentioned in Section 1.2.1, i.e., the system is continuously available for control, and the system is specifically designed to support the NC state. This system continuously differentiates the predefined intended control states (i.e. right and the left hand extensions) from the N C state. A s defined in Section 1.2.1, the N C state includes any brain activity other than the IC states. The two 3-state self-paced B C I systems introduced previously by Mi l l an et al and Scherer et al (Mil lan et al 2004, Scherer et al 2004) differ from our proposed system in that they only partially satisfy the three criteria mentioned in Section 1.2.1. A s stated earlier, although the evaluation of the L F - A S D had shown the potential capability of the system, the error rates were still not suitable for real-world applications. For a real-world application, one needs to have a system with a very low false positive rate. A false 4 A 4-state self-paced BCI has been introduced very recently (Scherer et al Aug 2007) just before the deadline for submitting this thesis to the library. This system generates mean true positive rate of 28.4% at the false positive rate of 16.9%. Results of the system introduced in this dissertation are compared with other BCI systems in Chapter 10. 15 positive rate of 2%, in the existing 2-state self-paced B C I , corresponded to two false activations every six seconds, which is not suitable for most practical applications. Therefore further improvements in the performance of the system were needed. Another reason for carrying out research towards improving the existing 2-state self-paced B C I was to build more knowledge to be able to design a 3-state self-paced B C I . In fact, the latest design of the 2-state system which we developed during the course of this project was directly used in the design of the 3-state self-paced B C I system. Figure 1.5 shows samples of the outputs of 2-state and 3-state self-paced BCIs . A 2-state self-paced brain computer interface generates an output which corresponds to two different brain states. In the L F - A S D design, when the user intends control by imagining a finger flexion, the output switches from the N C state to the IC state. In fact, the user activates the output by imagining a finger flexion. On the other hand, a 3-state self-paced B C I would have a 3-state output corresponding to three different brain states. Generally, in such a system, the output of the B C I is in an inactive state, i.e. the N C state. The corresponding intended control states IC1 and IC2, w i l l be activated when specific predefined brain states such as a right finger flexion and a left finger flexion is detected from the recorded brain signals. While a 2-state self-paced B C I system that is based on movement related potentials can be used to determine whether or not a movement, e.g. right index finger flexion, is intended, a 3-state B C I aims at not only detecting the person's intention to move but also distinguishes between two movements, e.g. right/left finger movements. A s a result, the 3-state B C I provides faster communication abilities and more flexibility in operating one or more devices. A s a very simple example, for a wheelchair control application, while a 2-state B C I can provide the command for e.g. turning right, a 3-state B C I would provide both turn right and turn left options. In a more complicated control paradigm, 2-state and 3-state self-paced BCIs can for example be used to navigate through different menu items on a user display and select the desired tasks (refer to Section 10 for a more detailed application example of 2-state and 3-state self-paced BCIs). 16 IC2 IC1 IC N C N C N C 3-state self-paced B C I 2-state self-paced B C I Figure 1.5 Sample outputs of the 2-state BCI (the LF-ASD) and proposed 3-state BCI The main contributions of this thesis, achieved through the course of pursuing the above goals are the following: 1) A comprehensive survey of the brain computer interfaces conducted from the signal processing point of view, 2) Introduction of the design of a 3-state self-paced B C I , 3) Introduction of two new movements (neuromechanisms) to control a 3-state B C I , 4) Improvement in the performance of the 3-state B C I using different feature extraction and classification algorithms. 5) Introduction of alternative adaptive classification schemes for the existing 2-state B C I , 6) Introduction of training-data generation methods for self-paced brain computer interfaces, 7) Improvement in the design of the feature extraction block of the 2-state B C I by customizing the parameter values of this block for each user, 8) Introduction of a feature extraction algorithm that improves the 2-state B C I , 9) Evaluation of the performance of the 2-state B C I system on ocular artifact contaminated E E G data, Although the contributions listed as 5 to 9 are directly attributed to improving the 2-state B C I system, the findings of these studies were directly incorporated in the design of the 3-state B C I system. 17 1.4. Thesis Scope and Organization This dissertation focuses on improving the existing 2-state self-paced brain computer interface and an introducing the design of a 3-state self-paced brain computer interface. Both systems intend to detect movement(s) intentions from the E E G signal. A t first, a comprehensive survey of more than 300 refereed papers in the B C I field was carried out. The focus was on signal processing aspects of the different brain computer interfaces. The aims of this study were to perform a historical analysis of signal processing in this field, to identify the trends in signal processing of BCIs and the methods that have been used for different BCIs and to introduce a possible taxonomy of signal processing in B C I designs. Chapter 2 of this dissertation presents this comprehensive survey. To improve the performance of the 2-state B C I , several studies focusing on improving the different building blocks of the current B C I system (as shown in Fig . 1.2) were conducted as follows: To improve the performance of the existing 2-state B C I design, methods for the classification stage and for the dimensionality reduction of the feature space were implemented. Chapter 3 presents the details and findings of this study (Borisoff et al 2004). Generating training-data for brain computer interfaces that aim at detecting intended tasks such as imagined finger flexions is challenging. In these systems, no external knowledge of the time of the actual performed task is known. Therefore, designing a B C I system without knowing the exact time that the user has performed the intended task is difficult. Chapter 4 presents and compares several methods of generating training-data for self-paced BCIs (Bashashati et al 2007a). These methods are specifically based on probability density estimation and clustering algorithms. A s the E E G characteristics during intended movements may vary from one subject to another, design parameters of a brain computer interface need to be customized for each subject. For the L F - A S D design, without such tuning, the system may detect non-MRP patterns in the ongoing E E G or its performance may not be optimal. Therefore, a method that customizes the parameter values of the feature extraction block of the current B C I system for each subject is proposed in Chapter 5 (Bashashati et al 2006a). The aim of this study is to 18 improve the performance of the B C I design and more importantly to ensure that the output is activated by movement attempts and not by other unwanted brain activity. Previously, these parameter values were determined based on the data of one subject and were then used for all subjects in the study. In another study, we focused on the feature extraction of the current B C I . A novel pattern recognition approach that models the trajectory of feature vectors to detect intentional control commands (attempted finger movements) was proposed and implemented. This novel approach resulted in significant improvements in the 2-state B C I system (Bashashati et al 2006b). The details of this study are presented in Chapter 6 o f this dissertation. In the original design o f the L F - A S D , the output of the system was blocked during the presence of ocular artifacts and the user could not control the output during these times. Analysis of the E E G data recorded in our B C I experiments show that an average of 22% of the data had contained ocular artifacts and the users were not able to control the output during these times. Therefore, a B C I system that does not block out ocular artifacts is desirable. What led us to believe that such a design is possible were the successful results obtained in Chapter 6 where the proposed design uses all past values of the features to determine whether or not a movement is performed. Therefore, in Chapter 7 the performance of the proposed system on ocular artifact contaminated data is evaluated (Bashashati et al 2007b). In addition, an evaluation method that can truly reflect and compare the performance of the B C I system with and without including ocular artifacts is introduced and discussed. Several studies were also conducted to design a 3-state self-paced B C I : A s previously mentioned, there are several neurological mechanisms that can be used to build a B C I system. Movement related potentials (MRP) were used as the mechanism to activate the output of the existing 2-state B C I (the L F - A S D ) . In this specific design, attempted right index finger flexions were used. Evaluation results of the L F - A S D showed the potential ability o f M R P s to activate the B C I . To operate a 3-state self-paced B C I system, two different neurological command sources for activating the output of the system are needed. We decided to use neuromechanisms related to movements as the source of controlling the new 3-state B C I . This is because of our previous experience in detecting movement patterns, the many scientific studies on movement related potentials, and the ease 19 of mapping and using this command source in real world applications. The first and the most important question was 'What are the optimal two movements that can be used to build a B C I ? ' . Based on the results of a survey conducted, the right hand and left hand extensions were likely to generate more differentiable patterns compared to other movements such as finger movements. A study involving human subjects was conducted to collect E E G data while the subjects were performing these two movements. A design of a 3-state B C I system which contains two major blocks is implemented. The first block (DET1) determines whether a movement is present or not. Once a movement is detected, the second block in the system (DET2) determines whether it is a right hand or a left hand extension. Two designs including the latest design of the L F - A S D were implemented and tested for DET1 which detects presence of movements. For D E T 2 , a design that is based on selecting the most informative frequency components of the E E G and a linear classifier was implemented and tested. Chapter 8 of this dissertation presents the details of this study (Bashashati et al 2007c). Chapter 9 explores possible improvements of the 3-state B C I system by employing different designs of DET1 and the use of other feature extraction methods. These designs utilize the history of the feature values and nonlinear classifiers to improve the performance of the 3-state self-paced B C I . Due to the modular design of the 3-state B C I , these algorithms are also directly useable in the designs of 2-state self-paced BCIs which detect the presence of an IC state in the ongoing E E G . The thesis is concluded in Chapter 10 where a summary of the main contributions of this thesis is presented. Potential research topics that can immediately follow this research and an application example of 2-state and 3-state self-paced BCIs are also presented in this chapter. Appendix A includes a copy of the approval from the University of British Columbia's behavioural research ethics board ( B R E B ) to conduct this study. Appendix B presents expanded tables of Chapter 2 of this thesis and Appendix C includes details of the methods that have been used in different parts of this thesis. Figure 1.6 provides a summary of the organization of this dissertation. 20 Chapter 1: Introduction Chapter 2: Comprehensive survey of signal processing of BCIs Improving 2-state self-paced BCI Chapter 3: Classification and dimensionality reduction Chapter 4: Methods for generating training-data for self-paced BCIs Chapter 5: User customization of a self-paced BCI Chapter 6: Improving the 2-state self-paced BCI by using the past history of features Chapter 7: Evaluating the performance of the 2-state self-paced BCI on ocular artefact contaminated data Designing a 3-stalc self-paced BCI Chapter 8: The first design of a 3-state self-paced BCI Chapter 9: Improvements to the original design of the 3-state self-paced BCI Chapter 10: Conclusions and future directions Figure 1.6 Organization of chapters 1.5. References Anderson C W , Stolz E A and Shamsunder S 1998 Multivariate autoregressive models for classification of spontaneous electroencephalographic signals during mental tasks IEEE Transactions on Biomedical Engineering 4 5 277-86 Babiloni F, Cincotti F, Lazzarini L , Mi l l an J, Mourino J, Varsta M , Heikkonen J, Bianchi L and Marciani M G 2000 Linear classification of low-resolution E E G patterns produced by imagined hand movements IEEE Transactions on Rehabilitation Engineering 8 186-8 21 Bashashati A , Fatourechi M , Ward R K and Birch G E 2006a User customization of the feature generator of an asynchronous brain interface Annals of Biomedical Engineering 34 1051-60 Bashashati A , Mason S G , Borisoff J F, Ward R K and Birch G E 2007a A comparative study on generating training-data for self-paced brain interfaces IEEE Transactions on Neural Systems and Rehabilitation Engineering 15 59-66 Bashashati A , Mason S, Ward R K and Birch G E 2006b A n improved asynchronous brain interface: Making use of the temporal history of the L F - A S D feature vectors J Neural Engineering 3 87-94 Bashashati A , Noureddin B , Ward R, Lawrence P and Birch G 2007b Effects of eye blinks on a self-paced brain interface Clinical Neurophysiology 118 1639-47. Bashashati A , Rabab Ward and Gary Birch 2007c Towards development of a 3-state self-paced brain interface based on hand extension movements J Computational Intelligence & Neuroscience In press Bayliss J D and Ballard D H 2000 A virtual reality test bed for brain-computer interface research IEEE Transactions on Rehabilitation Engineering 8 188-90 Birbaumer N , Ghanayim N , Hinterberger T, Iversen I, Kotchoubey B , Kubler A , Perelmouter J, Taub E and Flor H 1999 A spelling device for the paralysed Nature 398 297-8 Birbaumer N , Kubler A , Ghanayim N , Hinterberger T, Perelmouter J, Kaiser J, Iversen I, Kotchoubey B , Neumann N and Flor H 2000 The thought translation device (TTD) for completely paralyzed patients IEEE Transactions on Rehabilitation Engineering 8 190-3 Birch G E 1988 Single trial E E G signal analysis using outlier information PhD dissertation, University of Biritsh Columbia Birch G E , Bozorgzadeh Z and Mason S G 2002 Initial on-line evaluations of the L F - A S D brain-computer interface with able-bodied and spinal-cord subjects using imagined voluntary motor potentials IEEE Transactions on Neural Systems and Rehabilitation Engineering 10 219-24 22 Birch G E , Lawrence P D and Hare R D 1993 Single trial processing of event related potentials using outlier information IEEE Transactions on Biomedical Engineering 40 59-73 Borisoff J F, Mason S G , Bashashati A and Birch G E 2004 Brain-computer interface design for asynchronous control applications: Improvements to the L F - A S D asynchronous brain switch IEEE Transactions on Biomedical Engineering 51 985-92 Chapin J K , Moxon K A , Markowitz R S and Nicolelis M A 1999 Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex Nature neuroscience 2 664-70 Donchin E and Smith D B 1970 The contingent negative variation and the late positive wave of the average evoked potential Electroencephalography and clinical neurophysiology 29 201-3 Donchin E , Spencer K M and Wijesinghe R 2000 The mental prosthesis: Assessing the speed of a P300-based brain-computer interface IEEE Transactions on Rehabilitation Engineering 8 174-9 Donoghue J P 2002 Connecting cortex to machines: Recent advances in brain interfaces Nature neuroscience 5 1085-8 Farwell L A and Donchin E 1988 Talking off the top of your head: Towards a mental prosthesis utilizing event-related brain potentials Electroencephalography and clinical neurophysiology 80 510-23 Garrett D , Peterson D A , Anderson C W and Thaut M H 2003 Comparison of linear, nonlinear, and feature selection methods for E E G signal classification IEEE Transactions on Neural Systems and Rehabilitation Engineering 11 141-4 Glover A A , Onofrj M C, Ghilardi M F and Bodis-Wollner 1 1986 P300-like potentials in the normal monkey using classical conditioning and an auditory 'oddball' paradigm Electroencephalography and clinical neurophysiology 65 231-5 Graimann B , Huggins J E , Levine S P and Pfurtscheller G 2004 Toward a direct brain interface based on human subdural recordings and wavelet-packet analysis IEEE transactions on bio-medical engineering 51 954-62 23 Hochberg L R, Serruya M D , Friehs G M , Mukand J A , Saleh M , Caplan A H , Branner A , Chen D , Perm R D and Donoghue J P 2006 Neuronal ensemble control of prosthetic devices by a human with tetraplegia Nature 442 164-71 Isaacs R E , Weber D J and Schwartz A B 2000 Work toward real-time control of a cortical neural prothesis IEEE Transactions on Rehabilitation Engineering 8 196-8 Kohonen T 1990 Self-organizing map Proceedings of IEEE 78 1464-80 Kubler A , Nijboer F, Mellinger J, Vaughan T M , Pawelzik H , Schalk G , McFarland D J, Birbaumer N and Wolpaw J R 2005 Patients with A L S can use sensorimotor rhythms to operate a brain-computer interface Neurology 64 1115-1 Lai T N , Hinterberger T, Widman G , Schroder M , H i l l N , Rosenstiel W , Elger C E , Scholkopf B and Birbaumer N 2005 Methods towards invasive human brain computer interfaces Advances in Neural Information Processing Systems 17 737-44 Leuthardt E C, Schalk G , Wolpaw J R, Ojemann J G and Moran D W 2004 A brain-computer interface using electrocorticographic signals in humans 1 63-71 Levine S P, Huggins J E , Bement S L , Kushwaha R K , Schuh L A , Rohde M M , Passaro E A , Ross D A , Elisevich K V and Smith B J 2000 A direct brain interface based on event-related potentials IEEE Transactions on Rehabilitation Engineering 8 180-5 Lisogurski D and Birch G E 1998 Identification of finger flexions from continuous E E G as a brain computer interface Proc. IEEE Engineering in Medicine and Biology Society Inter. Conf. 20 2004-7 L i u X , McCreery D B , Carter R R, Bullara L A , Yuen T G and Agnew W F 1999 Stability of the interface between neural tissue and chronically implanted intracortical microelectrodes IEEE Transactions on Rehabilitation Engineering 7 315-26 Mason S G , Bashashati A , Fatourechi M , Navarro K F and Birch G E 2007 A comprehensive survey of brain interface technology designs Annals of Biomedical Engineering 3 5 137-69 Mason S G and Birch G E 2000 A brain-controlled switch for asynchronous control applications IEEE Transactions on Biomedical Engineering 47 1297-307 24 Mason S G and Birch G E 2003 A general framework for brain-computer interface design IEEE Transactions on Neural Systems and Rehabilitation Engineering 11 70-85 Mason S G , Birch G E and Ito M R 1991 A modified algorithm for single-trial signal extraction of low S N R events Proc. IASTED Inter. Conf. on Computers, Electronics, Communication and Control 257-61 Mason S G , Bozorgzadeh Z and Birch G E 2000 The L F - A S D brain-computer interface: On-line identification of imagined finger flexions in subjects with spinal cord injuries Proc. fourth Inter. ACM conf. on Assistive technologies 109-13 Mason S G , Moore M M and Birch G E 2003 Designing pointing devices using brain-computer interface technology Proc. IEEE 1st Inter. Conf. on Neural Engineering (Capri, Italy) 610-3 McFarland D J, Miner L A , Vaughan T M and Wolpaw J R 2000 M u and beta rhythm topographies during motor imagery and actual movements Brain topography 12 177-86 McFarland D , Lefkowicz A and Wolpaw J 1997 Design and operation of an EEG-based brain—computer interface with digital signal processing technology Behavior research methods, instruments & computers 29 337-45 Middendorf M , M c M i l l a n G , Calhoun G and Jones K S 2000 Brain-computer interfaces based on the steady-state visual-evoked response IEEE Transactions on Rehabilitation Engineering 8 211-4 Mi l l an J R, Renkens F, Mourino J and Gerstner W 2004 Noninvasive brain-actuated control of a mobile robot by human E E G IEEE Trans. Biomed. Eng. 51 1026-33. Miner L A , McFarland D and Wolpaw J R 1998 Answering questions with an electroencephalogram-based brain computer interface Arch Phys Med Rehabil 79 1029-33 Neuper C, Schloegl A and Pfurtscheller G 1999 Enhancement of left-right sensorimotor E E G differences during feedback-regulated motor imagery Journal of Clinical Neurophysiology 16 373-82 Nicolelis M A 2001 Actions from thoughts Nature 409 403-7 25 Nicolelis M A 2003 Brain-machine interfaces to restore motor function and probe neural circuits Nat Rev Neurosci 4(5) 417-22 Niedermeyer E and Lopes da Silva F 1998 Electroencephalography: Basic principles, clinical applications, and related fields Fourth Edition, ed C Mitchell . (Baltimore, Maryland U S A : Lippincott Wil l iams & Wilkins) Penny W D, Roberts S J, Curran E A and Stokes M J 2000 EEG-based communication: A pattern recognition approach IEEE Transactions on Rehabilitation Engineering 8 214-5 Perelmouter J and Birbaumer N 2000 A binary spelling interface with random errors IEEE Transactions on Rehabilitation Engineering 8 227-32 Pfurtscheller G and Berghold A 1989 Patterns of cortical activation during planning of voluntary movement Electroencephalography and clinical neurophysiology 72 250-8 Pfurtscheller G , Flotzinger D and Kalcher J 1993 Brain-computer interface - a new communication device for handicapped persons J Microcomputer App. 1 6 293-9 Pfurtscheller G and Lopes da Silva F Ff 1999 Event-related E E G / M E G synchronization and desynchronization: Basic principles Clinical Neurophysiology 1 1 0 1842-57 Pfurtscheller G , Neuper C, Flotzinger D and Pregenzer M 1997 EEG-based discrimination between imagination of right and left hand movement Electroencephalography and Clinical Neurophysiology 1 0 3 642-51 Pfurtscheller G , Neuper C, Guger C, Harkam W , Ramoser H , Schlogl A , Obermaier B and Pregenzer M 2000 Current trends in Graz brain-computer interface (BCI) research IEEE Transactions on Rehabilitation Engineering 8 216-9 Piccione F, Giorgi F, Tonin P, Priftis K , Giove S, Silvoni S, Palmas G and Beverina F 2006 P300-based brain computer interface: Reliability and performance in healthy and paralysed participants Clinical Neurophysiology 111 531-7 Ramoser H , Wolpaw J R and Pfurtscheller G 1997 EEG-based communication: Evaluation of alternative signal prediction methods Biomedizinische Technik 42 226-33 Roberts S J and Penny W 2003 Real-time brain-computer interfacing: A preliminary study using bayesian learning Medical and Biological Engineering and Computing 3 8 56-61 26 Rockstroh B , Birbaumer N , Elbert T and Lutzenberger W 1984 Operant control of E E G and event-related and slow brain potentials Biofeedback and Self-Regulation 9 139-60 Roder B , Rosier F, Hennighausen E and Nacker F 1996 Event-related potentials during auditory and somatosensory discrimination in sighted and blind human subjects Brain Research Cognitive Brain Research 4 77-93 Schalk G , Wolpaw J R, McFarland D J and Pfurtscheller G 2000 EEG-based communication: Presence of an error potential Electroencephalography and Clinical Neurophysiology 111 2138-44 Scherer R, Muller G R, Neuper C, Graimann B and Pfurtscheller G 2004 A n asynchronously controlled EEG-based virtual keyboard: Improvement of the spelling rate IEEE Transactions on Biomedical Engineering 51 979-84 Scherer R, Lee F, Schlogl A , Leeb R, Bischot H and Pfurtscheller G Aug2007 Towards Self-Paced Brain-Computer Communication: Navigation through Virtual Worlds IEEE Transactions on Biomedical Engineering, In press. Schwartz A B , Taylor D M and Tillery S I 2001 Extraction algorithms for cortical control of arm prosthetics Curr Opin Neurobiol 11 701-7 Sellers E W and Donchin E 2006 A P300-based brain-computer interface: Initial tests by A L S patients Clinical Neurophysiology 117 538-48 Serruya M D , Hatsopoulos N G , Paninski L , Fellows M R and Donoghue J P 2002 Instant neural control of a movement signal Nature 416 141-2 Sutter E 1984 The visual evoked response as a communication channel Proceedings of the IEEE Symposium on Biosensors 95-100 Sutter E 1992 The brain response interface: Communication through visually-induced electrical brain responses J Micro CompApp 15 31-45 Taylor D M , Tillery S I and Schwartz A B 2002 Direct cortical control of 3D neuroprosthetic devices Science 296 1829-32 27 Townsend G , Graimann B and Pfurtscheller G 2004 Continuous E E G classification during motor imagery—simulation of an asynchronous B C I IEEE Transactions on Neural Systems and Rehabilitation Engineering 12 258-65 Vaughan T M , Heetderks W J, Trejo L J, Rymer W Z, Weinrich M , Moore M M , Kubler A , Dobkin B H , Birbaumer N , Donchin E , Wolpaw E W and Wolpaw J R 2003 Brain-computer interface technology: A review of the second international meeting IEEE Transactions on Neural Systems and Rehabilitation Engineering 11 94-109 Vida l J 1977 Real-time detection of brain events in E E G Proceedings of the IEEE 65 633-41 Wolpaw J R, Birbaumer N , McFarland D J, Pfurtscheller G and Vaughan T M 2002 Brain-computer interfaces for communication and control Electroencephalography and Clinical Neurophysiology 113 767-91 Wolpaw J R, McFarland D , Neat G W and Forneris C A 1991 A n EEG-based brain-computer interface for cursor control Electroencephalography and Clinical Neurophysiology 78 252-9 Wolpaw J R, McFarland D and Pfurtscheller G 1998 EEG-based communication: Improved accuracy by response verification IEEE Transactions on Rehabilitation Engineering 6 326-33 Wolpaw J R, McFarland D J and Vaughan T M 2000 Brain-computer interface research at the Wadsworth center IEEE Transactions on Rehabilitation Engineering 8 222-6 Yom-Tov E and Inbar G F 2003 Detection of movement-related potentials from the electro-encephalogram for possible use in a brain-computer interface Medical and Biological Engineering and Computing 41 85-93 28 Chapter 2 A Survey of Signal Processing Algorithms in Brain-Computer Interfaces Based on Electrical Brain Signals5 2.1. Introduction The ultimate purpose of a direct brain computer interface (BCI) is to allow an individual with severe motor disabilities to have effective control over devices such as computers, speech synthesizers, assistive appliances, and neural prostheses. Such an interface would increase an individual's independence, leading to an improved quality of life and reduced social costs. A B C I system detects the presence of specific patterns in a person's ongoing brain activity that relates to the person's intention to initiate control. The B C I system translates these patterns into meaningful control commands. To detect these patterns, various signal processing algorithms are employed. Signal processing forms an important part of a B C I design, since it is needed in extracting the meaningful information from the brain signal. This paper summarizes the results of a comprehensive survey o f different signal processing schemes that have been used in B C I systems. It specifically focuses on the following signal processing components of a B C I : the pre-processing stage (which we refer to as the signal-enhancement stage), the feature extraction stage and the feature translation stage. To address all related B C I research, we include all the approaches that use standard scalp-recorded E E G as well as those that use epidural, subdural, or intracortical recordings. The aims of this study are (a) to make it easy to identify the signal processing methods employed in different B C I systems, and consequently to identify the methods that have not yet been explored, (b) to form a historical reference for new researchers in this field, and (c) to introduce a possible taxonomy of signal processing methods in brain computer interfaces. The organization of the paper is as follows: in Section 2.2, the general structure of a B C I system and the current neuromechanisms6 in B C I systems are presented. Section 2.3 details 5 A version of this chapter has been published: Bashashati A., Fatourechi M. , Ward R.K., and Birch G.E. (2007) A Survey of Signal Processing Algorithms in Brain Computer Interfaces Based on Electrical Brain Signals Journal of Neural engineering 4(2), R35-57. This work has also been resulted in two other publications as follows: Mason S.G., Bashashati A., Fatourechi M. , Navarro K.F., and Birch G.E. (2007) A Comprehensive Survey of Brain Interface Technology Designs Annals of Biomedical Engineering 35 137-69. Fatourechi M. , Bashashati A., Ward R.K., and Birch G.E. (2007) EMG and EOG Artifacts in Brain Interface Systems: A Survey Clinical Neurophysiology 118 480-94. 29 the procedure we followed to conduct this study. Results, discussions, and conclusions are in Sections 2.4-6, respectively. 2.2. General Structure of a B C I System Fig. 2.1 shows the functional model of a B C I system (Mason and Birch 2003). The figure depicts a generic B C I system in which a person controls a device in an operating environment (e.g., a powered wheelchair in a house) through a series of functional components. In this context, the user's brain activity is used to generate the control signals that operate the B C I system. The user monitors the state of the device to determine the result of his/her control efforts. In some systems, the user may also be presented with a control display, which displays the control signals generated by the B C I system from his/her brain activity. The electrodes placed on the head of the user record the brain signal from the scalp, or the surface of the brain, or from the neural activity within the brain, and convert this brain activity to electrical signals. The 'artifact processor' block shown in Fig. 2.1, removes the artifacts from the electrical signal after it has been amplified. Note that many transducer designs do not include artifact processing. The 'feature generator' block transforms the resultant signals into feature values that correspond to the underlying neurological mechanism employed by the user for control. For example, i f the user is to control the power of his/her M u (8-12Hz) and Beta (18-25Hz) rhythms, the feature generator would continually generate features relating to the power-spectral estimates of the user's M u and Beta rhythms. The feature generator generally can be a concatenation of three components, the 'signal-enhancement', the 'feature extraction', and the 'feature selection' components, as shown in Fig. 2.1. In some B C I designs, pre-processing is performed on the brain signal prior to the extraction of features so as to increase the signal-to-noise ratio of the signal. In this paper, we use the more general term 'signal- enhancement' to refer to the pre-processing stage. A feature selection component is sometimes added to the B C I system after the feature extraction stage. The aim of feature selection is to reduce the number of features and/or channels used so that 6 According to the Merriam-Webster Medical dictionary, a bodily regulatory mechanism based in the structure and functioning of the nervous system is called a neuromechanism. 30 very high dimensional and noisy data are excluded. Ideally, the features that are meaningful or useful in the classification stage are identified and chosen, while others (including outliers and artifacts) are omitted. The 'feature translator' translates the features into logical (device-independent) control signals, such as a two-state discrete output. The translation algorithm uses linear classification methods (e.g., classical statistical analyses) or nonlinear ones (e.g., neural networks). According to the definition in (Mason and Birch 2003), the resultant logical output states are independent of any semantic knowledge about the device or how it is controlled. A s shown in Fig. 2.1, a feature translator may consist of two components: 'feature classification' and 'post-processing'. The main aim of the feature classification component is to classify the features into logical control signals. Post-processing methods such as a moving average block may be used after feature classification to reduce the number of error activations of the system. The control interface translates the logical control signals (from the feature translator) into semantic control signals that are appropriate for the particular type of device used. Finally, the device controller translates the semantic control signals into physical control signals that are used by the device. The device controller also controls the overall behavior of the device. For more detail refer to (Mason and Birch 2003). Table 2.1 provides a simplified description of the B C I transducer components. electrodes Sijiikil Feature Feature 1 nluneemciii -* Extraction -* Selection Control Display Control Device -Interface •* Controller Feature Classification Post-processing Figure 2.1 Functional model of a BCI system (Mason and Birch 2003). Note the control display is optional. This review focuses on the shaded components of BCI systems. 31 T A B L E 2.1 T A X O N O M Y FOR BCI T R A N S D U C E R DESIGNS Item Terms Definition BCI Artifact Processor Transducer Feature Signal Generator Enhancement Feature Extraction Feature Selection Feature Feature Translator Classification Post-Processing Removes artifact from the input signal (1) Enhances signal-to-noise ratio of the brain signal (2) The output of this block is a signal with the same nature of the input (i.e. the output like the input is in the temporal domain). Generates the feature vectors Selects a subset of features Classifies the features into logical control signals Increases the performance after feature classification e.g., by blocking activations with low certainty 2.2.1. Electrophysiological Sources of Control in Current BCIs In B C I systems, electrophysiological sources refer to the neurological mechanisms or processes employed by a B C I user to generate control signals. Current BCIs fall into seven main categories, based on the neuromechanisms and recording technology they use. In (Wolpaw et al 2002) B C I systems are categorized as five major groups. These categories are sensorimotor activity, P300, V E P , SCP and activity of neural cell ( A N C ) . In this paper, two other categories were added: "Response to Mental Tasks" and "Multiple Neuromechanisms". B C I systems that use non-movement mental tasks to control a B C I (e.g. (Anderson et al 1995b, Mi l l an et al 1998)) assume that different mental tasks (e.g. solving a multiplication problem, imagining a 3D object, or mental counting) lead to distinct, task specific E E G patterns and aim to detect the patterns associated with these mental tasks from the E E G . B C I systems based on multiple neuromechanisms (e.g. (Gysels et al 2005)) use a combination of two or more of the above-mentioned neuromechanisms in a single design of a B C I system. Table 2.2 shows these categories with a short description of each. Note that although the designs that use direct cortical recordings are included as a separate group, direct cortical recording is a recording technology and not a neuromechanism. A s shown in Table 2.2, B C I designs that use sensorimotor activity as the neural source of control can be further divided into three sub-categories: those based on changes in brain rhythms (e.g. the M u and Beta rhythms), those based on movement related potentials (MRP) , and those based on other sensorimotor activity. 32 T A B L E 2.2 ELECTROPHYSIOLOGICAL ACTIVITIES USED IN BCI DESIGNS A N D THEIR DEFINITIONS Neuromechanism Short Description Sensorimotor activity rhythms Beta, Gamma)7 (Mu, and Changes in brain Mu rhythms in the range of 8-12 Hz and Beta rhythms in the range of 13-30 Hz both originate in the sensorimotor cortex and displayed when a person is not engaged in processing sensorimotor inputs or in producing motor outputs (Jasper and Penfield 1949). They are mostly prominent in frontal and parietal locations (Kozelka and Pedley 1990, Kubler et al 2001a, Niedermeyer and Lopes da Silva 1998). A voluntary movement results in a circumscribed desynchronization in the Mu and lower Beta bands (Pfurtscheller and Aranibar 1977). This desynchronization is called Event Related Desynchronization (ERD) and begins in the contralateral rolandic region about two seconds prior to the onset of a movement and becomes bilaterally symmetrical immediately before execution of movement (Pfurtscheller and Lopes da Silva 1999). After a voluntary movement, the power in the brain rhythms increases. This phenomenon, called Event Related Synchronization (ERS), is dominant over the contralateral sensorimotor area and reaches a maximum around 600 ms after movement offset (Pfurtscheller and Lopes da Silva 1999). Gamma rhythm is a high frequency rhythm in the EEG. Upon the occurrence of a movement, the amplitude of gamma rhythm increases. Gamma rhythms are usually more prominent in the primary sensory area. MRPs are low-frequency potentials that start about 1-1.5 seconds before a movement. They have bilateral distribution and present maximum amplitude at the vertex. Close to the movement, they become contralaterally preponderant (Babiloni et al 2004, Deecke and Kornhuber 1976, Hallett 1994). The sensorimotor activities that do not belong to any of the preceding categories are categorized as other sensorimotor activities. These activities are usually not restricted to a particular frequency band or scalp location and usually cover different frequency ranges. An example would be features extracted from an EEG signal fdtered to frequencies below 30 Hz. Such a range covers different event-related potentials (ERPs) but no specific neuromechanism is used. Movement-related potentials (MRPs) Other sensorimotor activities 7 In (Ramachandran and Histein 1998), references regarding the similarity between attempted movements and real movements in the ERD of Mu patterns are provided. An attempted movement occurs when a subject attempts to move some part of his/her body, but because of either a disability or the experiment control, the actual movement does not happen. Hence, all of the studies employing a specific brain rhythm as a source of control are associated with this category. Similarly, (Gevins et al 1989) has shown that imaginary (attempted) movements generate movement-related potentials (MRPs) similar to those generated by actual movements. Thus, neuromechanisms corresponding to attempted movements are grouped in the same category of real movement. 33 Neuromechanism Short Description Slow cortical potentials (SCPs) SCPs are slow, non-movement potential changes generated by the subject. They reflect changes in cortical polarization of the EEG lasting from 300 ms up to several seconds. Functionally, a SCP reflects a threshold regularization mechanism for local excitory mobilization (Neumann et al 2003, Wolpaw et al 2002). P300 Infrequent or particularly significant auditory, visual, or somatosensory stimuli, when interspersed with frequent or routine stimuli, typically evoke in the EEG over the parietal cortex a positive peak at about 300 ms after the stimulus is received. This peak is called P300 (Allison and Pineda 2003, Kubler et al 2001a). Visual evoked potentials (VEP) VEPs are small changes in the ongoing brain signal. They are generated in response to a visual stimulus such as flashing lights and their properties depend on the type of the visual stimulus (Kubler et al 2001a). These potentials are more prominent in the occipital area. If a visual stimulus is presented repetitively at a rate of 5-6 Hz or greater, a continuous oscillatory electrical response is elicited in the visual pathways. Such a response is termed steady-state visual evoked potentials (SSVEP). The distinction between VEP and SSVEP depends on the repetition rate of the stimulation (Gao et al 2003b). Response to mental tasks BCI systems based on non-movement mental tasks assume that different mental tasks (e.g., solving a multiplication problem, imagining a 3D object, and mental counting) lead to distinct, task-specific distributions of EEG frequency patterns over the scalp (Kubler et al 2001a). Activity of neural cells (ANC) It has been shown that the firing rates of neurons in the motor cortex are increased when movements are executed in the preferred direction of neurons. Once the movements are away from the preferred direction of neurons, the firing rate is decreased (Donoghue 2002, Olson et al 2005). Multiple neuromechanisms (MNs) BCI systems based on multiple neuromechanisms use a combination of two or more of the above-mentioned neuromechanisms. 2.3. Methods The B C I designs selected for this review include every journal and conference paper that met the following criteria: (1) One or more of the keywords B C I , B M I , D B I appeared in its title, abstract or keyword list; (2) The work described one or more B C I designs (the minimum design content that met the criteria was a BCI Transducer as described in Section 2.2). There were a few papers that only reported pre-processing techniques specifically designed for brain computer interfaces that use neural cortical recordings. These papers were reported in the pre-processing techniques. Papers that presented tutorials, descriptions of electrode technology, neuroanatomy, and neurophysiology discussions that might serve as the basis for a B C I were not included; 34 (3) Only papers published in English and in refereed international journals and conference proceedings were included; (4) Designs that use functional magnetic resonance imaging (fMRI) (Weiskopf et al 2003, Weiskopf et al 2004, Y o o et al 2004), magneto-encephalography ( M E G ) signals (Georgopoulos et al 2005), near infra red spectrum (NIRS), auditory evoked potentials (Hi l l et al 2004, Su Ryu et al 1999) and somatosensory evoked potentials (Yan Wang et al 2004) were not included in this paper; and (5) Papers were published prior to January 2006. Although no paper meeting the five criteria explained above was omitted from the analysis, some papers may have been missed unintentionally. The current work should thus be regarded as an initial step to build a public database that can be updated and evolved with time. In Tables 2.3-8, we categorized the papers according to the signal processing methods used. For each of the design blocks of a B C I system shown in Fig. 2.1 (signal-enhancement, feature extraction, feature selection, feature classification, and post-processing), we created a table that reports the signal processing techniques used in that block. Since most of the B C I designs that use neural cortical recordings do not contain a feature extraction component, we generated a separate table (Table 2.7) to report the translation schemes used in these designs. Feature extraction methods used in B C I systems are closely related to the specific neuromechanism(s) used by a B C I . For example, the feature extraction algorithms employed in VEP-based BCIs are used to detect the visual evoked potentials in the ongoing E E G . In B C I systems that operate on slow cortical potentials (SCP), the extracted features are mostly used for the purpose of identifying this specific phenomenon in the brain signal. Thus in Table 2.5, we categorize the feature extraction algorithms based on the seven neurological sources described in Table 2.2. For example, the methods used in VEP-based BCIs are assembled under a different category from those used in SCP-based BCIs . A more detailed version of Table 2.5 can be found in Table B . 1 of Appendix B . The different feature classification algorithms used in B C I systems are shown in Table 2.6. A s feature classification algorithms are also closely related to the type of the features that they classify, the feature classification algorithms are also categorized based on the feature 35 extraction methods. This can be found in Table B.2 of Appendix B . Appendix B also contains a detailed version of Table 2.7 where the classification algorithms for BCIs that use cortical neural recordings are shown. Categorizing the feature classification methods based on the feature extraction methods used does not necessarily limit the use of a specific feature classification to a specific feature extraction method. The same applies to the categorization of the feature extraction methods based on neuromechanisms used in B C I systems. The aim here is to provide as specific information as possible about signal processing in current B C I designs and the researchers can combine any feature extraction method and/or feature classification method from different categories i f necessary. Each table includes major classes corresponding to each design block. These classes were initially determined by our team and then refined after an initial pass through the selected papers. In some cases, each major class was further divided into more specific categories. The full classification template with all the major classes and sub-classes of each design component is listed in the left column of Tables 2.3, 2.4, 2.5 and 2.8, and in the left two major columns of Tables 2.6 and 2.7. Note that in this paper, major classes are written in bold type and sub-classes are represented in bold-italic type. For example, in Table 2.5 or Table B . l , SCP-based B C I designs that use some type of power spectral parameters of the E E G are categorized under the SCP-spectral parameters class, while a B C I design that is based on the movement-related potentials (MRP) and that uses the same method is categorized under the Sensorimtor Activi ty -PSD and Sensorimtor Act ivi ty-MftP-PSD classes in Table 2.5 and Appendix B , respectively. A s an example from Table 2.6, B C I designs that use linear discriminant analysis ( L D A ) classifiers are categorized under L D A . Table B.2 which has a more detailed version of Table 2.6, categorizes B C I designs that use P S D features and L D A classifier under P S D - L D A class. The category for each B C I design was determined by selecting the closest sub-class in the classification template. For the papers that reported multiple designs multiple classifications were recorded. The designs were categorized based only on what was reported in each paper. N o personal knowledge of an authors' related work was used in the classification. 36 In some cases, it was difficult to differentiate between the signal-enhancement, feature selection, and feature extraction design components of a brain computer interface. Based on the definitions in Table 2.1, the methods that satisfied the following four criteria were considered to be signal-enhancement methods: (1) The method was implemented to improve the signal-to-noise ratio of the brain signal. (2) The output of the block had the same nature as the input brain signal (i.e. the output stayed in the temporal domain). (3) The algorithm was directly performed on the brain signal and not on the features extracted from the brain signal. (4) The method did not handle artifacts. The common spatial patterns (CSP) method is an example of a method that satisfies the four above mentioned criteria and was thus categorized as a signal-enhancement method. Only designs that incorporated signal-enhancement algorithms other than the general band-pass filtering of the E E G , the power-line-effect rejection, and the traditional normalization of the signal were reported in the signal-enhancement section of this paper. 2.4. Results The detailed classification results of the survey are summarized in Tables 2.3-8 (refer to tables B . l - 3 in Appendix B for more detailed versions of tables 2.5-7). A s mentioned in Section 2.3, these six tables address the signal-enhancement, feature selection, feature extraction, feature classification and post-processing methods used. The references listed for each sub-class entry represent all the papers that reported on designs related to that sub-class. A s such, one can find all the designs that have specific attributes of interest. For example, i f one is interested in all B C I technology designs that have used parametric modeling (and specifically extracted A R parameters of the signal) to detect the sensorimotor activity, then all the references to the relevant papers can be found in Table 2.5 under Sensorimotor Activity-Parametric modeling (AR, AAR and ARX parameters). Alternatively, i f one is looking for designs that do not have a feature extraction block but directly apply the support-vector-machine ( S V M ) classification method on the brain signal, then these papers can be easily located in the N o n e - S V M class in Appendix C. Similarly, the designs that use the 37 S V M classification method, regardless of the feature extraction technique used, are categorized under the S V M class in Table 2.6. To enhance the clarity o f Tables 2.3-8, the following notations are used: (a) B C I designs based on multiple neuromechanisms (as defined in Table 2.2) are presented in separate categories such as M N : Sensorimotor Activi ty + Response to Mental tasks, which show that the design is based on the sensorimotor activity and the response to mental tasks neuromechanisms. (b) Two or more methods that are consecutively used in a design block are separated by A s an example, CSP-log transformation denotes a design that first applies common spatial patterns (CSP) on the signal and then applies a logarithmic function on the resulting time-series. (c) Two methods that are applied simultaneously in a design component are separated by "+". For example, AR parameters + PSD parameters corresponds to a design that uses both autoregressive (AR) and power-spectral-density (PSD) features in the feature-extraction block. (d) LRP + (CSP - log-transformation} denotes designs that use the two kinds of feature-extraction methods separated by "+". The first method is based on the extraction of lateralized readiness potentials (LRP) , and the second feature extraction method is based on consecutively applying C S P followed by a logarithm function on the signals that are grouped i n " { } " . (e) To facilitate readability, we have provided an index of terms in Appendix A . 38 Table 2.3 Pre-processing (signal enhancement) methods in BCI designs Pre-Processing method Reference ID Common average referencing (CAR) Surface Laplacian (SL) Independent component analysis (ICA) Common spatial patterns (CSP) Principal component analysis (PCA) Combined CSP and PCA Singular value decomposition (SVD) Common spatio-spatial patterns (CSSP) Frequency normalization (Freq-Norm) Local averaging technique (LAT) Robust Kalman filtering Common spatial subspace decomposition (CSSD) Wiener filtering Sparse component analysis Maximum noise fraction (MNF) Spike detection methods Neuron ranking methods (Cheng et al 2004, Fabiani et al 2004, Kubler et al 2005, Li et al 2004b, McFarland and Wolpaw 1998, McFarland et al 1997, McFarland et al 2003, Peters et al 2001, Ramoser et al 2000, Schalk et al 2000, Wolpaw et al 1997) (Babiloni et al 2000, Babiloni et al 2001b, Cincotti et al 2001, Cincotti et al 2003a, Dornhege et al 2004, Fabiani et al 2004, Gysels and Celka 2004, McFarland and Wolpaw 1998, McFarland et al 1997, McFarland et al 2003, McFarland et al 2005, Millan et al 2004a, Millan et al 2002a, Millan et al 2002b, Millan et al 1998, Millan et al 2000a, Millan 2004, Millan and Mourino 2003b, Millan et al 2004b, Muller et al 2003b, Peters et al 2001, Qin et al 2004b, Qin et al 2004a, Qin and He 2005, Qin et al 2005, Ramoser et al 2000, Schalk et al 2000, Wang et al 2004b, Wang et al 2004a, Wolpaw and McFarland 2004, Millan et al 2003a) (Bayliss and Ballard 1999, Bayliss and Ballard 2000a, Bayliss and Ballard 2000b, Erfanian and Erfani 2004, Gao Xiaorong et al 2004, Peterson et al 2005, Wu et al 2004, Serby et al 2005, Xu et al 2004a, Wang et al 2004c, Li et al 2004a) (Blanchard and Blankertz 2004, Dornhege et al 2003, Dornhege et al 2004, Guger et al 2000b, Krauledat et al 2004, Krauledat et al 2004, Muller et al 2003b, Pfurtscheller et al 2000, Pfurtscheller and Neuper 2001, Pfurtscheller and Neuper 2001, Ramoser et al 2000, Townsend et al 2004, Xu et al 2004b) (Chapin et al 1999, Guan et al 2004, Hu et al 2004, Isaacs et al 2000, Lee and Choi 2002, Lee and Choi 2003, Thulasidas et al 2004, Xu et al 2004a, Yoon et al 2005, Li et al 2004a) (Xu et al 2004b) (Trejo et al 2003) (Lemm et al 2005) (Bashashati et al 2005, Borisoff et al 2004, Fatourechi et al 2004, Fatourechi et al 2005, Yu et al 2002) (Peters etal2001) (Bayliss and Ballard 1999, Bayliss and Ballard 2000a) (Cheng et al 2004, Li et al 2004b, Liu et al 2003, Wang et al 2004d) (Vidal 1977) (Li et al 2004a) (Peterson et al 2005) (Obeidand Wolf 2004) (Sanchez et al 2004) 39 Table 2.4 Feature selection/dimensionality reduction methods in BCI designs Feature selection/dimensionality reduction method Reference ID Genetic algorithm (GA) Principal component analysis (PCA) Distinctive sensitive learning vector quantization (DSLVQ) Sequential forward feature selection (SFFS) Grid search method Relief method (Kira and Rendell 1992) Recursive feature/channel elimination (RFE) Support vector machine (SVM)-based recursive feature elimination Stepwise discriminant procedure Linear discriminant analysis (LDA) Fisher discriminant analysis (dimensionality reduction) Fisher discriminant-based criterion (feature selection) Zero-Norm Optimization (/0-opt) Orthogonal Least Square (OLS1) based on radial basis function (RBF) (Flotzinger et al 1994, Garrett et al 2003, Graimann et al 2003a, Graimann et al 2004, Peterson et al 2005, Scherer et al 2004, Schroder et al 2003, Tavakolian et al 2004, Yom-Tov and Inbar 2001, Yom-Tov and Inbar 2002) (Anderson et al 1995a, Bashashati et al 2005, Borisoff et al 2004, Fatourechi et al 2004, Fatourechi et al 2005) (Flotzinger et al 1994, Neuper et al 2005, Pfurtscheller et al 1996, Pfurtscheller et al 1997, Pfurtscheller et al 1998, Pfurtscheller et al 2000, Pfurtscheller and Neuper 2001, Pregenzer and Pfurtscheller 1995, Pregenzer and Pfurtscheller 1999) (Fabiani et al 2004, Keira and Aunon 1990) (Glassman 2005) (Millan et al 2002a) (Lai et al 2004, Schroder et al 2005) (Gysels et al 2005) (Vidal 1977) (Graimann et al 2003b) (Wang et al 2004d) (Cheng et al 2004, Lai et al 2004) (Lai et al 2004) (Xu et al 2004b) 40 T A B L E 2.5 F E A T U R E E X T R A C T I O N METHODS IN BCI DESIGNS. REFER TO APPENDIX 2 FOR A MORE DETAILED VERSION OF THIS T A B L E Neurological phenomenon Feature method Extraction Reference ID Spectral parameters Sensorimotor Activity Parametric modeling (AR, AAR & ARX parameters) TFR method C C T M Signal envelope - Cross-correlation Hjorth parameters Signal complexity Combination of different feature extraction methods LRP features (Babiloni et al 2000, Babiloni et al 2001a, Babiloni et al 2001b, Blanchard and Blankertz 2004, Boostani and Moradi 2004, Cho et al 2004, Cincotti et al 2001, Cincotti et al 2003a, Cincotti et al 2003b, Coyle et al 2005, Fabiani et al 2004, Flotzinger et al 1994, Garcia et al 2003b, Garrett et al 2003, Guger et al 2000b, Guger et al 2003a, Ivanova et al 1995, Kalcher et al 1992, Kalcher et al 1993, Kelly et al 2002b, Krauledat et al 2004, Krausz et al 2003, Kubler et al 2005, Lai et al 2004, Leeb and Pfurtscheller 2004, Lemm et al 2005, Mahmoudi and Erfanian 2002, Mason and Birch 2000, McFarland and Wolpaw 1998, McFarland et al 1997, McFarland et al 2003, McFarland et al 2005, Millan et al 2002a, Millan et al 2002b, Muller et al 2003c, Muller-Putz et al 2005b, Neuper et al 2003, Neuper et al 2005, Pfurtscheller et al 2005, Pfurtscheller et al 1993, Pfurtscheller et al 1994, Pfurtscheller et al 1996, Pfurtscheller et al 1997, Pfurtscheller et al 1998, Pfurtscheller et al 2000, Pfurtscheller and Neuper 2001, Pfurtscheller and Neuper 2001, Pfurtscheller and Neuper 2001, Pfurtscheller and Neuper 2001, Pfurtscheller et al 2003a, Pfurtscheller et al 2003b, Pineda et al 2003, Pregenzer and Pfurtscheller 1995, Pregenzer and Pfurtscheller 1999, Ramoser et al 2000, Schalk et al 2000, Scherer et al 2004, Sheikh et al 2003, Townsend et al 2004, Trejo et al 2003, Jia et al 2004, Wolpaw et al 1991, Wolpaw and McFarland 1994, Wolpaw et al 1997, Wolpaw et al 2000, Wolpaw et al 2003, Wolpaw and McFarland 2004, L i et al 2004a) (Burke et al 2002, Burke et al 2005, Graimann etal 2003b, Guger et al 1999, Guger et al 2000a, Guger et al 2003a, Guger et al 2003b, Haselsteiner and Pfurtscheller 2000, Huggins et al 2003, Kelly et al 2002b, Kelly et al 2002a, Lai et al 2004, Neuper et al 1999, Obermaier et al 2001b, Obermaier et al 2001a, Peters et al 2001, Pfurtscheller et al 1998, Pfurtscheller and Guger 1999, Pfurtscheller et al 2000, Pfurtscheller and Neuper 2001, Schloegl et al 1997a, Schloegl et al 1997b, Schlogl et al 2003, Schroder et al 2005, Sykacek et al 2003, Yoon et al 2005) (Bashashati et al 2005, Birch et al 2002, Birch et al 2003, Borisoff et al 2004, Bozorgzadeh et al 2000, Costa and Cabral 2000, Fatourechi et al 2004, Fatourechi et al 2005, Garcia et al 2003a, Garcia et al 2003b, Glassman 2005, Graimann et al 2003a, Graimann et al 2004, Huggins et al 2003, Lemm et al 2004, Lisogurski and Birch 1998, Mason and Birch 2000, Mason et al 2004, Pineda et al 2000, Qin et al 2004b, Qin and He 2005, Qin et al 2005, Yom-Tov and Inbar 2003) (Balbale et al 1999, Graimann et al 2003b, Graimann et al 2004, Huggins et al 1999, Huggins et al 2003, Levine et al 1999, Levine et al 2000) (Wang et al 2004b, Wang et al 2004a) (Boostani and Moradi 2004, Lee and Choi 2002, Obermaier et al 2001a, Obermaier et al 2001c, Pfurtscheller and Neuper2001) (Boostani and Moradi 2004, Roberts et al 1999, Trejo et al 2003) (Cheng et al 2004, Dornhege et al 2003, Dornhege et al 2004, Krauledat et al 2004, Mahmoudi and Erfanian 2002, Muller et al 2003b, Yom-Tov and Inbar 2001, Yom-Tov and Inbar 2002) (Blankertz et al 2002a, Blankertz et al 2003, Krauledat et al 2004) Neurological Feature Extraction phenomenon method Other None SCP Calculation of SCP amplitude TFR method Mixed filter None P300 Cross-correlation Stepwise discriminant analysis Matched filtering PPM TFR method Peak picking Area calculation Area and peak picking Not mentioned (calculated P300 but details not mentioned) None VEP Spectral parameters Lock-in amplifier Asymmetry ratio of different band powers Cross-correlation Amplitude between N2 and P2 peaks Reference ID (Coyle et al 2004, Huggins et al 2003, Hung et al 2005, LaCourse and Wilson 2003, L i et al 2004b, Liu et al 2003, Mason and Birch 2000, Pineda et al 2000, Qin et al 2004a, Qin et al 2005, Wang et al 2004d, Xu et al 2004b, Yom-Tov and Inbar 2003) (Barreto et al 1996a, Barreto et al 1996b, Blankertz et al 2002a, Lee and Choi 2002, Lee and Choi 2003, Mahmoudi and Erfanian 2002, Parra et al 2002, Parra et al 2003a, Schroder et al 2003, Trejo et al 2003) (Birbaumer et al 1999, Birbaumer et al 2000, Hinterberger et al 2003, Hinterberger et al 2004a, Hinterberger et al 2004b, Hinterberger et al 2005b, Hinterberger et al 2005a, Kaiser et al 2001, Kaiser et al 2002, Kubler et al 1999, Kubler et al 2001b, Kubler et al 1998, Neumann et al 2003, Neumann et al 2004) (Bostanov 2004, Hinterberger et al 2003) (Hinterberger et al 2003) (Hinterberger et al 2003, Schroder et al 2003) (Bayliss and Ballard 1999, Bayliss and Ballard 2000a, Bayliss and Ballard 2000b, Farwell and Donchin 1988) (Donchin et al 2000, Farwell and Donchin 1988) (Serby et al 2005) (Jansen et al 2004) (Bostanov 2004, Donchin et al 2000, Fukada S et al 1998, Glassman 2005, Jansen et al 2004, Kawakami et al 1996) (Allison and Pineda 2003, Allison and Pineda 2005, Bayliss et al 2004, Farwell and Donchin 1988) (Farwell and Donchin 1988) (Kaper and Ritter 2004b, Xu et al 2004a) (Bayliss 2003, Polikoff et al 1995) (Guan et al 2004, Jansen et al 2004, Kaper and Ritter 2004a, Kaper and Ritter 2004b, Kaper et al 2004, Thulasidas et al 2004) (Cheng Ming et al 2005, Cheng and Gao 1999, Cheng et al 2001, Cheng et al 2002, Gao et al 2003b, Kelly et al 2004, Kelly et al 2005c, Kelly et al 2005a, Kelly et al 2005b, Lalor et al 2005, Middendorf et al 2000, Muller-Putz et al 2005a, Wang et al 2005a, Wang et al 2004c) (Calhoun and McMillan 1996, McMillan and Calhoun 1995, Muller-Putz et al 2005a) (SuRyuefa/ 1999) (Sutter 1992) (Lee et al 2005) Neurological phenomenon Feature method Extraction Reference DD Response Mental tasks8 to ANC MN: Sensorimotor Activity + Response to Mental Tasks None Spectral parameters Parametric modeling (AR & AAR parameters) Signal Complexity Eigen values of correlation matrix LPC using Burg's method None Cross-covariance - PCA L B G vector quantization (VQ) Filtering - Rectification -Thresholding Averaging TFR methods None (Most of these designs model the relationship between neural firing rates and 'position and/or velocity and/or acceleration' of hand) Spectral parameters Parametric modeling TFR method (Guan et al 2005, Vidal 1977) (Bashashati et al 2003, Keirn and Aunon 1990, Rostov and Polak 1997, Liu et al 2005, Millan et al 1998, Palaniappan et al 2002, Palaniappan 2005, Peterson et al 2005, Polak and Rostov 1997, Polak and Rostov 1998, Wang et al 2005a) (Anderson et al 1995b, Anderson et al 1998, Garrett et al 2003, Huan and Palaniappan 2004, Keirn and Aunon 1990, Rostov and Polak 2000, Huan and Palaniappan 2005, Polak and Rostov 1998, Polak and Rostov 1999, Sykacek et al 2003) (Bashashati et al 2003, Tavakolian et al 2004) (Anderson etal 1998) (Rostov and Polak 1997) (Anderson et al 1995a, Panuccio et al 2002) (Isaacs et al 2000) (Darmanjian et al 2003) (Karniel et al 2002, Rositsky et al 2003, Reger et al 2000a, Reger et al 2000b) (Laubach et al 2000, Otto et al 2003, Vetter et al 2003) (Laubach et al 2000, Musallam et al 2004) (Black et al 2003, Byron et al 2005, Carmena et al 2003, Carmena et al 2005, Chapin et al 1999, Gao et al 2002, Gao et al 2003a, Hatsopoulos et al 2004, Hu et al 2004, Rarniel et al 2002, Remere et al 2004, Rennedy et al 2000, Rim et al 2005a, Rim et al 2005b, Lebedev et al 2005, Olson et al 2005, Patil et al 2004, Rao et al 2005, Roushe et al 2003, Sanchez et al 2002a, Sanchez et al 2002b, Sanchez et al 2003, Serruya et al 2003, Serruya et al 2002, Taylor et al 2002, Taylor et al 2003, Wessberg et al 2000, Wu et al 2002a, Wu et al 2002b) (Gysels and Celka 2004, Gysels et al 2005, Millan et al 2004a, Millan et al 2002b, Millan et al 2000a, Millan et al 2000b, Millan 2004, Millan and Mourino 2003b, Millan et al 2004b, Obermaier et al 200Id, Varsta et al 2000, Millan et al 2003a) (Curran et al 2004, Penny et al 2000, Roberts and Penny 2003, Sykacek et al 2004, Varsta et al 2000) (Garcia and Ebrahimi 2002, Garcia et al 2002, Garcia et al 2003c, Molina et al 2003, Varsta et al 2000) Designs that differentiate between relaxed state and movement tasks are considered in "Sensorimotor activity + Response to Mental Tasks" category. Neurological Feature phenomenon method Extraction Reference DD MN: SCP + other brain rhythms Combination of different features PLV Mean spectral coherence None SCP calculation + Power spectral parameters (Erfanian and Erfani 2004) (Gysels and Celka 2004, Gysels et al 2005) (Gysels and Celka 2004) (Mourino et al 2002, Rezek et al 2003) (Hinterberger and Baier 2005Mensh et al 2004) T A B L E 2.6 F E A T U R E CLASSIFICATION METHODS IN BCI DESIGNS THAT USE E E G A N D ECOG RECORDING T E C H N O L O G Y Feature translation method Reference ID Neural Networks (NN) MLP Committee ofMLPNN FIR-MLP NN Committee of Plat's RAN algorithm (Piatt 1991) Committee of NNs trained with Adaboost Committee of single Perceptrons with no hidden layers TBNN TDNN LVQ kMeans-LVQ fART-LVQ DSLVQ Growing Hierachical SOM ALN ANN Custom designed local NN Fuzzy AR TMAP Single layer NN RBF-NN Static neural classifier (Adaline) Gamma NN (Anderson et al 1995b, Anderson et al 1995a, Anderson et al 1998, Costa and Cabral 2000, Erfanian and Erfani 2004, Fukada S et al 1998, Garrett et al 2003, Haselsteiner and Pfurtscheller 2000, Huan and Palaniappan 2004, Hung et al 2005, Ivanova et al 1995, Mahmoudi and Erfanian 2002, Huan and Palaniappan 2005, Palaniappan 2005, Su Ryu et al 1999, Tavakolian et al 2004) (Millan et al 2002b, Millan et al 2000b, Varsta et al 2000) (Haselsteiner and Pfurtscheller 2000) (Millan et al 1998) (Boostani and Moradi 2004) (Peters e< a/ 2001) (Ivanova al 1995) (Barreto et al 1996a, Barreto et al 1996b) (Flotzinger et al 1994, Ivanova et al 1995, Kalcher et al 1992, Kalcher et al 1993, Pfurtscheller et al 1993, Pfurtscheller et al 1994, Pfurtscheller et al 1996, Pfurtscheller et al 1997, Pfurtscheller et al 1998, Pfurtscheller et al 2000, Pfurtscheller and Neuper 2001, Pregenzer and Pfurtscheller 1999) (Bashashati et al 2005, Birch et al 2002, Birch et al 2003, Borisoff et al 2004, Bozorgzadeh et al 2000, Fatourechi et al 2004, Fatourechi et al 2005, Lisogurski and Birch 1998, Mason and.Birch 2000, Mason et al 2004, Yom-Tov and Inbar 2003) (Borisoffera/2004) (Muller-Putz et al 2005a, Neuper et al 2005, Pregenzer and Pfurtscheller 1995, Pregenzer and Pfurtscheller 1999) (Liu et al 2005) (Rostov and Polak 1997, Polak and Kostov 1998, Polak and Rostov 1999, Kostov and Polak 2000) (Cincotti et al 2003b) (Mourino et al 2002, Millan et al 2002b, Millan et al 2000a, Millan and Mourino 2003b) (Palaniappan et al 2002) (Garcia et al 2002) (Hung et al 2005) (Barreto etal 1996a, Barreto et al 1996b) (Barreto et al 1996a, Barreto et al 1996b) Feature translation method Reference ID (R)LDA9 (Boostani and Moradi 2004, Bostanov 2004, Burke et al 2002, Burke et al 2005, Coyle et al 2005, Coyle et al 2004, Dornhege et al 2003, Dornhege et al 2004, Fabiani et al 2004, Fukada S et al 1998, Garcia et al 2003b, Garrett et al 2003, Guger et al 2000b, Guger et al 2003a, Guger et al 2003b, Hinterberger et al 2003, Huan and Palaniappan 2004, Huggins et al 2003, Kelly et al 2002b, Kelly et al 2002a, Kelly et al 2004, Kelly et al 2005c, Kelly et al 2005a, Krauledat et al 2004, Krausz et al 2003, Lalor et al 2005, Leeb and Pfurtscheller 2004, Lemm et al 2005, Mensh et al 2004, Muller et al 2003c, Muller-Putz et al 2005a, Muller-Putz et al 2005b, Muller et al 2003b, Neuper et al 1999, Neuper et al 2003, Obermaier et al 2001b, Pfurtscheller et al 1998, Pfurtscheller and Guger 1999, Pfurtscheller et al 2000, Pfurtscheller and Neuper 2001, Pfurtscheller et al 2003b, Schloegl et al 1997a, Schloegl et al 1997b, Townsend et al 2004, Jia et al 2004) (R)FLD (Babiloni et al 2001b, Blanchard and Blankertz 2004, Blankertz et al 2002a, Blankertz et al 2003, Cincotti et al 2001, Cincotti et al 2003a, Guger et al 1999, Guger et al 2000a, Hung et al 2005, Obermaier et al 2001a, Pfurtscheller et al 2003a, Scherer et al 2004, Li et al 2004a) Sparse FLD (Blankertz et al 2002a) MD-based classifier (Babiloni et al 2001a, Cincotti et al 2003a, Cincotti et al 2003b, Garcia and Ebrahimi 2002, Molina et al 2003) Nonlinear discriminant function (Fabiani et al 2004) Bayes quadratic classifier (Keirn and Aunon 1990) Bayesian classifier (linear classifier) (Curran et al 2004, Lemm et al 2004, Penny et al 2000, Roberts and Penny 2003) Linear Bayesian decision rule (Vidal 1977) Linear classifier based on time-warping (Mason and Birch 2000) Logistic regression (Parra et al 2002, Parra et al 2003a) Linear classifier (no details) (Ramoser et al 2000) Single layer Perceptron model (a linear (Li et al 2004b, Wang et al 2004d) classifier) 2-dimensional linear classifier trained (Cheng et al 2004) by a non-enumerative search procedure ZDA (Hinterberger et al 2003) LDS (Lee and Choi 2002) Gaussian classifier (Millan et al 2004a, Millan 2004, Millan et al 2004b , Millan et al 2003a) SSP (Babiloni et al 2000, Babiloni et al 2001a, Babiloni et al 2001b, Cincotti et al 2001, Cincotti et al 2003a, Millan et al 2002b, Millan et al 2000b) SOM-based SSP (Millan et al 2002b, Millan et al 2000b) 11 M M CHMM (Rezek et al 2003) Based techniques ARHMM (Panuccio et al 2002) HMM + SVM (Lee and Choi 2002, Lee and Choi 2003) 9 Regularization may be applied before LDA classification scheme. Feature translation method HMM SVM MI)3 CN2 C4.5 k-NN Threshold detector Linear combination - Threshold detector Continuous feedback + Threshold detector Linear combination - Continuous feedback Continuous feedback Continuous feedback using MD Continuous audio feedback Variatioanal Kalman filter Static classifier that is inferred with sequential variational inference Random forest algorithm Reference II) (Cincotti et al 2003b, Lee and Choi 2003, Liu et al 2003, Obermaier et al 2001a, Obermaier et al 2001c, Obermaier et al 200Id, Pfurtscheller and Neuper 2001, Sykacek et al 2003) (Blankertz et al 2002a, Guan et al 2004, Garcia et al 2003a, Garcia et al 2003b, Garcia et al 2003c, Garrett et al 2003, Glassman 2005, Gysels and Celka 2004, Gysels et al 2005, Hung et al 2005, Guan et al 2005, Kaper and Ritter 2004a, Kaper and Ritter 2004b, Kaper et al 2004, Lai et al 2004, Peterson et al 2005, Schroder et al 2003, Schroder et al 2005, Thulasidas et al 2004, Trejo et al 2003, Xu et al 2004b, Yom-Tov and Inbar 2001, Yom-Tov and Inbar 2002, Yom-Tov and Inbar 2003, Yoon et al 2005) (Ivanovae* al 1995) (Ivanovae/ al 1995) (Ivanova et al 1995, Millan et al 2002a) (Blankertz et al 2002a, Pineda et al 2000, Pregenzer and Pfurtscheller 1999) (Allison and Pineda 2003, Balbale et al 1999, Bayliss and Ballard 1999, Bayliss and Ballard 2000a, Bayliss and Ballard 2000b, Bayliss 2003, Bayliss et al 2004, Calhoun and McMillan 1996, Cheng Ming et al 2005, Cheng and Gao 1999, Cheng et al 2001, Cheng et al 2002, Donchin et al 2000, Farwell and Donchin 1988, Gao et al 2003b, Graimann et al 2003a, Graimann et al 2003b, Graimann et al 2004, Hinterberger et al 2003, Huggins et al 1999, Huggins et al 2003, Jansen et al 2004, Kawakami et al 1996, Kelly et al 2005b, Kostov and Polak 1997, Lee et al 2005, Levine et al 1999, Levine et al 2000, McMillan and Calhoun 1995, Middendorf et al 2000, Pfurtscheller et al 2005, Pineda et al 2003, Polak and Kostov 1997, Polikoff et al 1995, Qin et al 2004a, Qin et al 2004b, Qin and He 2005, Roberts et al 1999, Serby et al 2005, Sutter 1992, Wang et al 2004b, Wang et al 2004a, Xu et al 2004a, Yom-Tov and Inbar 2003) (Townsend et al 2004) (Birbaumer et al 1999, Birbaumer et al 2000, Hinterberger et al 2003, Hinterberger et al 2004a, Hinterberger et al 2004b, Hinterberger et al 2005b, Hinterberger et al 2005a, Kaiser et al 2001, Kaiser et al 2002, Kubler et al 1999, Kubler et al 2001b, Kubler et al 1998, Neumann et al 2003, Neumann et al 2004) (Fabiani et al 2004, Krausz et al 2003, Kubler et al 2005, McFarland and Wolpaw 1998, McFarland et al 1997, McFarland et al 2003, McFarland et al 2005, Schalk et al 2000, Sheikh et al 2003, Wolpaw and McFarland 1994, Wolpaw et al 1997, Wolpaw et al 2000, Wolpaw et al 2003, Wolpaw and McFarland 2004) (Bashashati et al 2003, Cho et al 2004, LaCourse and Wilson 2003, Middendorf et al 2000, Trejo et al 2003, Wolpaw et al 1991) (Schlogl et al 2003) (Hinterberger and Baier 2005) (Sykacek et al 2004) (Curran et al 2004, Sykacek et al 2004) (Neuper al 1999) T A B L E 2.7 F E A T U R E CLASSIFICATION METHODS IN BCI DESIGNS THAT A R E B A S E D O N N E U R A L CORTICAL RECORDINGS Feature Classification Reference D3 Neural Recurrent MLP Networks Neural network (RNN) MLP Feed-forward ANN ANN recurrent dynamic back-propagation ANN model LVQ Other Support vector machine regression (SVR) model Cosine tuning model (a linear model) Linear Gaussian models (LGM) implemented by Kalman filter Generalized linear models (GLA) Generalized additive models (GAM) Weighted linear combination of neuronal activity (Wiener filter: a linear model) Gamma filter (a linear model) Mixture of multiple models based on NMF (non-negative matrix factorization) Echo State Networks (ESN) - Optimal sparse linear mapping Linear model (no details mentioned) Threshold detector SVM Bayesian classifier Maximum likelihood-based model LPF (continuous signal) Direct translation of firing rate to cursor movement (continuous signal) k-NN H M M (Sanchez et al 2002a, Sanchez et al 2002b, Sanchez et al 2003) (Kim et al 2005b) (Patil etal 2004) (Chapin et al 1999) (Hatsopoulos et al 2004, Wessberg et al 2000) (Laubach et al 2000) (Karniel et al 2002) (Kim etal 2005b) (Black et al 2003, Kemere et al 2004, Taylor et al 2002, Taylor et al 2003) (Black et al 2003, Gao et al 2003a, Patil et al 2004, Sanchez et al2002a, Wu et al 2002a, Wu et al 2002b) (Black et al 2003, Gao et al 2003a) (Black et al 2003, Gao et al 2003a) (Carmena et al 2005, Hatsopoulos et al 2004, Kim et al 2005a, Kim et al 2005b, Lebedev et al 2005, Patil et al 2004, Sanchez et al 2002b, Serruya et al 2003, Serruya et al 2002) (Sanchez et al 2002b) (Kim et al 2005a) (Rao et al 2005) (Carmena et al 2003, Wessberg et al 2000) (Otto et al 2003, Roushe et al 2003, Vetter et al 2003) (Byron et al 2005, Hu et al 2004, Olson et al 2005) (Gao et al 2002, Hu et al 2004, Musallam et al 2004) (Hatsopoulos et al 2004, Kemere et al 2004, Serruya et al 2003) (Karniel et al 2002, Kositsky et al 2003, Reger et al 2000a, Reger et al 2000b) (Kennedy et al 2000) (Isaacs et al 2000) (Darmanjian et al 2003) T A B L E 2.8 POST PROCESSING METHODS IN BCI DESIGNS Post processing Reference ID ERN (event-related negativity)-based error correction Successive averaging and/or rejection option for 'moderated' posterior probabilities (choice of "unknown" output state) (SA-UK) Debounce (considering refractory period) (Bayliss et al 2004, Blankertz et al 2002b, Blankertz et al 2003, Parra et al 2003b, Schalk et al 2000) (Anderson et al 1995a, Bashashati et al 2005, Birch eUal 2002, Borisoff et al 2004, Fatourechi et al 2004, Fatourechi et al 2005, Gysels and Celka 2004, Millan et al 1998, Millan 2004, Millan and Mourino 2003b, Millan et al 2004b, Muller-Putz et al 2005b, Penny et al 2000, Roberts and Penny 2003, Townsend et al 2004, Vidal 1977, Millan et al 2003a) (Bashashati et al 2005, Borisoff et al 2004, Fatourechi et al 2004, Fatourechi et al 2005, Muller-Putz et al 2005b, Obeid and Wolf 2004, Pfurtscheller et al 2005, Townsend et al 2004) 2.5. Discussion Several points raised in the previous section deserve further comment. F ig . 2.2 summarizes the information in Tables 2.3, 2.4, and 2.8, which respectively address signal-enhancement, feature selection and post-processing algorithms in B C I designs. Specifically, this figure shows the number of B C I designs that use specific signal-enhancement, feature selection, and post-processing techniques. In the remainder of this section we highlight the top 3 or 4 methods that have been used in the signal processing blocks of B C I systems (as introduced in Section 2.2). O f the 96 B C I designs that employ signal-enhancement techniques before extracting the features from the signal, 32% use surface Laplacian (SL), 22% use either principal component analysis (PCA) or independent component analysis ( ICA), 14% use common spatial patterns (CSP) and 11% use common average referencing ( C A R ) techniques. 38 o f the reported B C I designs employ feature-selection algorithms; 26% of these 38 designs use genetic algorithms (GA) , 24% use distinctive sensitive learning vector quantization ( D S L V Q ) , and 13% use P C A . O f the 30 B C I designs that use post-processing algorithms to reduce the amount of error in the output of the B C I system, 57% use averaging techniques and consider rejecting activations that have low certainty, 27% consider using the debounce block (or refractory period) to deactivate the output for a short period of time when a false activation is detected, and 16%o use event-related negativity (ERN) signals to detect error activations. 49 0 5 10 15 20 25 30 35 Number of designs Figure 2.2 Signal-enhancement, feature selection and post-processing methods in BCI designs Fig. 2.3 summarizes the results presented in Table 2.5, and shows the number of B C I designs that are based on sensorimotor activity, SCP, V E P , P300, activity of neural cells, and 'response to mental tasks' and use different feature extraction techniques. Based on the results of Fig . 2.3, 41% of the BCIs that are based on the sensorimotor activity use power-spectral-density features, 16% rely on parametric modeling of the data, 13% use time-frequency representation (TFR) methods, and 6% do not employ any feature extraction methods. 74% of the SCP-based B C I designs calculate SCP signals using low-pass filtering methods, and 64% of the VEP-based BCIs use power-spectral features at specific frequencies. 26% of the BCIs based on P300 calculate the peaks of the signal in a specific time window to detect the P300 component of the E E G ; 22% use TFR-based methods, 22% use no feature extraction method, and 15% use cross-correlation with a specific template. 41%o of the B C I designs that use mental tasks to control a B C I use power-spectral features 50 Figure 2.3 Feature extraction methods in BCI designs based on Sensorimotor Activity, VEP, P300, SCP, Response to Mental Tasks, Activity of Neural Cells, and Multiple Neuromechanisms and 37% use parametric modeling of the input signal. A s most of the B C I designs that are based on neural cortical recordings mainly try to model the direct relationship between the neural cortical recordings and movements, they do not use a feature-extraction algorithm. 45% of the B C I designs that are based on multiple neuromechanisms rely on power spectral features, 17% use parametric modeling, and 17% use time-frequency representation (TFR) methods. Summarizing Tables 2.6 and 2.7, the number of B C I designs that use different feature classification algorithms are shown in Fig. 2.4. About 75% of the B C I designs use classification schemes that are not based on neural networks (NN). These are composed of 5 1 those methods that use threshold detectors as the feature classifier or as part of the feature classification scheme (27%), linear discriminant (either L D A or F L D ) classifiers (26%), those that show continuous feedback of the extracted features (16%), and those that use support-vector-machines ( S V M ) (11%). 27%) of the neural-network-based classifiers are based on the multi-layer perceptrons ( M L P ) neural network and 39% are based on learning-vector-quantization ( L V Q ) classification scheme. Figure 2.4 Feature classification methods in BCI designs. 'In/Out Modeling' refers to those A N C -based BCIs that directly map the input neural recordings to the output without using a feature extraction technique. 52 During our analysis of the literature, a number o f salient points about the signal processing methods emerged. We think that some of these points are worth sharing with the B C I research community. In the following sections we summarize some of them. Note that these observations are based on comments made by the researchers in their published papers and are also based on the trends in the literature; they do not cover all the methods reported in the literature. 2.5.1 . S i g n a l E n h a n c e m e n t Signal-enhancement (pre-processing) algorithms have been used for brain computer interfaces that are based on the E E G and the activity of the neural cells ( A N C ) , but no signal-enhancement algorithms have been applied on electrocorticogram (ECoG)-based brain computer interfaces. Given the huge difference in the characteristics of E E G and A N C , the signal-enhancement algorithms used in EEG-based and ANC-based BCIs have very little overlap; only P C A has been used in both groups. While ANC-based BCIs mostly aim at spike detection, ranking, and sorting neuron activities, EEG-based ones mostly transform or select E E G channels that yield better performance. Overall, the use of a pre-processing stage before feature extraction ( if applied) has been proven to be useful. The choice of a suitable pre-processing technique however is dependent on several factors such as the recording technology, number of electrodes, and neuromechanism of the B C I . Next, we discuss some of the techniques used in signal-enhancement in EEG-based B C I systems. Specifically, a discussion of spatial filtering including referencing methods and common spatial patterns (CSP) is presented. Since these methods are among the most used techniques that have become increasingly popular in B C I studies. 2.5.1.1. R e f e r e n c i n g me thods Referencing methods are considered as spatial filters. The proper selection of a spatial filter for any B C I is determined by the location and extent of the control signal (e.g., the M u rhythm) and of the various sources of E E G or non-EEG noise. The latter two are not completely defined and presumably vary considerably between different studies and both across and within individuals (McFarland et al 1997). For BCIs that use the M u and Beta rhythms, the common average referencing ( C A R ) and Laplacian methods are superior to the ear reference method. This may be because these 53 methods use high-pass spatial filters and enhance the focal activity from the local sources (e.g., the M u and the Beta rhythms) and reduce the widely distributed activity, including that resulting from distant sources (e.g. E M G , eye movements and blinks, visual alpha rhythm). Comparing the two variations of the Laplacian filtering methods (the large Laplacian and the small Laplacian), it is shown that the large Laplacian method is superior to the small Laplacian method in B C I systems that use the M u rhythm (McFarland et al 1997). Although an accurate Laplacian estimate from raw potentials requires many electrodes, one study showed that the recognition rates were increased by using a small number of electrodes only (Cincotti et al 2003a). In this case, the linear combination of channels implementing the Laplacian estimation was likely to have caused a favorable transformation of the signals to recognize different patterns in the ongoing E E G . 2.5.1.2. Common spatial patterns (CSP) C S P is a signal-enhancement method that detects patterns in the E E G by incorporating the spatial information of the E E G signal. Some of its features and limitations include the following: C S P is a signal-enhancement method that detects patterns in the E E G by incorporating the spatial information of the E E G signal. A n advantage of the C S P method is that it does not require the a-priori selection of subject-specific frequency bands. Knowledge of these bands, however, is necessary for the band-power and frequency-estimation methods (Guger et al 2000b). One disadvantage of the, CSP method is that it requires the use of many electrodes. However, the inconvenience of applying more electrodes can be rationalized by improved performance (Guger et al 2000b, Pfurtscheller et al 2000). The major problem in the application of C S P is its sensitivity to artifacts in the E E G . Since the covariance matrices are used as the basis for calculating the spatial filters, and are estimated with a comparatively small number of examples, a single trial contaminated with artifacts can unfortunately cause extreme changes to the filters (Guger et al 2000b, Ramoser et al 2000). Since, the CSP method detects spatial patterns in the E E G , any change in the electrode positions may render the improvements in the classification accuracy gained by this method useless. Therefore, this method requires almost identical electrode positions for all trials and sessions which may be difficult to accomplish (Ramoser et al 2000). 54 2.5.2. Feature Extraction In this section we discuss some of the feature extraction techniques that have received more attention in B C I systems. Specifically, time and/or frequency representation methods, parametric modeling, and specific techniques of modeling neural cortical recordings are discussed. 2.5.2.1. Time and/or Frequency Methods A signal, as a function of time, may be considered as a representation with perfect temporal resolution. The magnitude of the Fourier transform (FT) of the signal may be considered as a representation with perfect spectral resolution but with no temporal information. Frequency-based features have been widely used in signal processing because of their ease of application, computational speed, and direct interpretation of the results. Specifically, about l / 3 r d o f B C I designs have used power-spectral features. Due to the non-stationary nature of the E E G signals, these features do not provide any time domain information. Thus, mixed time-frequency representations (TFRs) that map a one-dimensional signal into a two-dimensional function of time and frequency are used to analyze the time-varying spectral content of the signals. It has been shown that T F R methods may yield performance improvements comparing to the traditional FT-based methods (e.g. (Qin et al 2005, Lemm at al 2003, Bostanov 2004)). Most of the designs that employ T F R methods use wavelet-based feature-extraction algorithms. The choice of the particular wavelet used is a crucial factor in gaining useful information from wavelet analysis. Prior knowledge of the physiological activity in the brain can be useful in determining the appropriate wavelet function. Correlative T F R (CTFR) is another time-frequency representation method that, besides the spectral information, provides information about the time-frequency interactions between the components of the input signal. Thus, with the C T F R the E E G data samples are not independently analyzed (as in the Fourier transform case) but their relationship is also taken into account. One drawback of the C T F R resides in its relative high sensitivity to noise. Consequently, the most important values of the C T F R , in terms of classification must be selected (Garcia et al 2003a, b). 55 2.5.2.2. Parametric Modeling Parametric approaches assume the time series under analysis to be the output of a given linear mathematical model. They require an a-priori choice of the structure and order of the signal generation mechanism model (Weitkunat 1991). The optimum model order is best estimated not only by maximizing the fitness but also by limiting the model's complexity. For noisy signals, i f the model's order is too high, spurious peaks in the spectra w i l l result. On the other hand, i f the order is too low, smooth spectra are obtained (Kel ly et al 2002a, Polak and Kostov 1998, Weitkunat 1991). For short E E G segments, parametric modeling results in better frequency resolution and a good spectral estimate. Note that parametric modeling may yield poor estimates i f the length of the E E G segments processed is too short (Birch 1988). For such modeling, there is no need for a-priori information about potential frequency bands, and there is no need to window the data in order to decrease the spectral leakage. Also , the frequency resolution does not depend on the number of data points (Guger et al 2003a, Polak and Kostov 1998, Weitkunat 1991). Estimating these parameters, however, is very sensitive to artifacts (Birch 1988, Guger etal 2003a). Special attention should be paid to the choice of the sampling rate in parametric modeling (Weitkunat 1991), since severely oversampled signals tend to show only very small amplitude differences between successive samples. Hence, low-order models produce small prediction errors, giving the false illusion that an adequate model has been obtained. The sampling rates dictated by the Nyquist criterion are recommended. 2.5.2.3. Modeling the Neural Firing Rates Extraction algorithms for motor control operate on spike trains, recorded from a population of cortical units, mostly with the purpose of predicting arm trajectories. Several extraction methods such as linear filtering methods and neural networks have been used to determine arm movement trajectories from neural firing rates. We summarize below a few important issues in modeling the neural firing rates. A more detailed critical discussion of extraction algorithms for cortical control of arm prosthetics can be found in (Schwartz et al 2001). 56 One limitation of linear filter methods is that they rely on an a-priori model of movement-related neuronal responses. Artificial neural network ( A N N ) solutions can optimize each cell's contribution to the population prediction (Schwartz et al 2001). Several of the algorithms used are based on the position of the moving limb. In the primary motor cortex at least, this parameter is more poorly represented than the velocity during movement. With most algorithms, the different sources of variability need to be specified explicitly because some sort of optimal function is being modeled to the cell response (Schwartz ef a/2001). The performance of the modeling techniques is constrained by their training sets and may be limited, both in terms of extrapolation beyond and interpolation within the training set when new data are applied. The success of the linear filters is due to the underlying linearity of the relationship between firing rate and movement direction. These filters are limited by the conditions used to fit their coefficients and may suffer from the same training constraints as A N N s (Schwartz et al 2001). 2.5.3. F e a t u r e Se lec t ion Feature selection algorithms are used in B C I designs to find the most informative features for classification. This is especially useful for B C I designs with high dimensional input data, as it reduces the dimension of the feature space. Since the feature selection block reduces the complexity of the classification problem, higher classification accuracies might be achieved. The experiments carried out in (Flotzinger et al 1994, Pregenzer and Pfurtscheller 1999) show that when feature selection is used, the classification accuracy is better than when all the features are used. Principal component analysis ( P C A ) and genetic algorithms (GA) are among the mostly used feature selection and/or dimensionality reduction methods in BCIs. P C A has also been widely used in pre-processing stage of B C I designs. P C A is a linear transformation that can be used for dimensionality reduction in a dataset while retaining those characteristics of the dataset that contribute most to its variance, by keeping lower-order principal components and ignoring higher-order ones. Such low-order components often contain the "most important" aspects of the data. P C A has the distinction of being the optimal linear transformation for keeping the subspace that has largest variance. P C A only finds linear subspaces, works best 57 i f the individual components have Gaussian distributtions, and is not optimized for class separability. One other possible application area of P C A is in classification stage, in which, P C A is applied for weighting input features. While a standard neural network, like the multi-layer pereceptrons ( M L P ) can do the necessary classification itself, in some cases doing a P C A in parallel and weighting input features can give better results as it simplifies the training of the rest of the system. Unlike P C A , G A s are heuristic search techniques in the problem space. G A s typically maintain a constant-sized population of individuals which represent samples of the space to be searched. Each individual is evaluated on the basis of its overall fitness with respect to the given application domain. New individuals (samples of the search space) are produced by selecting high performing individuals to produce "offspring" which retain many of the features of their "parents". This eventually leads to a population that has improved fitness with respect to the given goal. Genetic algorithms have demonstrated substantial improvement over a variety of random and local search methods (De Jong 1975). This is accomplished by their ability to exploit accumulating information about an initially unknown search space in order to bias subsequent search into promising subspaces. Since G A s are basically a domain independent search technique, they are ideal for applications where domain knowledge and theory is difficult or impossible to provide (De Jong 1975). A n important step in developing a GA-based search is defining a suitable fitness function. A n ideal fitness function correlates closely with the algorithm's goal, and yet may be computed quickly. Speed of execution is very important, as a typical genetic algorithm must be iterated many, many times in order to produce a usable result for a non-trivial problem. Definition of the fitness function is not strightforward in many cases and often is performed iteratively i f the fittest solutions produced by a G A are not what is desired. 2.5.4. F e a t u r e C l a s s i f i c a t i o n Linear classifiers are generally more robust than nonlinear ones. This is because linear classifiers have fewer free parameters to tune, and are thus less prone to over-fitting (Muller et al 2003a). In the presence of strong noise and outliers, even linear systems can fail. One way of overcoming this problem is to use regularization. Regularization helps limit (a) the 58 influence of outliers and strong noise, (b) the complexity of the classifier, and (c) the raggedness of the decision surface (Muller et al 2003a). It is always desirable to avoid reliance on nonlinear classification methods, i f possible, because these methods often involve a number of parameters whose values must be chosen appropriately. However, when there are large amounts of data and limited knowledge of the data, nonlinear methods are better suited in finding the potentially more complex structure in the data. In particular, when the source of the data to be classified is not well understood, using methods that are good at finding nonlinear transformation of the data is suggested. In these cases, kernel-based and neural-networks-based methods can be used to determine the transformations. Kernel-based classifiers are classification methods that apply a linear classification in some appropriate (kernel) feature space. Thus, all the beneficial properties of linear classification are maintained, but at the same time, the overall classification is nonlinear. Examples of such kernel-based classification methods are support-vector-machines (SVMs) and kernel Fisher discriminant ( K F D ) (Muller et al 2003a). For a more detailed critical discussion regarding linear and nonlinear classifiers in brain computer interfaces, refer to (Muller et al 2003a). Some B C I designs have used classification algorithms such as F I R - M L P and T B N N that utilize temporal information of the input data (Haselsteiner and Pfurtscheller 2000, Ivanova et al 1995). The motivation for using such classifiers is that the patterns to be recognized are not static data but time series. Thus, the temporal information of the input data can be used to improve the classification results (Haselsteiner and Pfurtscheller 2000). Uti l iz ing the temporal information of features is not necessarily performed directly in the classification stage, and can be done with a static classifier like M L P and a mapping of the temporal input data to static data. However, using classifiers such as F I R - M L P and T B N N that directly utilize temporal information may yield better performances as they are much better suited for exploiting temporal information contained in the time series to be classified. Regardless of the method that is used for exploiting temporal information, these approaches are preferred over static classification as they may increase the performance of B C I systems. Using a group (committee) of classifiers rather than using a single classifier might also yield to better performances of B C I systems. Only a few B C I designs have employed such an 59 approach in classifying features and achieved performance improvements (Mil lan et al 2002b, Mi l l an et al 2000b, Peters et al 2001, Varsta et al 2000). The classification accuracy of the committee depends on how much unique information each committee member contributes to classification. A committee of classifiers usually yields better classification accuracy than any individual classifier could provide, and can be used to combine information from several channels, i.e., from different spatial regions (Peters et al 2001). A s the number o f epochs available for evaluating a B C I system is small, using a technique that reduces the bias of the estimated performance on a specific data set is highly recommended. This is especially important when different architectures of a certain design are being compared. K-fo ld cross-validation and statistical significance tests are especially useful for these cases (e.g. refer to (Anderson et al 1998, Ke l ly et al 2002b, Lalor et al 2005, Obermaier et al 2001 d, Peterson et al 2005)). K-fold cross-validation can be used simply to estimate the generalization error of a given model, or it can be used for model selection by choosing one of several models that has the smallest estimated generalization error but it is not suitable for online evaluations. A value of 5 to 10 for K is recommended for estimating the generalization error. For an insightful discussion of the limitations of cross-validatory choice among several learning methods, see (Stone 1977). 2.5.5. Post-Processing Post-processing techniques can be utilized in most of the B C I designs to decrease the error rates. Some post-processing techniques can be designed specifically for a target application. For example, when a B C I system is used to activate a spelling device, some letters can be omitted without losing information. The system can also take into consideration the conditional probabilities of letters provided by one or two preceding letters and make corresponding suggestions to the patient (Kubler et al 1999). Such techniques may also be feasible for other applications and consequently increase the performance of the B C I systems. There is a possibility that just after the end of a trial, some features of the brain signal reveal whether or not the trial was successful (that is, whether the outcome was or was not what the subject desired). These features are referred to as error potentials and can be used to detect errors in a B C I system and void the outcome. This error detection approach was encouraged 60 by evidence that errors in conventional motor performances have detectable effects on the E E G recorded just after the error occurs (Falkenstein et al 1995, Falkenstein et al 2001, Gehring et al 1995). Whatever the nature of the error potential, the central decision for a B C I is how useful the error potential can be in detecting errors in single trials, and thereby improving accuracy. While its signal-to-noise ratio (SNR) is low, the error potential can improve the performance of a B C I system. In the meantime, better methods for recognizing and measuring the error potential could substantially improve its SNR, and thereby increase its impact on accuracy of a B C I system. Such error potentials have been used in a few B C I systems to increase the performance (Bayliss et al 2004, Blankertz et al 2002b, Blankertz et al 2003, Parra et al 2003b, Schalk et al 2000). Another useful technique in decreasing false activations of B C I systems is to consider a measure of confidence in classification. In such case, the output of the system can only be activated when the probability of the output being in an active state is greater than a given probability threshold or some criterion. Otherwise, the response of the B C I is considered "unknown" and rejected to avoid making risky decisions. This is a useful way of reducing false decisions of the system (e.g., (Cincotti et al 2003b, Mi l l an et al 1998, Penny et al 2000)) and might be used in any B C I design. Considering mechanisms like debouncing the output of B C I designs also can reduce the number of false activations (Bashashati et al 2005, Borisoff et al 2004, Fatourechi et al 2004, Fatourechi et al 2005, Muller-Putz et al 2005b, Obeid and W o l f 2004, Pfurtscheller et al 2005, Townsend et al 2004). These methods are specifically useful for so called asynchronous (self-paced) BCIs. Since false positives could happen in periods longer than just a few samples, using debouncing technique in a manner similar to the debouncing of physical switches is expected to improve false activation rates (but with a cost in decreased re-activation time). The debounce component continuously monitors the output of the classifier. After an activation is detected (e.g., a change in logical state from '0 ' to T in a binary classifier), the output is activated for one time sample, then the output is forced to an inactive state for Td-1 time samples, where Td is the debounce time period in samples. In some studies this time period is referred to as refractory period. A s the debounce period is increased, the false activation rate is decreased for a given true positive rate. However, with 61 increasing this time period, the re-activation time of the B C I system is impacted. The trade-off is clear and one needs to consider this for a given application. 2.6. Conclusions We have completed the first comprehensive survey of signal processing methods used in B C I studies and published prior to January 2006. The results of this survey form a valuable and historical cross-reference for methods used in the following signal processing components of a B C I design: (1) pre-processing (signal-enhancement), (2) feature selection, (3) feature extraction, (4) feature classification, and (5) post-processing methods. This survey shows which signal-processing techniques have received more attention and which have not. This information is also valuable for newcomers to the field, as they can now find out which signal-processing methods have been used for a certain type of a B C I system. Many signal processing methods have been proposed and implemented in various brain computer interfaces and comparison of these methods for different B C I applications would be a useful task. However, at this point we cannot perform this task given the diversity of brain computer interface systems from different aspects such as target application, neurological phenomena used, amount of data tested, number of subjects and the amount of training they have received, recording systems, and experimental paradigms. Also acknowledged in (McFarland et al 2006), we think that comparison of methods would be possible in well-designed systematic studies (Jackson et al 2006) and on established datasets like the B C I Competition datasets (Blankertz et al 2004, Blankertz et al 2006). We believe that, for a fair comparison of methods, more data would be needed as comparing methods with the data of one or two subjects does not necessarily guarantee the same findings on a larger subject pool. (Jackson et al 2006) has provided a step towards this goal by proposing some ways to establish a systematic study both in design and reporting the results and we think that this task would only be possible with the collective help of all the researchers in this field. We hope that this study w i l l spawn further discussion of signal processing schemes for B C I designs. The proposed taxonomy and classes defined in Tables 2.3-8, represent a proposed set of subcategories, not a final one, and we encourage others to revise or expand upon this initial set. Our direction in the future is to establish an online public-accessible database 62 where research groups wi l l be able to submit their signal processing designs as well as propose revisions/expansions of the proposed definitions and categories presented in this paper. 2.7. References All i son B Z and Pineda J A 2003 ERPs evoked by different matrix sizes: Implications for a brain computer interface (BCI) system IEEE Trans. Neural Syst. Rehabil. Eng. 11 110-3 Al l i son B Z and Pineda J A 2005 Effects of S O A and flash pattern manipulations on ERPs, performance, and preference: Implications for a B C I system Int J Psychophysiol 59 127-40 Anderson C W , Devulapalli S V and Stolz E A 1995a Signal classification with different signal representations Proc. IEEE Workshop on Neural Networks for Signal Processing 475-83 Anderson C W , Stolz E A and Shamsunder S 1995b Discriminating mental tasks using E E G represented by A R models Proc. 17th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Montreal) pp 875-6 Anderson C W , Stolz E A and Shamsunder S 1998 Multivariate autoregressive models for classification of spontaneous electroencephalographic signals during mental tasks IEEE Trans. Biomed. Eng. 45 277-86 Babiloni F, Cincotti F, Lazzarini L , Mi l l an J, Mourino J, Varsta M , Heikkonen J, Bianchi L and Marciani M G 2000 Linear classification of low-resolution E E G patterns produced by imagined hand movements IEEE Trans. Rehabil. Eng. 8 186-8 Babiloni F, Bianchi L , Semeraro F, Mi l l an J R, Mourino J, Cattini A , Salinari S, Marciani M G and Cincotti F 2001a Mahalanobis distance-based classifiers are able to recognize E E G patterns by using few E E G electrodes Proc. 23rd Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Istanbul) pp 651-4 Babiloni C, Babiloni F, Carducci F, Cappa S F, Cincotti F, Del Percio C, Miniussi C , Moretti D V , Rossi S, Sosta K and Rossini P M 2004 Human cortical responses during one-bit short-term memory. A high-resolution E E G study on delayed choice reaction time tasks Clin. Neurophysiol. 115 161-70 63 Babiloni F, Cincotti F, Bianchi L , Pirri G , Mi l l an J R, Mourino J, Salinari S and Marciani M G 2001b Recognition of imagined hand movements with low resolution surface laplacian and linear classifiers Med. Eng. Phys. 23 323-8 Balbale U H , Higgins J E , Bement S L and Levine S P 1999 Multi-channel analysis of human event-related cortical potentials for the development of a direct brain interface Proc. 21st Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society & Annual Fall meeting of the Biomedical Engineering Society (Atlanta, GA) p 447 Barreto A B , Taberner A M and Vicente L M 1996a Classification of spatio-temporal E E G readiness potentials towards the development of a brain-computer interface Proc. IEEE Southeast Conf (Raleigh, NC) pp 99-102 Barreto A B , Taberner A M and Vicente L M 1996b Neural network classification of spatio-temporal E E G readiness potentials Proc. IEEE 15th Southern Biomedical Engineering Conf. (Dayton, OH) pp 73-6 Bashashati A , Ward R K and Birch G E 2005 A new design of the asynchronous brain computer interface using the knowledge of the path of features Proc. 2nd IEEE-EMBS Conf. on Neural Engineering (Arlington, VA)pp 101-4 Bashashati A , Ward R K , Birch G E , Hashemi M R and Khalilzadeh M A 2003 Fractal dimension-based E E G biofeedback system Proc. 25th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Cancun, Mexico) pp 2220-3 Bayliss J D 2003 Use of the evoked potential P3 component for control in a virtual apartment IEEE Trans Neural Syst Rehabil Eng 11 113-6 Bayliss J D and Ballard D H 1999 Single trial P300 recognition in a virtual environment Proc. Int. ICSC Symp. on Soft Computing in Biomedicine (Genova, Italy) Bayliss J D and Ballard D H 2000a Recognizing evoked potentials in a virtual environment Advances in Neural Information Processing Systems vol 12, eds S A Solla, T K Leen and K R Muller (Cambridge, M A : M I T Press) pp 3-9 Bayliss J D and Ballard D H 2000b A virtual reality testbed for brain-computer interface research IEEE Trans Rehabil. Eng. 8 188-90 64 Bayliss J D , Inverso S A and Tentler A 2004 Changing the P300 brain computer interface Cyberpsychol. Behav. 7 694-704 Birbaumer N , Ghanayim N , Hinterberger T, Iversen I, Kotchoubey B , Kubler A , Perelmouter J, Taub E and Flor H 1999 A spelling device for the paralysed Nature 398 297-8 Birbaumer N , Kubler A , Ghanayim N , Hinterberger T, Perelmouter J, Kaiser J, Iversen I, Kotchoubey B , Neumann N and Flor H 2000 The thought translation device (TTD) for completely paralyzed patients IEEE Trans Rehabil Eng 8 190-3 Birch G E 1988 Single trial EEG signal analysis using outlier information PhD Thesis (Vancouver, Canada: The University of British Columbia) Birch G E , Bozorgzadeh Z and Mason S G 2002 Initial on-line evaluations of the L F - A S D brain-computer interface with able-bodied and spinal-cord subjects using imagined voluntary motor potentials IEEE Trans Neural Syst. Rehabil. Eng. 10 219-24 Birch G E , Mason S G and Borisoff J F 2003 Current trends in brain-computer interface research at the neil squire foundation IEEE Trans. Neural Syst. Rehabil. Eng. 11 123-6 Black M J, Bienenstock E , Donoghue J P, W u W and Gao Y 2003 Connecting brains with machines: The neural control of 2D cursor movement Proc. 2nd IEEE-EMBS Conf. on Neural Engineering (Arlington, VA) pp 580-3 Blanchard G and Blankertz B 2004 B C I competition 2003-data set Ha: Spatial patterns of self-controlled brain rhythm modulations IEEE Trans. Biomed. Eng. 51 1062-6 Blankertz B , Curio G and Muller K R 2002a Classifying single trial E E G : Towards brain computer interfacing Advances in Neural Information Processing Systems vol 14, eds. T G Dietterich, S Becker and Z Ghahramani (Cambridge, M A : M I T Press) pp 157-64. Blankertz B , Dornhege G , Sch |-nfer C, Krepki R, Kolmorgen J, Muller K R, Kunzmann V , Losch F and Curio G 2003 Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial E E G analysis IEEE Trans. Neural Syst. Rehabil. Eng. 11 127-31 Blankertz B , Muller K R, Curio G , Vaughan T M , Schalk G , Wolpaw J R, Schlogl A , Neuper C, Pfurtscheller G , Hinterberger T, Schroder M and Birbaumer N 2004 The B C I competition 65 2003: Progress and perspectives in detection and discrimination of E E G single trials IEEE Trans. Biomed. Eng. 51 1044-51 Blankertz B , Muller K R, Krusienski D J, Schalk G , Wolpaw J R, Schlogl A , Pfurtscheller G , M i l l a n Jdel R, Schroder M and Birbaumer N 2006 The B C I competition. I l l : Validating alternative approaches to actual B C I problems IEEE Trans. Neural Syst. Rehabil. Eng. 14 153-9 Blankertz B , Sch [-nfer C, Dornhege G and Curio G 2002b Single trial detection of E E G error potentials: A tool for increasing B C I transmission rates Proc. Int. Conf. on Artificial Neural Networks (Madrid, Spain) pp 1137-43 Boostani R and Moradi M H 2004 A new approach in the B C I research based on fractal dimension as feature and adaboost as classifier J. Neural Eng. 1 212-7 Borisoff J F, Mason S G , Bashashati A and Birch G E 2004 Brain-computer interface design for asynchronous control applications: Improvements to the L F - A S D asynchronous brain switch IEEE Trans. Biomed. Eng. 51 985-92 Bostanov V 2004 B C I competition 2003—data sets ib and l ib : Feature extraction from event-related brain potentials with the continuous wavelet transform and the t-value scalogram IEEE Trans. Biomed. Eng. 51 1057-61 Bozorgzadeh Z , Birch G E and Mason S G 2000 The L F - A S D B C I : On-line identification of imagined finger movements in spontaneous E E G with able-bodied subjects Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (Istanbul, Turkey) pp 109-13 Burke D , Ke l ly S, Chazal P d and Rei l ly R 2002 A simultaneous filtering and feature extraction strategy for direct brain interfacing Proc. 24th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Houston, TX) pp 279-80 Burke D P, Ke l ly S P, de Chazal P, Rei l ly R B and Finucane C 2005 A parametric feature extraction and classification strategy for brain-computer interfacing IEEE Trans. Neural Syst. Rehabil. Eng. 13 12-7 Byron O, He J P, H u J and Si J 2005 A conceptual brain machine interface system Proc. 1st Int. Conf on Neural Interface and Control (Wuhan, .China) pp 40-44 66 Calhoun G L and M c M i l l a n G R 1996 EEG-based control for human-computer interaction Proc. 3rd Symp. On Human Interaction with Complex Systems (Dayton, OH) pp 4-9 Carmena J M , Lebedev M A , Crist R E , O'Doherty J E , Santucci D M , Dimitrov D F, Patil P G , Henriquez C S and Nicolelis M A 2003 Learning to control a brain-machine interface for reaching and grasping by primates Plos Biol 1 193-208 Carmena J M , Lebedev M A , Henriquez C S and Nicolelis M A 2005 Stable ensemble performance with single-neuron variability during reaching movements in primates J Neurosci 25 10712-6 Chapin J K , Moxon K A , Markowitz R S and Nicolelis M A 1999 Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex Nat. Neurosci. 2 664-70 Cheng M and Gao S 1999 A n EEG-based cursor control system Proc. 21 s t Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society & Annual Fall meeting of the Biomedical Engineering Society (Atlanta, G A ) p 669 Cheng M , Gao X , Gao S and Wang B 2005 Stimulation frequency extraction in S S V E P -based brain-computer interface Proc. 1st Int. Conf. on Neural Interface and Control (Wuhan, China) pp 64-7 Cheng M , Gao X , Gao S and X u D 2002 Design and implementation of a brain-computer interface with high transfer rates IEEE Trans. Biomed. Eng. 49 1181-6 Cheng M , Jia W , Gao X , Gao S and Yang F 2004 M u rhythm-based cursor control: A n offline analysis Clin. Neurophysiol. 115 745-51 Cheng M , X u D , Gao X and Gao S 2001 Brain-computer interface with high transfer rates Proc. 8th Int. Conf. on Neural Information Processing (Shanghai, China) Cho B H , K i m S, Shin D I, Lee J H , Lee S M , K i m I Y and K i m S I 2004 Neurofeedback training with virtual reality for inattention and impulsiveness Cyberpsychol Behav. 7 519-26 Cincotti F, Bianchi L , Mi l l an J, Mourino J, Salinari S, Marciani M G and Babiloni F 2001 Brain computer interface: The use of low resolution surface laplacian and linear classifiers 67 for the recognition of imagined hand movements Proc. 23r Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Istanbul) pp 655-8 Cincotti F, Mattia D , Babiloni C, Carducci F, Salinari S, Bianchi L , Marciani M G and Babiloni F 2003a The use of E E G modifications due to motor imagery for brain-computer interfaces IEEE Trans. Neural Syst. Rehabil. Eng. 11 131-3 Cincotti F, Scipione A , Timperi A , Mattia D , Marciani M G, Mi l l an J, Salinari S, Bianchi L and Babiloni F 2003b Comparison of different feature classifiers for brain computer interfaces Proc. 25th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Cancun, Mexico) pp 645-7 Costa E J and Cabral E F 2000 EEG-based discrimination between imagination of left and right hand movements using adaptive gaussian representation Med. Eng. Phys. 22 345-8 Coyle D , Prasad G and McGinni ty T M 2004 Extracting features for a brain-computer interface by self-organising fuzzy neural network-based time series prediction Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 4371-4 Coyle D , Prasad G and McGinni ty T M 2005 A time-frequency approach to feature extraction for a brain-computer interface with a comparative analysis of performance measures EURASIP J. Appl. Signal Processing 2005 3141-51 Curran E , Sykacek P, Stokes M , Roberts S J, Penny W , Johnsrude I and Owen A M 2004 Cognitive tasks for driving a brain-computer interfacing system: A pilot study IEEE Trans Neural Syst Rehabil Eng 12 48-54 Darmanjian S, K i m S P, Nechbya M C, Morrison S, Principe J, Wessberg J and Nicolelis M A 2003 Bimodal brain-machine interface for motor control of robotic prosthetic Proc. IEEE/RSJInt. Conf. on Intelligent Robots and Systems (Las Vegas, NV) pp 3612-7 De Jong K A 1975 An analysis of the behaviour of a class of genetic adaptive systems PhD thesis (Ann Arbor, M I : University of Michigan) Diss. Abstr. Int. 36(10), 5140B, University Microfilms No. 76-9381. 44. Deecke L and Kornhuber H 1976 Voluntary finger movement in man: Cerebral potentials and theory Biol. Cybern. 23 99-119 68 Donchin E , Spencer K M and Wijesinghe R 2000 The mental prosthesis: Assessing the speed of a P300-based brain-computer interface IEEE Trans Rehabil Eng 8 174-9 Donoghue J P 2002 Connecting cortex to machines: Recent advances in brain interfaces Nat. Neurosci. 5 1085-8 Dornhege G , Blankertz B and Curio G 2003 Speeding up classification of multi-channel brain-computer interfaces: Common spatial patterns for slow cortical potentials Proc. 2nd IEEE-EMBS Conf. on Neural Engineering (Arlington, VA) pp 595-8 Dornhege G , Blankertz B , Curio G and Muller K R 2004 Boosting bit rates in noninvasive E E G single-trial classifications by feature combination and multiclass paradigms IEEE Trans. Biomed. Eng. 51 993-1002 Erfanian A and Erfani A2004 ICA-based classification scheme for EEG-based brain-computer interface: The role of mental practice and concentration skills Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 235-8 Fabiani G E , McFarland D J, Wolpaw J R and Pfurtscheller G 2004 Conversion of E E G activity into cursor movement by a brain-computer interface (BCI) IEEE Trans. Neural Syst. Rehabil. Eng. 12 331-8 Falkenstein M , Hielscher H , Dziobek I, Schwarzenau P, Hoormann J, Sunderman B and Hohnsbein J 2001 Action monitoring, error detection, and the basal ganglia: an E R P study Neuroreport 12 157-61 Falkenstein M , Hohnsbein J and Hoormann J 1995 Event-related potential correlates of errors in reaction tasks Perspectives of event-related brain potentials research ed Karmos et al (Amsterdam: Elsevier) pp 287-296 Farwell L A and Donchin E 1988 Talking off the top of your head: Towards a mental prosthesis utilizing event-related brain potentials Electroencephalogr Clin Neurophysiol 8 0 510-23 Fatourechi M , Bashashati A , Borisoff J F, Birch G E and Ward R K 2004 Improving the performance of the L F - A S D brain computer interface by means of genetic algorithm Proc. IEEE Symposium on Signal Processing and Information Technology (Rome, Italy) pp 38-41 69 Fatourechi M , Bashashati A , Ward R K and Birch G E 2005 A hybrid genetic algorithm approach for improving the performance of the L F - A S D brain computer interface Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (Philadelphia, PA) pp 345-8 Flotzinger D , Pregenzer M and Pfurtscheller G 1994 Feature selection with distinction sensitive learning vector quantisation and genetic algorithms Proc. IEEE Int. Conf On Neural Networks (Orlando, FL) pp 3448-51 Fukada S, Tatsumi D , Tsujimoto H and Inokuchi S 1998 Studies of input speed of word inputting system using event-related potential Proc. 20th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Hong Kong) pp 1458-60 Gao Y , Black M J, Bienenstock E , Shoham S and Donoghue J P 2002 Probabilistic inference of hand motion from neural activity in motor cortex Advances in Neural Information Processing Systems vol 14, eds. T G Dietterich, S Becker and Z Ghahramani (Cambridge, M A : M I T Press) pp 213-20 Gao Y , Black M J, Bienenstock E , W u W and Donoghue J P 2003a A quantitative comparison of linear and non-linear models of motor cortical activity for the encoding and decoding of arm motions Proc. 25th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Hong Kong) pp 189-92 Gao X , X u D , Cheng M and Gao S 2003b A BCI-based environmental controller for the motion-disabled IEEE Trans. Neural Syst. Rehabil. Eng. 11 137-40 Gao X , X u N , Hong B , Gao S and Yang F 2004 Optimal selection of independent components for event-related brain electrical potential enhancement Proc. IEEE Int. Workshop on Biomedical Circuits and Systems (Singapore) pp S3/5/INV-S3/5/1-4 Garcia G and Ebrahimi T 2002 Time-frequency-space kernel for single EEG-tr ial classification Presented at the 5th Nordic Signal Processing Symp. (Norway) Garcia G , Ebrahimi T and Vesin J M 2002 Classification of E E G signals in the ambiguity domain for brain computer interface applications Proc. IEEE 14th Int. Conf. on Digital Signal Processing (Santorini, Greece)_pp 301-305 70 Garcia G , Ebrahimi T and Vesin J M 2003a Correlative exploration of E E G signals for direct brain-computer communications Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (Hong Kong) pp 816-9 Garcia G , Ebrahimi T and Vesin J M 2003b Support vector E E G classification in the fourier and time-frequency correlation domains Proc. 1st IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 591-4 Garcia G N , Ebrahimi T, Vesin J M and V i l l c a A 2003 c Direct brain-computer communication with user rewarding mechanism Proc. IEEE Int. Symp. in Information Theory (Yokohama, Japan) pp 221. Garrett D , Peterson D A , Anderson C W and Thaut M H 2003 Comparison of linear, nonlinear, and feature selection methods for E E G signal IEEE Trans. Neural Syst. Rehabil. Eng. 11 141-4 Gehring W J, Coles M G H , Meyer D E , Donchin E 1995 A brain potential manifestation of error-related processing," In: Perspectives of event-related brain potentials research EEG Suppl 44 ed Karmos et al (Amsterdam: Elsevier) pp 261-72 Georgopoulos A P, Langheim F J, Leuthold A C and Merkle A N 2005 Magnetoencephalographic signals predict movement trajectory in space Exp. Brain Res. 167 132-5 Gevins A , Bressler S L , Morgan N H , Cutillo B A , White R M , Greer D S and files J 1989 Event-related covariances during a bimanual visuomotor taks. I. methods and analysis of stimulus- and response-locked data Electroencephalogr Clin Neurophysiol 74 58-75 Glassman E L 2005 A wavelet-like filter based on neuron action potentials for analysis of human scalp electroencephalographs IEEE Trans. Biomed. Eng. 52 1851-62 Graimann B , Huggins J E , Levine S P and Pfurtscheller G 2003a Detection of E R P and E R D / E R S patterns in single E C o G channels Proc. 1st IEEE-EMBS Conf on Neural Engineering (Capri Island, Italy) pp 614-7 Graimann B , Huggins J E , Levine S P and Pfurtscheller G 2004 Toward a direct brain interface based on human subdural recordings and wavelet-packet analysis IEEE Trans. Biomed. Eng. 51 954-62 71 Graimann B , Huggins J E , Schlogl A , Levine S P and Pfurtscheller G 2003b Detection of movement-related desynchronization patterns in ongoing single-channel electrocorticogram IEEE Trans. Neural Syst. Rehabil. Eng. 11 276-81 Guan C, Thulasidas M and W u J 2004 High performance P300 speller for brain-computer interface Proc. IEEE Int. Workshop on Biomedical Circuits and Systems (Singapore) pp S3/5/INV-S3/13-16 Guger C, Edlinger G , Harkam W , Niedermayer I and Pfurtscheller G 2003a How many people are able to operate an EEG-based brain-computer interface (BCI)? IEEE Trans. Neural Syst. Rehabil. Eng. 11 145-7 Guger C, Edlinger G and Pfurtscheller G2003b How many people are able to operate an EEG-based brain-computer interface (BCI)? Presented at the 2nd Int. Meeting on Brain-Computer Interfaces for Communication and Control (Albany, NY) Guger C, Muller G , Neuper C, Krausz G , Niedermayer I and Pfurtscheller G 2000a Brain-computer communication system: The EEG-based control of a hand orthesis in a quadriplegic patient Presented at the Int. Conf. on Computers Helping People with Special Needs (Karlsruhe, Germany) Guger C, Ramoser H and Pfurtscheller G 2000b Real-time E E G analysis with subject-specific spatial patterns for a brain-computer interface (BCI) IEEE Trans. Rehabil. Eng. 8 447-56 Guger C, Schloegl A , Walterspacher D and Pfurtscheller G 1999 Design of an EEG-based brain-computer interface (BCI) from standard components running in real-time under windows Biomed Tech (Berl) 44 12-6 Gysels E and Celka P 2004 Phase synchronization for the recognition of mental tasks in a brain-computer interface IEEE Trans. Neural Syst. Rehabil. Eng. 12 406-15 Gysels E , Renevey P and Celka P 2005 SVM-based recursive feature elimination to compare phase synchronization computed from broadband and narrowband E E G signals in brain-computer interfaces Signal Process. 8 5 2178-89 72 Guan J, Chen Y , L i n J, Y u n Yuan and Huang M 2005 N 2 components as features for brain computer interface Proc. Ist Int. Conf. on Neural Interface and Control (Wuhan, .China) pp 45-9 Hallett M 1994 Movement-related cortical potentials Electromyogr. Clin. Neurophysiol. 3 4 5-13 Haselsteiner E and Pfurtscheller G 2000 Using time-dependent neural networks for E E G classification IEEE Trans. Rehabil. Eng. 8 457-63 Hatsopoulos N , Joshi J and O'Leary J G 2004 Decoding continuous and discrete motor behaviors using motor and premotor cortical ensembles J Neurophysiol. 9 2 1165-74 H i l l N J, La i T N , Bierig K , Birbaumer N and Scholkopf B 2004 Attention modulation of auditory event-related potentials in a brain-computer interface Proc. IEEE Int. Workshop on Biomedical Circuits and Systems (Singapore) pp S3/5/INV-S3/17-20 Hinterberger T and Baier G 2005 Parametric orchestral sonification of E E G in real time IEEE Multimedia 12 70-9 Hinterberger T, Kubler A , Kaiser J, Neumann N and Birbaumer N 2003 A brain-computer interface (BCI) for the locked-in: Comparison of different E E G classifications for the thought translation device Electroencephalogr Clin Neurophysiol 1 1 4 416-25 Hinterberger T, Neumann N , Pham M , Kubler A , Grether A , Hofmayer N , Wilhelm B , Flor H and Birbaumer N 2004a A multimodal brain-based feedback and communication system Exp Brain Res 1 5 4 521 -6 Hinterberger T, Veit R, Wilhelm B , Weiskopf N , Vatine J J and Birbaumer N 2005a Neuronal mechanisms underlying control of a brain-computer interface Eur. J. Neurosci. 2 1 3169-81 Hinterberger T, Weiskopf N , Veit R, Wilhelm B , Betta E and Birbaumer N 2004b A n E E G -driven brain-computer interface combined with functional magnetic resonance imaging (fMRI) IEEE Trans. Biomed. Eng. 51 971-4 73 Hinterberger T, Wilhelm B , Mellinger J, Kotchoubey B and Birbaumer N 2005b A device for the detection of cognitive brain functions in completely paralyzed or unresponsive patients IEEE Trans. Biomed. Eng. 52 211-20 H u J, Si J, Olson B P and He J 2004 Principle component feature detector for motor cortical control Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 4021-4 Huan N J and Palaniappan R 2004 Neural network classification of autoregressive features from electroencephalogram signals for brain-computer interface design/ . Neural Eng. 1 142-50 Huggins J E , Levine S P, Bement S L , Kushwaha R K , Schuh L A , Passaro E A , Rohde M M , Ross D A , Elisevich K V and Smith B J 1999 Detection of event-related potentials for development of a direct brain interface J Clin Neurophysiol 16 448-55 Huggins J E , Levine S P, Fessler J A , Sowers W M , Pfurtscheller G , Graimann B , Schloegl A , Minecan D N , Kushwaha R K , Bement S L , Sagher O and Schuh L A 2003 Electrocorticogram as the basis for a direct brain interface: Opportunities for improved detection accuracy Proc. 1st IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 587-90 Hung C I, Lee P L , W u Y T, Chen L F, Y e h T C and Hsieh J C 2005 Recognition of motor imagery electroencephalography using independent component analysis and machine classifiers Ann. Biomed. Eng. 3 3 1053-70 Isaacs R E , Weber D J and Schwartz A B 2000 Work toward real-time control of a cortical neural prothesis IEEE Trans Rehabil. Eng. 8 196-8 Ivanova I, Pfurtscheller G and Andrew C 1995 Al-based classification of single-trial E E G data Proc. 17th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Montreal, Canada) pp 703-4 Jackson M M , Mason S G and Birch G E 2006 Analyzing trends in brain interface technology: A method to compare studies Ann. Biomed. Eng. 3 4 859-78 Jansen B H , A l l am A , Kota P, Lachance K , Osho A and Sundaresan K 2004 A n exploratory study of factors affecting single trial P300 detection IEEE Trans. Biomed. Eng. 51 975-8 74 Jasper H and Penfield W 1949 Electrocortiograms in man: Effect o f voluntary movement upon the electrical activity of the precentral gyrus Arch. Psychiat. Nervenkr. 183 163-74 Jia W , Zhao X , L i u H , Gao X , Gao S and Yang F 2004 Classification of single trial E E G during motor imagery based on E R D Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 5-8 Kaiser J, Kubler A , Hinterberger T, Neumann N and Birbaumer N 2002 A non-invasive communication device for the paralyzed Minim Invasive Neurosurg 45 19-23 Kaiser J, Perelmouter J, Iversen I H , Neumann N , Ghanayim N , Hinterberger T, Kubler A , Kotchoubey B and Birbaumer N 2001 Self-initiation of EEG-based communication in paralyzed patients Electroencephalogr Clin Neurophysiol 112 551-4 Kalcher J, Flotzinger D and Pfurtscheller G 1992 A new approach to a brain-computer-interface (BCI) based on learning vector quantization (LVQ3) Proc. 14th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Paris, France) pp 1658-9 Kalcher J, Flotzinger D and Pfurtscheller G 1993 Graz brain-computer interface: A n E E G -based cursor control system Proc. 15th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Diego, CA) pp 1264-5 Kaper M , Meinicke P, Grossekathoefer U , Lingner T and Ritter H 2004 B C I competition 2003—data set l i b : Support vector machines for the P300 speller paradigm IEEE Trans. Biomed. Eng. 51 1073-6 Kaper M and Ritter H 2004a Generalizing to new subjects in brain-computer interfacing Proc. 26 t h Annual Int. Conf. o f the I E E E Engineering in Medicine and Biology Society (San Francisco, C A ) pp 4363-6 Kaper M and Ritter H 2004b Progress in P300-based brain-computer interfacing Proc. IEEE Int. Workshop on Biomedical Circuits and Systems (Singapore) pp S3/5/INV-S3/59-12 Karniel A , Fleming K M , Sanguineti V , Alford S and Mussa-Ivaldi F A 2002 Dynamic properties of the lamprey's neuronal circuits as it drives a two-wheeled robot Proc. SAB '2002 Workshop on Motor Control in Humans and Robots: On the Interplay of Real Brains and Artificial Devices (Edinburgh, Scotland) pp 29-36 75 Kawakami T, Inoue M , Kobayashi Y and Nakashima K 1996 Application of event related potentials to communication aids Proc. 18th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Amsterdam, Holland) pp 2229-31 Keirn Z A and Aunon J I 1990 A new mode of communication between man and his surroundings IEEE Trans. Biomed. Eng 37 1209-14 Ke l ly S, Burke D , Chazal P d and Rei l ly R 2002a Parametric models and classification for direct brain interfaces Proc. Irish Signal and Systems Conf. (Cork, Ireland) pp 25-26 K e l l y S, Burke D , Chazal P d and Rei l ly R 2002b Parametric models and spectral analysis for classification in brain-computer interfaces Proc. 14th IEEE International Conf. on Digital Signal Proceesing (Santorini, Greece) pp 307-10 Ke l ly S P, Lalor E , Finucane C and Rei l ly R B 2004 A comparison of covert and overt attention as a control option in a steady-state visual evoked potential-based brain computer interface Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 4725-8 Ke l ly S P, Lalor E , Rei l ly R B and Foxe J J 2005a Independent brain computer interface control using visual spatial attention-dependent modulations of parieto-occipital alpha Proc. IEEE-EMBS Conf. on Neural Engineering (Arlington, VA) pp 667-70 Ke l ly S P, Lalor E C, Finucane C, McDarby G and Rei l ly R B 2005b Visual spatial attention control in an independent brain-computer interface IEEE Trans. Biomed. Eng 52 1588-96 K e l l y S P, Lalor E C, Rei l ly R B and Foxe J J 2005c Visual spatial attention tracking using high-density S S V E P data for independent brain-computer communication IEEE Trans. Neural Syst Rehabil Eng 13 172-8 Kemere C, Shenoy K V and Meng T H 2004 Model-based neural decoding of reaching movements: A maximum likelihood approach IEEE Trans Biomed Eng 51 925-32 Kennedy P R, Bakay R A , Moore M M , Adams K and Goldwaithe J 2000 Direct control of a computer from the human central nervous system IEEE Trans Rehabil Eng 8 198-202 76 K i m K H , K i m S S and K i m S J 2005a Superiority of nonlinear mapping in decoding multiple single-unit neuronal spike trains: A simulation study J Neurosci Methods 150 202-11 K i m S P, Rao Y N , Erdogmus D , Sanchez J C, Nicolelis M A L and Principe J C 2005b Determining patterns in neural activity for reaching movements using nonnegative matrix factorization EURASIP J Appl Signal Processing 19 3113-21 K i r a K and Rendell L A 1992 The feature selection problem: Traditional methods and a new algorithm Proc. of the 10th National Conf on Artificial Intelligence (San Jose, CA) pp 129-34 Kositsky M , Karniel A , Alford S, Fleming K M and Mussa-Ivaldi F A 2003 Dynamical dimension of a hybrid neurorobotic system IEEE Trans Neural Syst Rehabil Eng 11 155-9 Kostov A and Polak M 1997 Prospects of computer access using voluntary modulated E E G signal Proc. ECPD Symposium on Brain & Consciousness (Belgrade, Yugoslavia) pp 233-6 Kostov A and Polak M 2000 Parallel man-machine training in development of EEG-based cursor control IEEE Trans Rehabil Eng 8 203-5 Kozelka J W and Pedley T A 1990 Beta and mu rhythms J. Clin. Neurophysiol. 7 191-207 Krauledat M , Dornhege G , Blankertz B , Losch F, Curio G and Muller K 2004 Improving speed and accuracy of brain-computer interfaces using readiness potential features Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, G4jpp4511-5 Krausz G , Scherer R, Korisek G and Pfurtscheller G 2003 Critical decision-speed and information transfer in the "graz brain-computer interface" Appl.Psychophysiol.Biofeedback 28 233-40 Kubler A , Kotchoubey B , Hinterberger T, Ghanayim N , Perelmouter J, Schauer M , Fritsch C, Taub E and Birbaumer N 1999 The thought translation device: A neurophysiological approach to communication in total motor paralysis Exp Brain Res 124 223-2 Kubler A , Kotchoubey B , Kaiser J, Wolpaw J R and Birbaumer N 2001a Brain-computer communication: Unlocking the locked in Psychol Bull 127 358-75 77 Kubler A , Kotchoubey B , Salzmann H P, Ghanayim N , Perelmouter J, Homberg V and Birbaumer N 1998 Self-regulation of slow cortical potentials in completely paralyzed human patients Neurosci Lett 252 171-4 Kubler A , Neumann N , Kaiser J, Kotchoubey B , Hinterberger T and Birbaumer N P 2001b Brain-computer communication: Self-regulation of slow cortical potentials for verbal communication Arch Phys Med Rehabil 82 1533-9 Kubler A , Nijboer F, Mellinger J, Vaughan T M , Pawelzik H , Schalk G , McFarland D J, Birbaumer N and Wolpaw J R 2005 Patients with A L S can use sensorimotor rhythms to operate a brain-computer interface Neurology 64 Ml5-1 LaCourse J R and Wilson E 2003 Brainiac: A brain computer link Proc. IEEE Instrumentation and Measurement Technology Conf (Vail, CO) pp 587-92 La i T N , Schroder M , Hinterberger T, Weston J, Bogdan M , Birbaumer N and Scholkopf B 2004 Support vector channel selection in B C I IEEE Trans. Biomed. Eng 51 1003-10 Lalor E , K e l l y S, Finucane C, Burke R, Smith R, Rei l ly R and McDarby G 2005 Steady-state VEP-based brain computer interface control in an immersive 3-D gaming environment EURASIP J. Appl. Signal Processing 19 315 6-64 Laubach M , Wessberg J and Nicolelis M A 2000 Cortical ensemble activity increasingly predicts behaviour outcomes during learning of a motor task Nature 405 567-71 Lebedev M A , Carmena J M , O'Doherty J E , Zacksenhouse M , Henriquez C S, Principe J C and Nicolelis M A 2005 Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface J Neurosci 25 4681 -93 Lee H and Choi S 2002 PCA-based linear dynamical systems for multichannel E E G classification Proc. 9th Int Conf on Neural Information Processing (Singapore) pp 745-9 Lee H and Choi S 2003 P C A + H M M + S V M for E E G pattern classification Proc. I E E E 7 t h Int. Symp. on Signal Processing and Its Applications (Paris, France) pp 541-4 Lee P L , W u C H , Hsieh J C and W u Y T 2005 Visual evoked potential actuated brain computer interface: A brain-actuated cursor system Electronics Letters 41 832-4 78 Leeb R and Pfurtscheller G 2004 Walking through a virtual city by thought Proc. 26 t n Annual Int. Conf. of the I E E E Engineering in Medicine and Biology Society (San Francisco, C A ) pp 4503-6 Lemm S, Blankertz B , Curio G and Muller K R 2005 Spatio-spectral filters for improving the classification of single trial E E G IEEE Trans. Biomed. Eng 52 1541-8 Lemm S, Schafer C and Curio G 2004 B C I competition 2003-data set III: Probabilistic modeling of sensorimotor mu rhythms for classification of imaginary hand movements IEEE Trans. Biomed. Eng 51 1077-80 Levine S P, Huggins J E , Bement S L , Kushwaha R K , Schuh L A , Passaro E A , Rohde M M and Ross D A 1999 Identification of electrocorticogram patterns as the basis for a direct brain interface J Clin Neurophysiol 16 439-47 Levine S P, Huggins J E , Bement S L , Kushwaha R K , Schuh L A , Rohde M M , Passaro E A , Ross D A , Elisevich K V and Smith B J 2000 A direct brain interface based on event-related potentials IEEE Trans Rehabil Eng 8 180-185 L i Y , Cichocki A , Guan C and Qin J 2004a Sparse factorization preprocessing-based offline analysis for a cursor control experiment Proc. IEEE Int. Workshop on Biomedical Circuits and Systems (Singapore) pp S3/5/INV-S3/5/5-8 L i Y , Gao X , L i u H and Gao S 2004b Classification of single-trial electroencephalogram during finger movement IEEE Trans. Biomed. Eng 51 1019-25 Lisogurski D and Birch G E 1998 Identification of finger flexions from continuous E E G as a brain computer interface Proc. 20th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Hong Kong) pp 2004-7 L i u H , Wang J and Zheng C 2005 Mental tasks classification and their E E G structures analysis by using the growing hierarchical self-organizing map Proc. 1st Int. Conf. on Neural Interface and Control (Wuhan, China) pp 115-8 L i u H S, Gao X , Yang F and Gao S 2003 Imagined hand movement identification based on spatio-temporal pattern recognition o f E E G Proc. IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 599-602 79 Mahmoudi B and Erfanian A 2002 Single-channel EEG-based prosthetic hand grasp control for amputee subjects Proc. 24th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Houston, TX) pp 2406-7 Mason S G and Birch G E 2000 A brain-controlled switch for asynchronous control applications IEEE Trans. Biomed. Eng 47 1297-307 Mason S G and Birch G E 2003 A general framework for brain-computer interface design IEEE Trans. Neural Syst. Rehabil. Eng 11 70-85 Mason S G , Bohringer R, Borisoff J F and Birch G E 2004 Real-time control of a video game with a direct brain—computer interface / . Clin. Neurophysiol 21 404-8 McFarland D and Wolpaw J R 1998 EEG-based communication and control: Short-term role of feedback IEEE Trans. Rehabil. Eng 6 7-11 McFarland D J, Anderson C W , Muller K R, Schlogl A and Krusienski D J 2006 B C I meeting 2005—workshop on B C I signal processing: Feature extraction and translation IEEE Trans. Neural Syst. Rehabil. Eng 1 4 135-8 McFarland D J, McCane L M , David S V and Wolpaw J R 1997 Spatial filter selection for EEG-based communication Electroencephalogr. Clin. Neurophysiol 1 0 3 386-94 McFarland D J, Sarnacki W A , Vaughan T M and Wolpaw J R 2005 Brain-computer interface (BCI) operation: Signal and noise during early training sessions Clin. Neurophysiol 1 1 6 56-62 McFarland D J, Sarnacki W A and Wolpaw J R 2003 Brain-computer interface (BCI) operation: Optimizing information transfer rates Biol. Psychol 6 3 237-51 M c M i l l a n G R and Calhoun G L 1995 Direct brain interface utilizing self-regulation of steady-state visual evoked response (SSVER) Proc. RESNA '95 Annual Conference (Vancouver, BC) pp 693-5 Mensh B D , Werfel J and Seung H S 2004 B C I competition 2003-data set ia: Combining gamma-band power with slow cortical potentials to improve single-trial classification of electroencephalographic signals IEEE Trans. Biomed. Eng 51 1052-6 80 Middendorf M , M c M i l l a n G , Calhoun G and Jones K S 2000 Brain-computer interfaces based on the steady-state visual-evoked response IEEE Trans Rehabil Eng 8 211-4 Mi l l an Jdel R, Renkens F, Mourino J and Gerstner W 2004a Noninvasive brain-actuated control of a mobile robot by human E E G IEEE Trans. Biomed. Eng 51 1026-33 Mi l l an J R, Renkens F, Mourino J and Gerstner W 2003a Non-invasive brain-actuated control of a mobile robot Proc. 18th International Joint Conference on Artificial Intelligence (Acapulco, Mexico) pp 1121-6 Mi l l an J R, Renkens F, Mourino J and Gerstner W 2004b Brain-actuated interaction Artificial Intelligence 159 241-59. Mi l l an J, Franze M , Mourino J, Cincotti F and Babiloni F 2002a Relevant E E G features for the classification of spontaneous motor-related tasks Biol. Cybern 86 89-95 Mi l l an J, Mourino J, Franze M , Cincotti F, Varsta M , Heikkonen J and Babiloni F 2002b A local neural classifier for the recognition of E E G patterns associated to mental tasks IEEE Trans. Neural Net 13 678-86 Mi l l an J R 2004 On the need for on-line learning in brain-computer interfaces Proc. Annual Inter. Joint Conf. on Neural Networks (Budapest, Hungary) Mil l an J R, Mourino J, Babiloni F, Cincotti F, Varsta M and Heikkonen J 2000a Local neural classifier for EEG-based recognition of mental tasks Proc. IEEE-INNS-ENNS Inter. Joint Conf on Neural Networks (Como, Italy) pp 632-6 Mi l l an J R, Mourino J, Cincotti F, Varsta M , Heikkonen J, Topani F, Marciani M G , Kaski K and Babiloni F 2000b Neural networks for robust classification of mental tasks Proc. 22nd Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Chicago, IL) pp 1380-2 M i l l a n J R, Mourino J, Marciani M G , Babiloni F, Topani F, Canale I, Heikkonen J and Kaski K 1998 Adaptive brain interfaces for physically-disabled people Proc. 20th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Hong Kong) pp 2008-11 81 M i l l a n J R and Mourino J 2003b Asynchronous B C I and local neural classifiers: A n overview of the adaptive brain interface project IEEE Trans. Neural Syst. Rehabil. Eng 11 159-61 Mol ina G G , Ebrahimi T and Vesin J M 2003 Joint time-frequency-space classification of E E G in a brain-computer interface application EURASIP J. Applied Signal Processing 7 713-29 Mourino J, Chiappa S, Jane R and Mi l l an J R 2002 Evolution of the mental states operating a brain-computer interface Proc. of the Inter. Federation for Medical and Biological Engineering (Vienna, Austria) pp 600-1 Muller K R, Anderson C W and Birch G E 2003a Linear and nonlinear methods for brain-computer interfaces IEEE Trans. Neural Syst. Rehabil. Eng 11 165-9 Muller K R, Curio G , Blankertz B and Dornhege G 2003b Combining features for B C I Advances in Neural Information Processing Systems (NIPS02), vol . 15, eds S Becker, S Thrun and K Obermayer, pp 1115-22. Muller G R, Neuper C and Pfurtscheller G 2003c Implementation of a telemonitoring system for the control of an EEG-based brain-computer interface IEEE Trans. Neural Syst. Rehabil. Eng 11 54-9 Muller-Putz G R, Scherer R, Brauneis C and Pfurtscheller G 2005a Steady-state visual evoked potential (SSVEP)-based communication: Impact of harmonic frequency components J neural engineering 2 123-30 Muller-Putz G R, Scherer R, Pfurtscheller G and Rupp R 2005b EEG-based neuroprosthesis control: A step towards clinical practice Neurosci. Lett 382 169-74 Musallam S, Cornell B D , Greger B , Scherberger H and Andersen R A 2004 Cognitive control signals for neural prosthetics Science 305 258-62 Huan N J and Palaniappan R 2005 Classification of mental tasks using fixed and adaptive autoregressive models of E E G signals Proc. 27 th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Shanghai, China) pp 633-6 82 Neumann N , Hinterberger T, Kaiser J, Leins U , Birbaumer N and Kubler A 2004 Automatic processing of selfrregulation of slow cortical potentials: Evidence from brain-computer communication in paralysed patients Electroencephalogr Clin Neurophysiol 1 1 5 628-35 Neumann N , Kubler A , Kaiser J, Hinterberger T and Birbaumer N 2003 Conscious perception of brain states: Mental strategies for brain-computer communication Neuropsychologia 4 1 1028-36 Neuper C, Muller G R, Kubler A , Birbaumer N and Pfurtscheller G 2003 Clinical application of an EEG-based brain-computer interface: A case study in a patient with severe motor impairment Clin Neurophysiol 1 1 4 399-409 Neuper C, Scherer R, Reiner M and Pfurtscheller G 2005 Imagery of motor actions: Differential effects of kinesthetic and visual-motor mode of imagery in single-trial E E G Brain Res. Cogn. Brain Res 2 5 668-77 Neuper C, Schloegl A and Pfurtscheller G 1999 Enhancement of left-right sensorimotor E E G differences during feedback-regulated motor imagery J. of Clin Neurophysiol 16 373-82 Niedermeyer E and Lopes da Silva F 1998 Electroencephalography: Basic Principles, Clinical Applications, and Related Fields Fourth Edition ed C Mitchell (Baltimore, Maryland U S A : Lippincott Will iams & Wilkins) Obeid I and W o l f P D 2004 Evaluation of spike-detection algorithms for a brain-machine interface application IEEE Trans. Biomed. Eng 51 905-11 Obermaier B , Guger C, Neuper C and Pfurtscheller G 2001a Hidden markov models for online classification of single trial E E G data Pattern Recognition Letters 22 1299-309 Obermaier B , Muller G R and Pfurtscheller G 2001b "Virtual keyboard" controlled by spontaneous E E G activity IEEE Trans. Neural Syst. Rehabil. Eng 11 636-41 Obermaier B , Munteanu C, Rosa A and Pfurtscheller G 2001c Asymmetric hemisphere modeling in an offline brain-computer interface IEEE Trans Syst Man Cybern 3 1 536-40 Obermaier B , Neuper C, Guger C and Pfurtscheller G 2001 d Information transfer rate in a five-Classes brain-computer interface IEEE Trans Neural Syst. Rehabil. Eng 9 283-8 83 Olson B P, Si J, H u J and He J 2005 Closed-loop cortical control of direction using support vector machines IEEE Trans. Neural Syst. Rehabil. Eng 13 72-80 Otto K J, Vetter R J, Marzullo T C and Kipke D R 2003 Brain-machine interfaces in rat motor cortex: Implications of adaptive decoding algorithms Proc. IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 100-3 Palaniappan R 2005 Brain computer interface design using band powers extracted during mental tasks Proc. IEEE-EMBS Conf. on Neural Engineering (Arlington, VA) pp 321-324 Palaniappan R, Paramesran R, Nishida S and Saiwaki N 2002 A new brain-computer interface design using fuzzy A R T M A P IEEE Trans Neural Syst Rehabil Eng 10 140-8 Panuccio A , Bicego M and Murino V 2002 A hidden markov model-based approach to sequential data clustering Proc. Joint IAPR Inter. Workshop on Structural, Syntactic, and Statistical Pattern Recognition (Windsor, Canada) pp 734-42 Parra L , Alv ino C, Tang A , Pearlmutter B , Yeung N , Osman A and Sajda P 2002 Linear spatial integration for single-trial detection in encephalography Neuroimage 1 7 223-30 Parra L , Alv ino C, Tang A , Pearlmutter B , Yeung N , Osman A and Sajda P 2003a Single-trial detection in E E G and M E G : Keeping it linear Neurocomputing 52-54 177-83 Parra L C, Spence C D , Gerson A D and Sajda P 2003b Response error correction—a demonstration of improved human-machine performance using real-time E E G monitoring IEEE Trans. Neural Syst. Rehabil. Eng 11 173-7 Patil P G , Carmena J M , Nicolelis M A and Turner D A 2004 Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain-machine interface Neurosurgery 55 27-35 Penny W D , Roberts S J, Curran E A and Stokes M J 2000 EEG-based communication: A pattern recognition approach IEEE Trans. Rehabil. Eng 8 214-5 Peters B O, Pfurtscheller G and Flyvbjerg H 2001 Automatic differentiation of multichannel E E G signals IEEE Trans. Biomed. Eng 48 111-6 84 Peterson D A , Knight J N , Ki rby M J, Anderson C W and Thaut M H 2005 Feature selection and blind source separation in an EEG-based brain-computer interface EURASIP J. Appl. Signal Processing 19 3128-40 Pfurtscheller G and Aranibar A 1977 Event-related cortical desynchronization detected by power measurements of scalp E E G Electroencephalogr. Clin. Neurophysiol 42 817-26 Pfurtscheller G , Flotzinger D and Kalcher J 1993 Brain-computer interface - a new communication device for handicapped persons J Microcomputer Applications 16 293-9 Pfurtscheller G , Flotzinger D and Neuper C 1994 Differentiation between finger, toe and tongue movement in man based on 40-hz E E G Electroencephalogr Clin Neurophysiol 90 456-60 Pfurtscheller G and Guger C 1999 Brain-computer communication system: EEG-based control of hand orthesis in a tetraplegic patient Acta Chir. Austriaca 31 23-5 Pfurtscheller G , Kalcher J, Neuper C, Flotzinger D and Pregenzer M 1996 On-line E E G classification during externally-paced hand movements using a neural network-based classifier Electroencephalogr Clin Neurophysiol 99 416-25 Pfurtscheller G and Lopes da Silva F H 1999 Event-related E E G / M E G synchronization and desynchronization: Basic principles Clin. Neurophysiol 110 1842-57 Pfurtscheller G , Muller G R, Pfurtscheller J, Gerner H J and Rupp R 2003a Thought ' -control of functional electrical stimulation to restore hand grasp in a patient with tetraplegia Neurosci. Lett 351 33-6 Pfurtscheller G , Muller-Putz G R, Pfurtscheller J and Rupp R 2005 EEG-based asynchronous B C I controls functional electrical stimulation in a tetraplegic patient EURASIP J. Appl. Signal Processing 19 3152-5 Pfurtscheller G and Neuper C 2001 Motor imagery and direct brain-computer communication Proc. IEEE 89 1123-34 Pfurtscheller G , Neuper C, Flotzinger D and Pregenzer M 1997 EEG-based discrimination between imagination of right and left hand movement Electroencephalogr Clin Neurophysiol 103 642-51 85 Pfurtscheller G , Neuper C, Guger C, Harkam W , Ramoser H , Schlogl A , Obermaier B and Pregenzer M 2000 Current trends in graz brain-computer interface (BCI) research IEEE Trans. Rehabil. Eng 8 216-9 Pfurtscheller G , Neuper C, Muller G R, Obermaier B , Krausz G , Schlogl A , Scherer R, Graimann B , Keinrath C, Skliris D , Wortz M , Supp G and Schrank C 2003b Graz-BCI: State of the art and clinical applications IEEE Trans Neural Syst. Rehabil. Eng 11 177-80 Pfurtscheller G , Neuper C, Schlogl A and Lugger K 1998 Separability of E E G signals recorded during right and left motor imagery using adaptive autoregressive parameters IEEE Trans. Rehabil. Eng 6 316-25 Pineda J A , Al l i son B Z and Vankov A 2000 The effects of self-movement, observation, and imagination on mu rhythms and readiness potentials (RP's): Toward a brain-computer interface (BCI) IEEE Trans Rehabil Eng 8 219-22 Pineda J A , Silverman D S, Vankov A and Hestenes J 2003 Learning to control brain rhythms: Making a brain-computer interface possible IEEE Trans. Neural Syst. Rehabil. Eng 11 181-4 Piatt J 1991 A resource-allocating network for function interpolation Neural Comput 3 213-25 Polak M and Kostov A 1997 Development of brain-computer interface: Preliminary results Proc. 19 t h Annual Int. Conf. of the I E E E Engineering in Medicine and Biology Society (Chicago, IL)pp 1543-6 Polak M and Kostov A 1998 Feature extraction in development of brain-computer interface: A case study Proc. 20th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Hong Kong) pp 2058-61 Polak M and Kostov A 1999 Training setup for control of neural prosthesis using brain-computer interface Proc. 21 s t Annual Int. Conf. of the I E E E Engineering in Medicine and Biology Society & Annual Fal l meeting of the Biomedical Engineering Society (Atlanta, G A ) p446 Polikoff J B , Bunnell H T and Borkowski W 1995 Toward a P300-based computer interface Proc. RESNA '95 Annual Conf. (Vancouver, Canada) pp 178-80 86 Pregenzer M and Pfurtscheller G 1995 Distinction sensitive learning vector quantization ( D S L V Q ) - application as a classifier based feature selection method for a brain computer interface Proc. 4th Int. Conf. on Artificial Neural Networks (Cambridge, UK) pp 433-6 Pregenzer M and Pfurtscheller G 1999 Frequency component selection for an EEG-based brain to computer interface IEEE Trans. Rehabil. Eng 7 413-9 Qin L , Deng J, Ding L and He B 2004a Motor imagery classification by means of source analysis methods Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 4356-8 Qin L , Ding L and He B 2004b Motor imagery classification by means of source analysis for brain-computer interface applications J. Neural Eng 1 135-41 Qin L and He B 2005 A wavelet-based time-frequency analysis approach for classification of motor imagery for brain-computer interface applications J. Neural Eng 2 65-72 Qin L , Kamousi B , L i u Z M , Ding L and He B 2005 Classification of motor imagery tasks by means of time-frequency-spatial analysis for brain-computer interface applications Proc. IEEE-EMBS Conf. on Neural Engineering (Arlington, VA) pp 374-6 Ramachandran V S and Histein W 1998 The perception of phantom limbs Brain 121 1603-30 Ramoser H , Muller-Gerking J and Pfurtscheller G 2000 Optimal spatial filtering of single trial E E G during imagined hand movement IEEE Trans. Rehabil. Eng 8 441-6 Rao Y N , K i m S -, Sanchez J C, Erdogmus D , Principe J C, Carmena J M , Lebedev M A and Nicolelis M A 2005 Learning mappings in brain machine interfaces with echo state networks Proc. IEEE Inter. Conf. on Acoustics, Speech and Signal Processing (Philadelphia, PA) pp v/233-v/236 Reger B D , Fleming K M , Sanguineti V , Alford S and Mussa-Ivaldi F A 2000a Connecting brains to robots: The development of a hybrid system for the study of learning in neural tissues Proc. Vllth Inter. Conf. on Artificial Life (Portland, OR) pp 263-72 Reger B D , Fleming K M , Sanguineti V , Alford S and Mussa-Ivaldi F A 2000b Connecting brains to robots: A n artificial body for studying the computational properties of neural tissues ArtifLife 6 307-24 87 Rezek I, Roberts S and Sykacek P 2003 Ensemble coupled hidden markov models for joint characterization of dynamic signals Proc. 9th Int. Workshop on Artificial Intelligence and Statistics (Key West, Florida) ed C M Bishop and B J Frey Roberts S J and Penny W 2003 Real-time brain-computer interfacing: A preliminary study using bayesian learning Med Biol Eng Comput 38 56-61 Roberts S J, Penny W and Rezek I 1999 Temporal and spatial complexity measures for electroencephalogram based brain-computer interfacing Med Biol Eng Comput 37 93-8 Roushe P, Clement R, Visser C, Cruz J and Si J 2003 "Neurabotics": A n integration of neurons, rats and robots for advanced studies of brain-computer interface systems Proc. IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 462-465 Sanchez J C, Carmena J M , Lebedev M A , Nicolelis M A , Harris J G and Principe J C 2004 Ascertaining the importance of neurons to develop better brain-machine interfaces IEEE Trans. Biomed. Eng 51 943-53 Sanchez J C, Erdogmus D , Rao Y N , Principe J and Nicolelis M A 2002a A comparison between nonlinear mappings and linear state estimation to model the relation from motor cortical neuronal firing to hand movements Proc. SAB Workshop on Motor Control in Humans and Robots (Edinburgh, Scotland) pp 59-65 Sanchez J C, Erdogmus D , Rao Y N , Principe J, Nicolelis M A and Wessberg J 2003 Learning the contributions of the motor, premotor, and posterior parietal cortices for hand trajectory reconstruction in a brain machine interface Proc. IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 59-62 Sanchez J C, K i m S P, Erdogmus D , Rao Y N , Principe J, Wessing J and Nicolelis M A 2002b Input-output mapping performance of linear and nonlinear models for estimating hand trajectories from cortical neuronal firing patterns Proc. IEEE Workshop on Neural Networks and Signal Processing (Martigny, Switzerland) pp 139-48. Schalk G , Wolpaw J R, McFarland D J and Pfurtscheller G 2000 EEG-based communication: Presence of an error potential Clin. Neurophysiol 111 2138-44 88 Scherer R, Muller G R, Neuper C, Graimann B and Pfurtscheller G 2004 A n asynchronously controlled EEG-based virtual keyboard: Improvement of the spelling rate IEEE Trans. Biomed. Eng 5 1 979-84 Schloegl A , Lugger K and Pfurtscheller G 1997a Using adaptive autoregressive parameters for a brain-computer-interface experiment Proc. 19th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Chicago, IL) pp 1533-5 Schloegl A , Neuper C and Pfurtscheller G 1997b Subject specific E E G patterns during motor imaginary Proc. 19th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Chicago, IL) pp 1530-2 Schlogl A , Keinrath C, Scherer R and Pfurtscheller G 2003 Information transfer of an E E G -based brain computer interface Proc. IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 641-4 Schroder M , La i T N , Hinterberger T, Bogdan M , H i l l N J, Birbaumer N , Rosenstiel W and Scholkopf B 2005 Robust E E G channel selection across subjects for brain-computer interfaces EURASIP J. Appl. Signal Processing 1 9 3103-12 Schroder M , Bogdan M , Rosenstiel W , Hinterberger T and Birbaumer N 2003 Automated E E G feature selection for brain computer interfaces Proc. IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 626-9 Schwartz A B , Taylor D M and Tillery S I 2001 Extraction algorithms for cortical control of arm prosthetics Curr Opin Neurobiol 11 701-7 Serby H , Yom-Tov E and Inbar G F 2005 A n improved P300-based brain-computer interface IEEE Trans. Neural Syst. Rehabil. Eng 13 89-98 Serruya M , Hatsopoulos N , Fellows M , Paninski L and Donoghue J 2003 Robustness of neuroprosthetic decoding algorithms Biol cybern 88 219-28 Serruya M D, Hatsopoulos N G , Paninski L , Fellows M R and Donoghue J P 2002 Instant neural control of a movement signal Nature 4 1 6 141-2 89 Sheikh H , McFarland D J, Sarnacki W A and Wolpaw J R 2003 Electroencephalographic(EEG)-based communication: E E G control versus system performance in humans Neurosci. Lett 3 4 5 89-92 Stone M 1977 Asymptotics for and against cross-validation Biometrika 6 4 29-35 Su Ryu C, Song Y , Y o o D S, Choi S, Moon S S and Sohn J H 1999 EEG-based discrimination between yes and no Proc. 21 s t Annual Int. Conf. of the I E E E Engineering in Medicine and Biology Society & Annual Fal l meeting of the Biomedical Engineering Society (Atlanta, G A ) p444 Sutter E E 1992 The brain response interface: Communication through visually-induced electrical brain responses J Micro CompApp 15 31-45 Sykacek P, Roberts S, Stokes M , Curran E , Gibbs M and Pickup L 2003 Probabilistic methods in B C I research IEEE Trans Neural Syst Rehabil Eng 11 192-5 Sykacek P, Roberts S J and Stokes M 2004 Adaptive B C I based on variational bayesian kalman filtering: A n empirical evaluation IEEE Trans. Biomed. Eng 5 1 719-27 Tavakolian K , Nasrabadi A M and Rezaei S 2004 Selecting better E E G channels for classification of mental tasks Proc. IEEE Inter. Symp. on Circuits and Systems pp 537-40 Taylor D M , Tillery S I and Schwartz A B 2002 Direct cortical control of 3D neuroprosthetic devices Science 2 9 6 1829-32 Taylor D M , Til lery S I H and Schwartz A B 2003 Information conveyed through brain-control: Cursor versus robot IEEE Trans Neural Syst Rehabil Eng 11 195-9 Thulasidas M , Guan C, Ranganatha S, W u J K , Zhu X and X u W 2004 Effect of ocular artifact removal in brain computer interface accuracy Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 4385-8 Townsend G , Graimann B and Pfurtscheller G 2004 Continuous E E G classification during motor imagery—simulation of an asynchronous B C I IEEE Trans. Neural Syst. Rehabil. Eng 12 258-65 90 Trejo L J, Wheeler K R, Jorgensen C C, Rosipal R, Clanton S T, Matthews B , Hibbs A D , Matthews R and Krupka M 2003 Multimodal neuroelectric interface development IEEE Trans Neural Syst. Rehabil. Eng 11 199-204 Varsta M , Heikkonen J, M i l l a n J d R and Mourino J 2000 Evaluating the performance of three feature sets for brain-computer interfaces with an early stopping M L P commitee Inter. Conf. on Pattern Recognition (Barcelona, Spain) pp 907-10 Vetter R J, Otto K J, Marzullo T C and Kipke D R 2003 Brain machine interfaces in rat motor cortex: Neuronal operant conditioning to perform a sensory detection task Proc. IEEE-EMBS Conf. on Neural Engineering (Capri Island, Italy) pp 637-40 Vida l J J 1977 Real-time detection of brain events in E E G Proc. IEEE 6 5 633-41 Wang D , Kochiyama T, L u S and W u J 2005a Measurement and analysis of electroencephalogram (EEG) using directional visual stimuli for brain computer interface Proc. Inter. Conf. on Active Media Technology (Takamatsu, Japan) pp 34-9 Wang T, Deng J and He B 2004a Classifying EEG-based motor imagery tasks by means of time-frequency synthesized spatial patterns Clin. Neurophysiol 1 1 5 2744-53 Wang T, Deng J and He B 2004b Classification of motor imagery E E G patterns and their topographic representation Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 4359-62 Wang Y , Wang R, Gao X and Gao S 2005b Brain-computer interface based on the high-frequency steady-state visual evoked potential Proc. 1st Inter. Conf. on Neural Interface and Control (Wuhan, China) pp 37-9 Wang Y , Zhang Z , Gao X and Gao S 2004c Lead selection for SSVEP-based brain-computer interface Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (San Francisco, CA) pp 4507-10 Wang Y , Zhang Z , L i Y , Gao X , Gao S and Yang F 2004d B C I competition 2003-data set IV: A n algorithm based on C S S D and F D A for classifying single-trial E E G IEEE Trans. Biomed. Eng 51 1081-6 91 Weiskopf N , Mathiak K , Bock S W , Schamowski F, Veit R, Grodd W , Goebel R and Birbaumer N 2004 Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI) IEEE Trans. Biomed. Eng 5 1 966-70 Weiskopf N , Veit R, Erb M , Mathiak K , Grodd W , Goebel R and Birbaumer N 2003 Physiological self-regulation of regional brain activity using real-time functional magnetic resonance imaging (fMRI): Methodology and exemplary data Neuroimage 1 9 577-86 Weitkunat R 1991 Digital Biosignal Processing (New York: Elsevier Science) Wessberg J, Stambaugh C R, Kral ik J D , Beck P D , Laubach M , Chapin J K , K i m J, Biggs S J, Srinivasan M A and Nicolelis M A 2000 Real-time prediction of hand trajectory by ensembles of cortical neurons in primates Nature 4 0 8 361-5 Wolpaw J R, Birbaumer N , McFarland D J, Pfurtscheller G and Vaughan T M 2002 Brain-computer interfaces for communication and control Clin. Neurophysiol 1 1 3 767-91 Wolpaw J R, Flotzinger D , Pfurtscheller G and McFarland D J 1997 Timing of EEG-based cursor control J. Clin. Neurophysiol 14 529-38 Wolpaw J R, McFarland D , Neat G W and Forneris C A 1991 A n EEG-based brain-computer interface for cursor control Electroencephalogr Clin Neurophysiol 7 8 252-9 Wolpaw J R and McFarland D J 1994 Multichannel EEG-based brain-computer communication Electroencephalogr Clin Neurophysiol 9 0 444-9 Wolpaw J R and McFarland D J 2004 Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans Proc. Natl. Acad. Sci. U. S. A. 101 17849-54 Wolpaw J R, McFarland D J and Vaughan T M 2000 Brain-computer interface research at the wadsworth center IEEE Trans. Rehabil. Eng 8 222-6 Wolpaw J R, McFarland D J, Vaughan T M and Schalk G 2003 The wadsworth center brain-computer interface (BCI) research and revelopment program IEEE Trans Neural Syst. Rehabil. Eng 11 204-7 W u W , Black M J, Gao Y , Bienenstock E , Serruya M D and Donoghue J P 2002a Inferring hand motion from multi-cell recordings in motor cortex using a kalman filter Proc. SAB Workshop on Motor Control in Humans and Robots (Edinburgh, Scotland) pp 66-73 92 W u W , Black M J, Gao Y , Bienenstock E , Shaikhounix A and Donoghue J P 2002b Neural decoding of cursor motion using a kalman filter Advances in Neural Information Processing Systems 1 5 133-40 W u R C, Liang S F, L i n C T and Hsu C F 2004 Applications of event-related-potential-based brain computer interface to intelligent transportation systems Proc. IEEE Inter. Conf. on Networking, Sensing and Control (Taipei, Taiwan) pp 813-8 X u N , Gao X , Hong B , Miao X , Gao S and Yang F 2004a B C I competition 2003-data set l ib : Enhancing P300 wave detection using ICA-based subspace projections for B C I applications IEEE Trans. Biomed. Eng 51 1067-72 X u W , Guan C, Siong C E , Ranganatha S, Thulasidas M and W u J 2004b High accuracy classification of E E G signal Proc. 17th Inter. Conf. on Pattern Recognition (Cambridge, UK) pp 391-4 Y a n Wang, Sutherland M T, Sanfratello L L and Tang A C 2004 Single-trial classification of E R P S using second-order blind identification (SOBI) Proc. IEEE Inter. Conf. on Machine Learning and Cybernetics (Shanghai, China) pp 4246-51 Yom-Tov E and Inbar G F 2001 Selection of relevant features for classification of movements from single movement-related potentials using a genetic algorithm Proc. 23rd Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Istanbul, Turkey) pp 1364-6 Yom-Tov E and Inbar G F 2002 Feature selection for the classification o f movements from single movement-related potentials IEEE Trans Neural Syst. Rehabil. Eng 10 170-7 Yom-Tov E and Inbar G F 2003 Detection of movement-related potentials from the electro-encephalogram for possible use in a brain-computer interface Med Biol Eng Comput 4 1 85-93 Y o o S S, Fairneny T, Chen N K , Choo S E, Panych L P, Park H , Lee S Y and Jolesz F A 2004 Brain-computer interface using fMRI : Spatial navigation by thoughts Neuroreport 15 1591-5 Y o o n H , Yang K and Shahabi C 2005 Feature subset selection and feature ranking for multivariate time series IEEE Trans, on Knowledge and Data Engineering 1 7 1186-98 93 Y u Z , Mason S G and Birch G E 2002 Enhancing the performance of the L F - A S D brain-computer interface Proc. 24th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (Houston, TX) pp 2443-4 2.8. Appendix A. Index of Terms T A B L E 2.9 INDEX OF TERMS Index term Description AAR Adaptive auto-regressive AEP Auditory evoked potential AGR Adaptive Gaussian representation ALN Adaptive logic network ANC Activity of neural cells ANN Artificial neural networks AR Auto-regressive ARTMAP Adaptive resonance theory MAP ARX Autoregressive with exogenous input BPF Band-pass filter C4.5 -CAR Common average referencing CBR Changes in brain rhythms C C T M Cross-Correlation based template matching CER Coarse-grained entropy rate C H M M Coupled hidden markov model CN2 -CSP Common Spatial Patterns CSSD Common spatial subspace decomposition CSSP Common spatio-spectral patterns CTFR Correlative time-frequency representation CTFSR Correlative time-frequency-space representation DFT Discrete Fourier transform DSLVQ Distinctive sensitive learning vector quantization ERD Event related desynchronization ERN Event related negativity ERS Event related synchronization FLD Fisher's linear discriminat FFT Fast Fourier transform Freq-Norm Frequency normalization GA Genetic algorithm G A M Generalized additive models GLA Generalized linear models GPER Gaussian process entropy rates H M M Hidden Markov Model ICA Independent component analysis IFFT Inverse fast Fourier transform k-NN k-nearest neighbour LDA Linear Discriminant Analysis LDS Linear dynamical system L G M Linear Gaussian models implemented by Kalman filter 94 Index term Description LMS Least mean square LPC Linear predictive coding LPF Low pass filter LRP Lateralized readiness potential L V Q Learning vector quantization MD Mahalanobis distance MLP Multi-Layer perceptron neural networks MN Multiple neuro-mechanisms MNF Maximum noise fraction MRA Movement related activity NID3 -NMF Non-negative matrix factorization NN Neural networks OLS1 Orthogonal least square OPM Outlier processing method PCA Principal Component Analysis (a.k.a. Karhounen Loeve Transform) PLV Phase locking values PPM Piecewise Prony method PSD Power spectral density RBF Radial basis function RFE Recursive feature/channel elimination RNN Recurrent neural network SA-UK Successive averaging and/or considering choice of unknown SCP Slow cortical potentials SE spectral entropy SFFS Sequential forward feature selection SL Surface Laplacian SOFNN Self organizing feature neural network SOM Self organizing map SSEP Somatosensory evoked potential SSP Signal space projection SSVEP Steady state visual evoked potential STD Standard Deviation SVD Singular value decomposition SVM Support vector machine SVR Support vector machine regression SWDA Stepwise discriminant analysis TBNN Tree-based neural network TFR Time-frequency representation VEFD Variable epoch frequency decomposition VEP Visual evoked potential WE Wavelet entropy WK Wiener-Khinchine ZDA Z-scale based discriminant analysis 95 Chapter 3 Brain Computer Interface (BCI) Design for Asynchronous Control Applications: Improvements to the LF-ASD Asynchronous Brain Switch 1 0 3.1. Introduction Over the past 15 years, several research groups throughout the world have developed direct brain-computer interface (BCI 1 1 ) devices as possible alternative communication and control solutions for individuals with severe disabilities. For a review of the field, see (Mason and Birch 2003, Nicolelis 2003, Wolpaw et al 2000, Wolpaw et al 2002). B C I technology functions by mapping a user's cortical activity associated with an intentional B C I control paradigm (such as attempted finger movements) directly to application-specific control signals. Thus, control of various devices such as neural prostheses is possible by thinking alone, bypassing traditional interface pathways unusable by individuals with severe disabilities. Several different approaches to the design of B C I technology based on signals from scalp electrodes (i.e. electroencephalograms, E E G ) or implanted electrodes have been reported for various communications and control applications. A l l these systems can be represented by the common functional model presented in Fig . 3.1 (Mason and Birch 2003) (or a variant of this model (Mason et al 2003)). In this model, the User desires to control a Device (via the Device's control hardware, termed the Device Controller) through a series of components. These components can collectively be called a B C I Interface Device (conceptually similar to a keyboard, mouse or joystick). To aid the discussion below, the B C I Interface Device can be divided into two parts. The components between User and Control Interface can be treated as a single component, a B C I Transducer 1 2, which functions in a manner similar to a physical transducer like a dial or switch. The role of the B C I Transducer is to translate the 1 0 A version of this chapter has been published. © 2004 IEEE. Reprinted, with permission, from (Borrisof J., Mason S., Bashashati A., and Birch G. (2004) Brain Computer Interface (BCI) Design for Asynchronous Control Applications: Improvements to the LF-ASD Asynchronous Brain Switch IEEE Trans. Biomed. Eng. 51 985-92) 1 1 Also known as Brain-Machine Interface (BMI), Direct Brain Interface (DBI) or Adaptive Brain Interface (ABI) technology. The term BCI will be used in this paper. 1 2 The term BCI Transducer introduced in (Mason et al 2003) is synonymous to the term BCI Control originally proposed by Mason and Birch (Mason and Birch 2003). The former is chosen because of the functional similarity to physical transducers. 96 User's brain activity into logical (or device-independent) control signals. The role of the Control Interface is to translate the logical control signals into semantically meaningful information or commands. It may do this over time providing feedback through a Control Display. ' x ' '—-jr—1 1 i 1 ' 1 user-reported error Operating Environment I BCI Transducer 1 I BCI Interface Device ' Figure 3.1 Functional model of a BCI System depicting the principle functional components (Mason and Birch 2003, Mason et al 2003). Note the Control Display is optional. There are many B C I Transducer designs presented in the literature. However few have been designed specifically for asynchronous control. The concept of asynchronous control for B C I Systems was introduced in (Mason and Birch 2000). A s a brief review, B C I Systems are designed to operate in the following general control paradigm. From an O F F state, a B C I System is turned O N using some mechanism. (At this point in the development of B C I systems, an attendant is needed to turn all existing B C I Systems O N ) 1 3 . With the system turned O N , the User controls the system for a period of time, and then the system is turned O F F using a command or some other mechanism. This sequence is shown in Fig. 3.2. For an asynchronous B C I system, once the system is O N , the User affects the B C I Transducer output when they want by intentionally changing their brain state. In between periods of this 1 3 Developing an automated switch to turn the system on is recognized as a difficult problem similar to the open microphone problem with speech recognition. Such a switch has to differentiate between all possible innate brain states and the system ON state. In practical terms, the mechanism will probably be implemented in the future as a sequence of commands, where each step in the sequence confirms the User's intent to turn the system ON. 97 Intentional Control (IC) the User is in a N o Control (NC) state 1 4 - they may be idle, daydreaming, thinking about a problem or lunch, or performing some other action, but they are not trying to control the B C I Transducer. This form of intermittent control has been termed asynchronous control (Mason and Birch 2000). To operate in this paradigm, B C I Transducers are designed to respond only when there is intentional User control and maintain an inactive state output during times when the User is in a N C state. The performance of a User operating a two-state asynchronous B C I Transducer can be measured in percentage successful switch activations during the IC state (i.e., True Positive error rate) and percentage false switch activation during the N C state (False Positive error rate). (Note, in most B C I System evaluations reported in the literature, the allowable times for intentional User control are restricted to periods defined by a computer. Thus these evaluations have not tested their B C I technology for general intermittent or asynchronous operating paradigms. Since the User's input is synchronized with the external computer, this type of control has been termed synchronous control (Mason and Birch 2000). In these experimental systems the B C I technology is tested only during intentional User control. The response of the B C I Transducer during the N C state is not tested.) system system ON system OFF OFF _ L • J L L OFF signal intentional User control Figure 3.2 General asynchronous system control sequence. Even though asynchronous (or intermittent) control is the most natural mode of interaction, it has received relatively little attention in the field. A s recognized in (Wolpaw et al 2002), this is an important problem that requires more attention. Only a few B C I Transducers (Birch et a\ 1993, Levine et al 2000, Mason and Birch 2000, Mi l l an and Mourino 2003 Yom-Tov and Inbar 2003) have been specifically designed (and tested) for asynchronous control. Each of 1 4 The IC and NC states have also been referred to as "active" and "idle" states [6]. 98 the proposed transducers produces a two-state discrete output and as such wi l l be referred to as an Asynchronous Brain Switch (ABS) in the remainder of this paper. Recent on-line studies with the Low-Frequency Asynchronous Switch Design ( L F - A S D ) have demonstrated total mean classification accuracies over 96% with spinal cord-injured and able-bodied subjects (Birch et al 2002a,b). Despite these encouraging results, our experience to date indicates that these error rates are too high for individuals with spinal cord injuries (our target population) in most practical asynchronous control applications. For practical applications, one needs to focus on low false positive rates. From our experience, false positive rates above 2% cause excess frustration and distraction in subjects (Birch et al 2002a). This paper presents the results of an off-line study to evaluate four new design implementations of the L F - A S D transducer. The new designs incorporated combinations of E E G energy normalization, feature space dimensionality reduction, and an alternative classification scheme. E E G recordings of attempted finger movements were collected from our target population of individuals with spinal cord injuries (SCI subjects) as well as able-bodied ( A B ) subjects. 3.2. New Asynchronous Brain Switch Designs The four new A B S designs tested in this study were variations of the original L F - A S D transducer design (Mason and Birch 2000). The new designs were named n L F - A S D - ( A , B , C and D) as they were based on a set of normalized-energy, low-frequency features. Fig. 3.3 presents a block diagram with original components shown in white and the new components shown in gray. The four designs use various configurations of the new components and codebook generation methods as summarized in Table 2.1. electrode array Feature Extractor ENT U LF-ASD Feature Generator KLT Feature Translator 1-NN moving debounce Classifier average k Codebook generation scheme Figure 3.3 Components of nLF-ASD transducers with new components shown in gray, where E N T : Energy Normalization Transform, K L T = Karhunen-Loeve Transform, and 1 - N N = 1 -Nearest Neighbor. 99 T A B L E 3.1 CONFIGURATIONS OF THE N E W A S Y N C H R O N O U S B R A I N SWITCH DESIGNS, W H E R E N L F - A S D = N O R M A L I Z E D - L O W - F R E Q U E N C Y A S Y N C H R O N O U S SWITCH DESIGN, ENT = E N E R G Y N O R M A L I Z A T I O N T R A N S F O R M , K L T = K A R H U N E N - L O E V E T R A N S F O R M , F U Z Z Y A R T = F U Z Z Y A D A P T I V E R E S O N A N C E THEORY, LVQ3 = L E A R N I N G V E C T O R QUANTIZATION. Design Name ENT KLT Debounce Codebook Generation NLF-ASD-A X X K-MEANS + LVQ3 NLF-ASD-B X X FUZZY ART+ LVQ3 NLF-ASD-C X X X K-MEANS+ LVQ3 NLF-ASD-D X X X FUZZY ART+LVQ3 3.3. Description of New Components and Methods The Energy Normalization Transform (ENT) component, introduced in ( Y u et al 2002), demonstrated that normalizing input energy results in greater class separation between IC and N C periods of movement and reduces system sensitivity to variations in E E G energy. The output of the transform, y(n), is calculated from the input, x(n), using x{n) y(n) = W N £ xin-s)1 where WN is the size of the input data window. For this work, the optimal was 51 as determined by previous work (Yu et al 2002). The Karhunen-Loeve Transform ( K L T ) component was used to reduce the 6-dimensional feature space produced by the L F - A S D Feature Generator to a 2-dimensional space using the K L T algorithm (Jayant 1984). This component was added to test i f a reduction in feature space could be accomplished without performance degradation. A reduced feature space potentially offers greater insight and hence improved customization of a particular individual's system parameters. The Debounce component was introduced to reduce the number of false switch activations. Observations from previous studies (Birch et al 2002a, Lisogurski and Birch 1998) indicated that switch activations were multiple samples long and false positives occurred in clusters. Statistical analysis of the activation lengths and inter-activation periods, shown as Lj and Pj in Fig. 3.4A, confirmed these initial informal observations. Debouncing the switch output in a 100 manner similar to the debouncing of physical switches was expected to improve false activation rates (but with a cost in decreased re-activation time). The debounce component continuously monitors the output of the moving average block. After an activation is detected (i.e., a change in state from 0 to 1), the output is set to the logical state 1 for one sample then the output is forced to an inactive state 0 for Tdb - 1 samples, where T a D is the debounce time period in samples. In this study various Tdb values were evaluated. Switch output 4 - » Time (samples) 10 15 20 25 30 39 Length of Activation Block (samples - 16 Hz) 700 600 600 400 300 200 100 0 16 32 48 84 80 96 112 128 144 Inter-Activation Period (samples - 16 Hz) Figure 3.4 Distribution of false switch activations. A . Schematic diagram showing the metrics used to characterize false switch activations: Lj = length of the ith switch activation block (i.e. the number of activations that occur in consecutive sample times); and P; = the rth inter-activation period (i.e. the distance from start of the rth activation block to the end of the ith+l block). B. Histogram of switch activation block length (Lj values). C. Histogram of inter-activation period (Pi values). The arrows indicate the primary debounce values used in evaluation of the new A B S designs. Two Codebook Generation Methods were used to generate a codebook for the 1-NN classifier from training data. The first method was that employed by Mason and Birch (Mason and Birch 2000). In this method, the k-means algorithm (Kohonen 1990) with 3 vectors per class was used to generate initial clustering of the two data sets. This was followed by Learning Vector Quantization (LVQ3) (Kohonen 1990) to generate the final codebook. The second method used a self-organizing neural model called fuzzy adaptive resonance theory (fuzzyART) (Carpenter et al 1991) to perform independent data clustering 101 of the two data sets. The fuzzy A R T algorithm provides a data-driven selection of the number of code vectors in the codebook which is an advantage over the original k-means (which uses a static number of code vectors). This neural network accepts analog inputs one at a time and develops an adaptive categorization of the input data based on the user defined degree of similarity of patterns in clusters. Familiar inputs activate their category, whereas unfamiliar inputs trigger either adaptive learning by an existing category, or commitment of a new category. The algorithm has one parameter, the vigilance parameter, which was set to between 0.45 and 0.6 in order to constrain the total number of codebook vectors for both IC states and N C states to be less than nine. The means of the resulting clusters were fed to L V Q 3 to generate the final codebook. 3.4. Evaluation Methods For this study the new designs were evaluated off-line and the performance of the new A B S designs were summarized in receiver operating characteristic (ROC) curves. These results were compared to the performance of the original L F - A S D design. The data used in this evaluation was collected from subjects positioned 150 cm in front o f a computer monitor. E E G was recorded from six bi-polar electrode pairs positioned over the supplementary motor area and the primary motor cortex (defined with reference to the International 10-20 System at F i - F C i , F 2 - F C 2 , F 2 - F C 2 , F C i - C i , F C 2 - C Z , and F C 2 - C 2 ) . Electro-oculographic (EOG) activity was measured as the potential difference between two electrodes, placed to the corner and below the right eye. Ocular artifact was considered present when the difference between the E O G electrodes exceeded ±25 uV. A l l signals were sampled at 128 H z (see (Birch et al 2002a) for details). The subjects used in this study were four subjects with a high-level SCI and four able-bodied subjects. SCI subjects (all males and right handed) were between 33 and 56 years old. A l l SCI subjects had no residual sensation or motor function in the hands (spinal cord injuries between C3-C4 and C5-C6) and no other compounding physical or emotional conditions that may have interfered with the study (e.g. none were ventilator dependent). A B subjects (3 males and 1 female) were all right handed and between 31 and 57 years old. Data was collected from the subjects performing a guided task. A t random intervals (mean 7 seconds), a 2cm white circle was displayed on the subject's monitor for % second, cuing 102 them to attempt a movement. In response to the cue, subjects tried to activate the brain switch by attempting to move their right index finger 1 second after the cue appeared1 5. (The 1 second delay was used to avoid visual evoked potential effects from the cue.) To train the subjects for appropriate response timing, a yellow dot was flashed 1 second after the white circle in order to provide a timing cue. After 5 to 15 minutes of practice, this cue was only provided once every 4-8 attempts as a reference for the subject to self check their timing. Data collected when a yellow dot was displayed was not used for the codebook training phase or performance evaluation of this study. The period between trials was varied; however, subjects attempted an average of 15 switch activations in approximately 2 minutes of recording. For each subject, an average of 10 such recordings was collected every day for 6 days. The first days' recordings were used to compute the codebooks for the subsequent 5 days of evaluation. The ability of subjects to control the A B S designs was evaluated with percentage of correct activations during IC states (true positives, TPs) and percentage of false switch activations during N C states (false positives, FPs). A TP was identified i f the A B S was activated at least once in the window 0.5 seconds before and 1 second after the time of expected movement, a method similar to that employed by others (Levine et al 2000, Yom-Tov and Inbar 2003). Multiple activations in this window were considered as a single activation. FPs were assessed in the periods before the appearance of a white circle and after the end of the activation window. Periods during which ocular artifact occurred were not evaluated. 3.5. Results and Discussion 3.5.1. Debounce Analysis The previous observations and the results of the distribution analysis of switch activations necessitated an investigation into the impact of debounce on the various switch designs. A s an example, the R O C curve for subject SC2 is shown in Fig. 3.5A. Also shown is an expanded view of the R O C curve (Fig. 3.5B) to emphasize the performance characteristics of the switch at FP rates below 8%. (To improve the presentation of the results, only FP rates below 8% wi l l be displayed in further R O C curves.) Table 3.2 shows mean TP rate for all 15Both subject groups used the same neurological mechanism to drive the brain switch: an attempted finger flexion. This resulted in no movement in SCI subjects, and an actual finger flexion in AB subjects. 103 eight subjects processed with the n L F - A S D - A and interpolated for the given FP rate of 2% for eight different values of debounce. Figure 3.5 ROC curves for Subject SC2 using system design nLF-ASD-A. A . Analysis of the 8 different values of debounce (Tab) used in this study with corresponding Areas under the ROC curve to indicate overall system performance. B. An expanded view of A . Only those values of false positives below 8% are shown for clarity. 104 T A B L E 3.2 M E A N TRUE POSITIVE RATES FOR A L L SUBJECTS WITH DIFFERENT D E B O U N C E V A L U E S , WHERE N L F - A S D = N O R M A L I Z E D - L O W - F R E Q U E N C Y A S Y N C H R O N O U S SWITCH DESIGN, STD = S T A N D A R D DEVIATION OF T R U E POSITIVE RATES, S = SECONDS, TDB.= D E B O U N C E PERIOD. NOTE: A L L F A L S E POSITIVE RATES = 2%. Debounce (TDB) in Samples (and Time) none (0 sec) 6 (0.38 s) 8 (0.5 s) 16 (1.0 s) 32 (2.0 s) Design Name mean STD mean STD mean STD mean STD mean STD L F - A S D 0.13 0.09 0.36 0.14 0.44 0.15 0.51 0.15 0.70 0.11 N L F - A S D - A 0.21 0.05 0.54 0.10 0.61 0.10 0.68 0.10 0.83 0.06 N L F - A S D - B 0.21 0.05 0.55 0.09 0.62 0.09 0.68 0.09 0.83 0.05 N L F - A S D - C 0.19 0.05 0.53 0.09 0.61 0.09 0.66 0.09 0.82 0.06 N L F - A S D - D 0.20 0.05 0.53 0.09 0.60 0.09 0.66 0.09 0.81 0.06 A s the debounce period increased, the FP rate decreased for a given TP rate. These results are in line with the application of debounce in other physical transducer designs. A s the length of the debounce period increased the re-activation time of a B C I transducer is impacted. The tradeoff is clear and one needs to consider this for a given application. For the remainder of the results reported here, the debounce period of 16 samples (1 second) was chosen for data presentation because it provides a practical minimum re-activation time while rejecting the majority of the clustered FP activations (see arrow in Fig. 3.4C). 3.5.2. New vs. Old Transducer Designs The performance of the new switch design methods with the previous design ( L F - A S D ) at fixed debounce periods was compared. The results of all five processing methods for subject SC2 are shown in Fig. 3.6. The TP rates at 2% FP for all eight subjects are shown in Table 3.3. The results show that all four new n L F - A S D - X variants performed similarly with no statistically significant differences. N o differences were detected between codebooks generated using k-means or fuzzyART. A s well , no differences were detected with data transformed to 2-dimensional feature vectors. Because performance is not worsened with lower dimensional data, the Feature Extractors with K L T wi l l probably be preferred in future studies because of the benefits of a lower feature space dimensionality, i.e., easier interpretation of feature space characteristics which can facilitate and guide classifier design choices. The fact that no differences were shown with the different schemes suggests alternate approaches need to be explored to achieve performance improvements. Such 105 approaches may include 1) more complex classifiers such as support vector machines ( S V M ) (Burges 1998) or fuzzy A R T M A P (Carpenter 1992); 2) customization of the L F - A S D Feature Generator parameters to individual subjects (which to date has not been done in this or any previous study); and/or 3) the exploration of different E E G features. Figure 3.6 ROC curves for Subject SC2 with debounce set to 16 samples. Analysis of the five different processing methods used in this study. Only those values of false positives below 8% are shown for clarity. A l l four new transducer designs performed better than the previous L F - A S D transducer. This is most likely due to the presence of the E N T block in the processing path which increases the separation between IC and N C feature vectors and stabilizes the scale of input signals to the L F - A S D processor ( Y u et al 2002). The mean R O C curves for all eight subjects are shown in Fig . 3.7. The n L F - A S D - A system was used to compare the new methods with the previous L F - A S D because of its relative computational simplicity and the fact that the previous ( L F - A S D ) codebooks were generated with this method. The n L F - A S D - A system performed significantly better than the previous design, as seen with a mean TP rate increase of 33% for a FP rate of 2% (from 51% to 68%; p < 0.05, 2-way analysis of variance). Performance improvements can also be seen in individual subjects (Table 3.3) when E N T processing was used (for instance with subject S C I , TP rate improved from 45% to 71%), although for three able-bodied subjects (AB2-4) the improvements were minimal. It should 106 also be noted that the TP performance results presented in Table 3.3 represent total system classification accuracies greater than 97%. This is due to the fact that the L F - A S D is continually classifying inputs every 16 t h o f a second and the fact that the majority o f User's time is spent in an N C state which for a 2% FP rate represents an accuracy of 98% during this state alone. False Positives Figure 3.7 ROC curve of the true positive rate means for all 8 subjects (depicted with standard error of the mean error bars) with debounce set to 16 samples. Solid line = LF-ASD. Dashed line = nLF-ASD-A. The difference is significant (p < 0.05, 2-way A N O V A ) . Only those values of false positives below 8% are shown for clarity. T A B L E 3.3 T R U E POSITIVE RATES FOR INDIVIDUAL SUBJECTS. NOTE: A L L F A L S E POSITIVE RATES = 2%. D E B O U N C E V A L U E S OF 16 A R E SHOWN. Subject Design Name SCI SC2 SC3 SC4 AB1 AB2 AB3 AB4 LF -ASD 0.45 0.47 0.46 0.35 0.32 0.73 0.65 0.67 NLF -ASD-A 0.71 0.76 0.47 0.63 0.72 0.81 0.67 0.66 N L F - A S D - B 0.72 0.75 0.50 0.63 0.67 0.79 0.69 0.66 NLF-ASD-C 0.67 0.74 0.49 0.59 0.68 0.78 0.68 0.65 N L F - A S D - D 0.67 0.74 0.50 0.56 0.69 0.79 0.68 0.66 107 3.5.3. Spinal Cord-Injured Subjects vs. Abie-Bodied Subjects The performances of spinal cord-injured subjects versus able-bodied subjects were compared with the original L F - A S D and the new n L F - A S D - A system design (Fig. 3.8). For the new system designs, little or no differences between SCI subjects and A B subjects were detected (Fig. 3.8A, Table 3.3). Interestingly, A B subjects performed better than SCI subjects on the L F - A S D system (Fig. 3.8B), although these results were not statistically significant (p > 0.05, 2-way A N O V A ) . Differences between SCI subjects and A B subjects are readily evident from Table 3.3. For example, three SCI subjects showed major improvements with E N T processing compared to only one A B subject. The differences in performance between SCI subjects and A B subjects using the older L F - A S D may reflect differences in E E G energy levels during attempted motor movements by the SCI subjects compared to A B subjects, although the ensemble average movement related potentials ( M R P ) for these groups of subjects are similar (Birch et al 2002a,b). Because SCI subjects are able to operate the L F -A S D , which is presumed to be activated by M R P s (Mason and Birch 2000), (Birch et al 2002a,b), it seems likely that SCI subjects produce similar M R P s to those of A B subjects. Others have suggested the same (Decety and Biosson 1990, Green et al 1998, Green et al 1999), although there is some disagreement in the field (Levy et al 1990, Levt et al 1991, Streletz et al 1995). The results presented here suggest it is possible that SCI subjects have only differences in relative E E G signal energy levels, an issue mitigated with the use of an energy normalization transform. Thus, the similarity of SCI subjects and A B subjects suggest that the L F - A S D with energy normalization may be a viable B C I for individuals with a disability. 108 Figure 3.8 ROC curve of the mean true positive rates for A B subjects vs. SCI subjects (depicted with standard error of the mean error bars) with debounce set to 16 samples. Solid line = able-bodied subjects. Dashed line = spinal cord-injured subjects. A . System design nLF-ASD-A. B. System design LF-ASD. 3.6. Conclusion In conclusion, the error characteristics of the new asynchronous brain switch designs were significantly better than the L F - A S D design with true positive rate increases of approximately 33% for false positive rates in the range o f 1-2%. The improvement was attributed to the addition of an energy normalization transform (ENT). The most significant improvements with E N T were found in spinal cord-injured subjects, indicating possible energy differences in E E G recorded from SCI and A B subjects. One of the next steps is to evaluate these new designs in an on-line study. 109 The four new n L F - A S D designs had no significant performance differences between them. The main conclusion from this finding is that we can use a two-dimensional feature space without performance degradation. This may improve our ability to understand the feature space and thus select better classifier designs. This finding also suggests that the 1-NN classification algorithm was not sensitive to the codebook generation method. This work has succeeded in decreasing the error rates of our A B S designs. Although this decrease in error rate is encouraging, practical experience indicates that further improvements are needed. Thus, our future work w i l l explore the use of self-learning classification schemes like S V M or fuzzy A R T M A P instead of the 1 - N N classifier. We wi l l also explore customization of the L F - A S D Feature Generator parameters (which have never been adjusted for individual subjects) or develop different feature extraction methods. A n exciting finding of this study, which confirms the findings of (Birch et al 2002a), was that spinal cord-injured subjects can operate these new A B S designs to the same ability as able-bodied subjects. Both populations are achieving promising control accuracies with current technology. Thus, A B subjects using a finger movement are good predictors of the controllability of the n L F - A S D technologies by SCI subjects using an attempted finger movement. This evidence supports our use of able-bodied subjects as proxies for our target population in future development efforts. The results of this study support our belief that A B S technology wi l l dramatically increase the level of independence for individuals with high level spinal cord injuries in the foreseeable future. 3.7. References Birch G E , Bozorgzadeh Z and Mason S G 2002a Initial on-line evaluation of the L F - A S D brain-computer interface with able-bodied and spinal-cord subjects using imagined voluntary motor potentials IEEE Trans. Neural Syst. Rehabil. Eng 10 219. Birch G E , Lawrence P D and Hare R D 1993 Single trial processing of event related potentials using outlier information IEEE Trans Biomed Eng 40 59-73. Birch G E , Mason S G and Borisoff J F 2002b Current Trend in Brain-Computer Interface Research at the Ne i l Squire Foundation IEEE Trans. Neural Syst. Rehabil. Eng 11 123-6. 110 Burges C J C 1998 A Tutorial on Support Vector Machines for Pattern Recognition Data Mining and Knowledge Discovery 2 121-67. Carpenter G A , Grossberg S, Markuzon N , Reynolds J H and Rosen D B 1992 Fuzzy A R T M A P : A neural network architecture for incremental supervised learning of analog multidimensional maps IEEE Trans.on Neural Networks 3 698-713. Carpenter G A , Grossberg S and Rosen D B 1991 Fuzzy A R T : an adaptive resonance algorithm for rapid, stable classification of analog patterns Proc. Inter. Joint Conf. on Neural Networks (Seattle, USA) pp 411-6. Decety J and Boisson D 1990 Effect of brain and spinal cord injuries on motor imagery European Archives of Clinical Neuroscience 240 39-43. Green J B , Sora E , Bia ly Y , Ricamato A and Thatcher R W 1998 Cortical sensorimotor reorganization after spinal cord injury: an electroencephalographic study Neurology 50 111 5-21. Green J B , Sora E , Bia ly Y , Ricamato A and Thatcher R W 1999 Cortical sensorimotor reorganization after paraplegia: an E E G study Ne urology 53 736-43. Jayant N S and N o l l P 1984 Digital Coding of Waveforms (NJ: Prentice Hall) . Kohonen T 1990 The Self-Organizing Map Proc IEEE 78 1464-80. Levine S P, Huggins J E , Bement S L , Kushwaha R, Schuh L A , Rohde M M , Passaro E A , Ross D A , Elisevich K V and Smith R J 2000 A Direct Brain Interface Based on Event-Related Potentials IEEE Trans Reh Eng 8 180-5. Levy W G J, Amassian V E , Schmid U D and Jungreis C 1991 Mapping of motor cortex gyral sites non-invasively by transcranial magnetic stimulation in normal subjects and patients Electroencephalography and Clinical Neurophysiology 43 51-75. Levy W G J, Amassian V E , Traad M and Cadwell J 1990 Focal magnetic coil stimulation reveals motor cortical system reorganized in humans after traumatic quadriplegia Brain Research 510 130-4. I l l Lisogurski D and Birch G E 1998 Identification of finger flexions from continuous E E G as a brain computer interface Proc. 20th Annual Inter. Conf. IEEE Engineering in Medicine and Biology Society (Hong Kong) Mason S G and Birch G E 2000 A brain-controlled switch for asynchronous control applications IEEE Trans. Biomed. Eng 4 7 1297-307. Mason S G and Birch G E 2003 A general framework for brain-computer interface design IEEE Trans. Neural Syst. Rehabil. Eng 11 70-85. Mason S G , Moore M M and Birch G E 2003 Designing Pointing Devices using Brain-Computer Interface Technology Proc. 1st Int. IEEE EMBS Conf. on Neural Engineering (Capri, Italy) pp 26-3. Mi l l an J R and Mourino J 2003 Asynchronous B C I and local neural classifiers: an overview of the Adaptive Brain Interface project IEEE Trans. Neural Syst. Rehabil. Eng 111 59-61. Nicolelis M A 2003 Brain-machine interfaces to restore motor function and probe neural circuits Nat. Rev. Neurosci. 4 417-22. Streletz L J, Belevich J K , Jones S M , Bhushan A , Shah S H and Herbison G J 1995 Transcranial magnetic stimulation: cortical motor maps in acute spinal cord injury Brain Topography 7 245-50. Wolpaw J R, Birbaumer N , Heetderks W J, McFarland D J, Peckham P H , Schalk G , Donchin E , Quatrano L A , Robinson C J and Vaughan T M 2000 Brain-computer interface technology: a review of the first international meeting IEEE Trans. Rehabil. Eng 8 164-73. Wolpaw J R, Birbaumer N , McFarland D J, Pfurtscheller G and Vaughan T M 2002 Brain-computer interfaces for communication and control Clin.Neurophysiol. 1 1 3 767-91. Yom-Tov E and Inbar G F 2003 Detection of movement-related potentials from the electro-encephalogram for possible use in a brain-computer interface Med. Biol. Eng Comput. 4 1 85-93. Y u Z , Mason S G and Birch G E 2002 Enhancing the Performance of the L F - A S D Brain-Computer Interface Proc. IEEE 2nd Joint EMBS-BMES Conf. (Houston, TX) 112 Chapter 4 A Comparative Study on Generating Training-Data for Self-Paced Brain Interfaces 1 6 4.1. Introduction Direct Brain Interface (BI) systems provide an alternative communication and control solution for individuals with severe motor disabilities, bypassing impaired interface pathways. In a B I system, the cortical activity associated with an intentional control command is mapped directly to application-specific control signals. Various devices, such as neural prostheses, can then be controlled by cognitive processes alone. For a review of the field, see (Mason and Birch 2003, Mason et al 2006, Nicolelis 2003, Vaughan et al 2003, Woplaw et al 2002). Many B I transducer designs have been presented in the literature. Few of them, however, have been designed specifically for asynchronous or self-paced control. The concept of self-paced control of B I systems was introduced in (Mason and Birch 2000) as "asynchronous control". In a self-paced BI , the users affect the BI transducer output whenever they want by intentionally changing their brain state. Between periods of intentional control (IC), users are said to be in a no-control (NC) state - they may be idle, daydreaming, thinking about a problem or lunch, or performing any other action, but they are not trying to control the B I transducer. To operate in this paradigm, BI transducers are designed to respond only when there is intentional user control and to remain inactive when the user is in an N C state. In contrast, most B i s operate only during specific periods determined by the system (not the user). The latter operating paradigm is referred to as synchronous or synchronized control (Mason and Birch 2000, Mason and Birch 2005). Although self-paced control is the most natural mode of interaction, it has received relatively little attention. Only a few BI transducers (Birch et al 1993, Borisoff et al 2004, Levine et al 2000, Graimann et al 2004, Mason and Birch 2000, Mi l l an and Mourino 2003, Yom-Tov and Inbar 2003, Scherer et al 2004, Townsend et al 2004) have been specifically designed and tested for self-paced A version of this chapter has been published. © 2007 IEEE. Reprinted, with permission, from (Bashashati A., Mason S.G., Borisoff J., Ward R.K., and Birch G. (2007) A Comparative Study on Generating Training-Data for Self-Paced Brain Interfaces IEEE Trans. Neural Systems and Rehabilitation 15 59-66) 113 control. A s recognized in (Wolpaw et al 2002), it is an important issue that deserves more attention from the research community. Like any control system dependent on pattern recognition or machine learning, before a user can operate the system, the system needs to be trained. System training typically refers to training the classifier component of the system, a process that requires well-defined training-data that includes representative samples of each class of data. For self-paced BI systems, generation of the training-data is especially challenging. A s an example, consider designing a BI system that is activated by the user attempting or imagining a specific movement. Furthermore, consider that most BI systems are designed for use by individuals with severe motor disabilities. Wi th these individuals, there is no observable indicator of their intent. Wi th no indication of intent, the exact time that the person actually imagined or intended to perform the movements is not known. Thus, little or no knowledge is available to label the training-data for the intentional control data class. In contrast, an able-bodied subject can perform the desired movements and one can observe his/her movement as the indicator of intent. Even for able-bodied individuals, when the BI system aims at detecting imagined mental tasks (e.g. imagined movement), no indication of intent is available. In such cases, generating training-data for such a system is also problematic. Wi th the lack of an observable indication of intent, one possibility is to use the time that the intentional control (IC) task such as an imagined movement is expected to be performed. However, since the actual time of IC is not precisely known, the use of the expected time of IC can result in poor training-data and consequently worse system performance. This problem is also observed in other research fields such as the cases when satellite images need to be classified but no knowledge of the ground-truth is available. In such cases, the expert's interpretation of satellite images is used to generate training-data (Chia-Tang et al 2003) or unsupervised signal processing methods are employed to detect specific objects in the image (Kersten et al 2005). Because the signal-to-noise ratio of the brain signal is very low and the E E G patterns associated with different brain states are not visually differentiable in a single trial basis, these approaches are not useful in generating training-data for BI systems. 114 This paper introduces and then evaluates three methods to generate system training-data for self-paced BI systems. These methods rely on the external knowledge of the 'approximate' time of the intended control (IC), i.e. the time of expected attempted movement. In such cases, the exact time of the IC state is not known. Thus the extracted features around the approximate time of IC could belong to the N C or IC class. Comparing the statistical likelihood of mixed N C and IC class features to N C class features, the proposed methods estimate which class each feature belongs to and labels them accordingly in order to tune the various system components such as the classifier. In Section 4.4, the performance of the proposed methods is compared to four other methods of generating training-data. These four methods are simply based on the expected time of the IC. To compare the performance of the proposed methods, the methods are applied to the E E G data of four able-bodied and four spinal-cord-injured (SCI) subjects which were previously recorded in self-paced BI experiments at the N e i l Squire Brain Interface Lab, Vancouver, Canada. We compare the performance of these methods on the classifier of a specific self-paced brain interface, named the L F - A S D , as a representative example of self-paced B i s in the literature (Mason and Birch 2000). 4.2. Data Generation Methods In the following subsections, we introduce the proposed methods for generating training-data for self-paced brain interfaces. These methods are based on an experimental protocol where several attempts at intentional control (IC) are measured assuming the trial structure shown in Fig. 4.1. For each trial, the experimental system provides a timing cue. The subjects are instructed to attempt intentional control in response to this cue 1 7 . The cue determines the "time of the expected IC or T E I C which is when the subjects are instructed to attempt intentional control. A time window that spans T seconds before to T seconds after this time is called the 'expected response window' or E R W . The purpose of the proposed methods is to extract the most l ikely feature (or features) that corresponds to IC, of all those mixed IC and N C features that occurred during the E R W . These extracted IC features are then used as training-data for the B I system. 1 7 Cues can be offset by T C u e from actual TEIC to avoid stimulus-related artifacts. 115 The proposed methods require approximately four minutes of E E G data related to the no control (NC) class. These data can be extracted from the interval times between IC trials (as shown in Fig. 4.1) or separately during any time interval when the subject does not intend issuing control. ERW t=TEIC-Ts t=TEIC+Ts TEIC-Tc Time (s) NC class features Mixed IC and NC| class features NC class features Figure 4.1 Structure of an IC trial, where TEIC: time of the expected IC, ERW: expected response window, IC: intended control, NC: no control, T C u e : time of the cue appearance. 4.2.1. Parzen-Based Training-Data Generation (TR_Genl) The first method, named T R _ G e n l generates the probability density function (PDF) of the known samples which lie outside the E R W . Since these samples are known to be of the N C type, the resulting P D F is denoted by fNc(x).fNc(x) is then used to rank the samples within the E R W such that the feature vectors that have higher probability of belonging to the N C class are ranked lower than those feature vectors that have a low probability of belonging to the N C class. In fact, the higher ranked features would most l ikely belong to the IC class rather than the N C class. Details of the procedure for generating the training-data with T R _ G e n l method is shown in Fig. 4.2. This procedure has two main stages: In the first stage, the P D F of the N C features (/NC(X)) is estimated using the Parzen's probability density estimation method (Cacoullos 1966, Parzen 1962). The Parzen method yields accurate results when a large population of data is available. For the problem at hand, because o f the availability of a large population of N C state features (Cacoullos 1966, Parzen 116 1962) and because of the findings in (Glavinovic 1996) that shows Parzen density estimation outperforms frequency histogram estimation, the estimation of the probability density distribution using the Parzen method should yield more accurate results than those given by other probability density estimation methods. The kernel function and the kernel size are the two important parameters of the Parzen's probability density estimation method. A Gaussian kernel with kernel size of N~k,n as recommended in (Cacoullos 1966) was used, where TV is the number of available data, n is the dimension of the data, and k is chosen as a number between 0 and 1. To obtain k, we used simulated Gaussian distributed data and obtained the best estimated P D F results with k=0.15 and used this value in our method. NC state features Probability density estimation of NC features Mixed class features from each trial Estimating the probability of FV being NC using fNC(x) Best IC feature selection Data base of IC training-data Figure 4.2 IC state training-data generation using T R G e n l method, IC: intended control, NC: no control, FV: feature vector,/NC(X)'- probability density function of NC features. In the second stage, we consider the features extracted from the E R W time duration. The class to which each of these features belongs is not known. These features are processed with the goal of selecting amongst them the best representative features of the IC class. The probability density distribution of the N C feature vectors computed in the previous step is used to estimate the probability of each of these features belonging to the N C class. For each E R W , the top ranked feature (the feature that is least l ikely to belong to the N C class) is selected as an IC class feature. This procedure is repeated for each E R W . The resulting set of IC class features are selected as IC training-data. To select the features of the N C training-data, the features from outside the E R W are considered and the probabilities of these features belonging to N C class are calculated separately for each feature. Then, the top H-75% of N C class features that has the highest probability of being N C are selected as N C class training-data. 117 4.2.2. kMeans-Based Training-Data Generation (TR_Gen2) The second method, called TR_Gen2, separates IC from N C features in the E R W based on their initial clustering. A s this clustering method only separates the features into two classes and does not provide the label of the resulting clusters, the probability density function of the N C class features (/NC(X)) is used to label the resulting clusters as N C or IC class. The same procedure as described in Section 4.2.1 is used to estimate the probability density function (/NC(X)) o f the N C class. A s Fig. 4.3 shows, the E R W features are fed into the k-Means clustering algorithm (Kohonen 1990) using k=2, i.e. two feature clusters. Using /NC(X), each of the two resulting clusters is labeled as an N C or IC class. The method that labels the clusters is as follows: the average of each o f the features o f the two resulting clusters is calculated. \Jsm%fNc(x), the features of the cluster whose average has less probability of being an N C class is considered as the IC class cluster and the other cluster is considered as the N C class cluster. This procedure is repeated for each E R W and all features belonging to all the IC clusters form the set of IC class features to be used as the IC training-data. To generate the N C class training-data, the same procedure as described for the T R G e n l method is used. N C state Probability density estimation of features N C features (fNC(x)) Mixed class features from each trial k-Means clustering Estimating the probability of the average of each cluster being N C usm%fNC(x) Selecting the Data base of IC cluster that — • training-data belongs to IC Figure 4.3 IC state training-data generation using TR_Gen2 method, IC: intended control, NC: no control, /NC(X)'- probability density function of N C features. 4.2.3. Averaged kMeans-Based Training-Data Generation (TR_Gen3) The third method, named TR_Gen3, is also based on the initial kMeans clustering of the E R W features, as explained in Section 4.2.2. After clustering the E R W features and labeling them, the average feature of all IC features in the cluster is calculated and used as the representative IC class feature for that E R W . This procedure is repeated for each E R W 118 resulting in a set of IC class features for the training set. The rationale behind using the average (and not all the features) of the IC cluster is as follows: when all the features in the IC cluster are used, there might be some features that are very close to the N C class but they were considered as IC class features. B y averaging the features of the IC cluster, the effect of such features is hopefully reduced. 4.3. Evaluation The proposed training-data generation schemes were evaluated on data previously recorded for self-paced BI experiments at the Ne i l Squire Society, Vancouver, Canada. The proposed methods were used to train the feature classifier of the L F - A S D brain switch (Mason and Birch 2000). The results are evaluated with standard performance metrics and are presented in Section 4.4. Fig. 4.4 shows the block diagram of the latest L F - A S D design. Using a custom-designed template matching algorithm, the features from each of the six incoming bipolar channels are calculated. Overall, six feature values corresponding to each of the six bipolar channels are generated. Then, the Karhunen-Loeve transform (i.e. Principal Component Analysis) is used to reduce the dimensions of the resultant features from six two. Every 1/16 th o f a second, a 1-nearest neighbor (1-NN) feature classifier classifies each feature to either the IC or N C class. A moving average and a debounce block are then used to further improve the classification accuracy of the system by reducing the number of false switch activations. Specific details on the latest design and implementation of the L F - A S D can be found in (Borisoff et al 2004). To generate the codebooks for the 1 - N N classifier, the k-means algorithm (Kohonen 1990) with three vectors per class state is followed by Learning Vector Quantization ( L V Q 3 ) (Kohonen 1990) to find the final codebook in the feature space. In the L V Q algorithm, the learning rate, a , is set to 15/(length of training data) and s to 0.25. The algorithm is stopped after it runs 2000 times or i f the template does not change significantly according to a predefined threshold. 119 electrode array Feature Extractor ENT LF-ASD Feature |—| KLT Generator Feature Translator 1-NN Classifier moving average debounce Codebook generation mechanism Figure 4.4 Components of the Low Frequency Asynchronous Switch Design (the LF-ASD) (from (Borisoff et al 2004)). ENT: energy normalization transform, K L T : Karhounen-Loeve transform, 1-N N : one nearest neighbour 4.3.1. Experimental Data The subjects consisted of four subjects with a high-level spinal cord injury (SCI) (level of injury between C3-4 and C5-6) and four able-bodied subjects. A l l subjects were male except for one female subject. They were all right handed and between 31 and 57 years old. A l l SCI subjects had no residual sensation or motor function in the hands. A l l the subjects had signed the consent form required by the Behavioral Research Ethics Board ( B R E B ) of the University of British Columbia. The E E G data were collected from subjects positioned 150 cm in front of a computer monitor. The E E G signal was recorded from six bipolar electrode pairs positioned over the supplementary motor area and the primary motor cortex (defined with reference to the International 10-20 System at F1-FC1, Fz -FCz , F2-FC2, F C 1 - C 1 , F C z - C z , and FC2-C2) . Electro-oculographic (EOG) activity was measured as the potential difference between two electrodes, placed at the corner and below the right eye. A n eye-blink artifact was considered present when the difference between the E O G electrodes exceeded ±25 p V . A l l signals were amplified, then filtered by a 0.1 to 30Hz band-pass filter and then sampled at 128Hz by a P C equipped with a 12-bit analog to digital converter. The data were collected from the subjects while performing a guided task, in 2-minute sub-sessions, over a session period of 1.5 hours. The sub-sessions contained both the N C and IC state periods as shown in Fig . 4.1 where T c u e - l s , T=0.5s and T E I C corresponds to time of attempted movement. A t random intervals of 5.6 to 7.0 seconds (mean of 6.7 seconds), a 2cm white circle cue was displayed on the subject's monitor for lA second, prompting them to attempt a movement. In response to this cue, the subject tried to activate the brain switch by 120 attempting to move the right index finger one second (Tc u e ) after the cue appeared. The one-second delay (Tc u e ) was used to avoid the visual evoked potential effects from the activity. The time, one second after the cue, is called "time of expected (attempted) intended control (TEIC)". A s the time to perform the movement attempt depended on a user's response, the movement attempt was not necessarily performed at T E I C and would be different from subject to subject and attempt to attempt. Both subject groups used the same neurological mechanism to drive the brain switch: an attempted right index finger flexion. This resulted in no movement in subjects with high level SCI, and an actual finger flexion in able-bodied subjects. For each subject, an average of 80 trials was collected every day for 6 days. Besides recording the E E G data during movement attempts, in each session, the E E G data were also recorded for several 2-minutes periods while the person was in a specific no control (NC) state. The sub-sessions that only contained N C state E E G data were gathered while subject was in different N C states such as attentive eyes opened while looking at a picture on the monitor, doing a search task, attentive eyes closed etc. The reason this type of data was recorded was to evaluate the performance of the BI design in different N C state periods. 4.3.2 Method of Evaluation The proposed training-data generation schemes were evaluated on all available data, i.e., the sub-sessions that contained mixed N C and IC state data and the sub-sessions that included only N C state periods. Specifically, the sub-sessions that contained only N C state E E G data were used to generate the probability density distribution of N C class features (fNc) as explained in Section 4.2. In this experiment we chose H=75% and T=0.5s (for more details refer to Section 4.2). The ability of the subjects to control the L F - A S D B I switch was evaluated by 1) the percentage of correct activations during E R W s (referred to as true positives, TPs) and 2) the percentage of false switch activations during N C states (false positives, FPs). A TP was identified i f the BI system was activated at least once in a window spanning 0.5 seconds before to 1 second after the time of the expected IC (TEIC). This method is similar to that employed previously (Borisoff et al 2004, Graimann et al 2004, Levine et al 2000, Mason and Birch 2000, Yom-Tov and Inbar 2003). FPs were assessed in the periods before the 121 appearance of a white circle and after the end of the activation window. Periods during which ocular artifacts occurred were blocked by the system and not included in the evaluation. To assess the performance of the L F - A S D system, a method based on 5-fold stratified cross-validation is used. In this approach, the trials are randomly divided into five groups of equal trial numbers. Then the classifier is trained using the data of one of the five groups and the system's performance is evaluated using the data of the remaining groups. A s shown in (Bashashati et al 2006), for this specific BI system, evaluating the performance of the system on one cross-validation set provides an accurate measure of performance without having to repeat this procedure for each of the five cross-validation sets. Details of the evaluation method and the procedure detailing how the trials are randomly picked from continuous E E G are found in (Bashashati et al 2006). The above three proposed methods for generating the training-data were compared to the following four other schemes used to generate the training-data for the IC class: (1) the single feature specifically at T E I C are picked from each E R W (referred to as T R _ T E I C ) , (2) three features around T E I C are picked from each E R W (TR_3TEIC), (3) all the features in E R W are picked from each trial (TR_A11), and (4) one feature is randomly picked in the E R W s ( T R R a n ) . To generate the N C class training-data, the same procedure as described for T R _ G e n l method which selects the top H=75% of features that belong to the N C state is used. The 1-NN classifier of the L F - A S D design was trained seven separate times using training-data generated by each of the seven methods. Receiver operating characteristic (ROC) curves were generated for the B I system associated with each of the training-data generation methods. Then, for each subject, the area under the specific region of interest of the R O C curves ( A R O I - R O C C ) was used to compare the performance of each method. For practical applications, one needs to focus on low FP rates. From our experience, FP rates above 2% cause subjects excess frustration and distraction in subjects (Birch et al 2002). Thus, A R O I R O C C for FPs less than 2% are calculated and used for comparison. Note that we have not measured frustration in a systematic way; our knowledge is based on informal subject interviews conducted at the end of the study. 122 To assess the statistical significance of the results, for each subject, the algorithms were repeated 10 times, each time with a random set of trials as training and test data. 4.4. Results Table 4.1 shows the area under R O C curves ( A R O I _ R O C C ) for FP<2% for the BI system associated with the seven above mentioned training-data generation methods averaged across the 10 runs of the algorithms for each subjects. T A B L E 4.1 A R E A U N D E R ROC C U R V E (FOR FP<2%) A V E R A G E D O V E R 10 RUNS OF THE ALGORITHMS FOR E A C H SUBJECT FOR DIFFERENT TRAINING-DATA GENERATION METHODS Subject Area under specific region of ROC curve (for FP<2%) *104 TR Genl TR Gen2 TR Gen3 TR TEIC TR 3TEIC TR All TR Ran AB1 80.89 72.67 79.08 67.04 62.70 69.63 55.58 AB2 90.64 77.09 91.13 53.13 45.49 61.65 54.12 AB3 70.45 52.12 71.78 59.89 60.75 70.76 71.01 AB4 82.57 66.32 83.58 45.96 60.19 81.39 59.20 sen 52.22 44.36 49.27 48.52 51.14 50.61 49.53 SCI2 76.37 69.16 76.85 56.90 59.76 78.12 53.62 SCI3 34.22 32.47 35.56 39.63 35.74 36.57 43.41 SCI4 79.34 75.85 82.82 53.21 70.66 72.88 56.64 Average 70.84 61.25 71.26 53.04 55.80 65.20 55.39 Table 4.2 shows the results of the paired T-test to examine the hypotheses whether each of T R _ G e n l , TR_Gen2 and TR_Gen3 generate significantly better results than the other alternative methods stated in Section 4.3.2. A s shown in Tables 4.1 and 4.2, the performances of two of the three proposed methods (TR_Genl and TR_Gen3) are better than the four other methods. These two methods generate almost the same performance results. Results o f the significance tests also show that these two methods generated significantly higher A R O I -R O C C than other methods ( p « 0 . 0 0 3 ) . Table 4.3 shows the averaged true positive (TP) rates (across the eight subjects) at fixed false positive (FP) rates of 1% and 2%. Fig. 4.5 also shows the average R O C curves of the eight subjects for each of the seven methods of training-data generation. A s this figure shows, the T R _ G e n l and TR_Gen3 methods generate higher true positives (TP) for each level of false positives (FP). For example, at the FP rate of 1% as shown in Table 4.3, two of the proposed methods increase the TP rates from 27.3-34.2% to 37.1-37.6% which correspond to 2.9-10.3% improvement. A t FP rate of 2%, the proposed 123 methods increase the T P rate from 50.8-58.4% to 61.9-62.2%) which corresponds to 3.5-11.4% improvement. Comparing TR_Gen3 with the best performing alternative method, TR_A11, the TP rates increase 3.4% and 3.5% at FP rates of 1% and 2%, respectively. T A B L E 4.2 O V E R A L L SIGNIFICANCE L E V E L (P-VALUE) OF IMPROVEMENTS A N D P E R C E N T A G E OF IMPROVEMENTS (IN PARENTHESES) OF THE A R E A U N D E R ROC C U R V E (FOR FP<2%) W H E N USING THE PROPOSED TRAINING-DATA GENERATION M E T H O D . ITEMS THAT HAS '- ' INDICATE P E R F O R M A N C E D E C R E A S E A N D THUS NO V A L U E S IS REPORTED. Significance level (p-value) using "paired T-test" and percentage of improvements (in parentheses) TR Genl TR Gen2 TR Gen3 TR TEIC TR 3TEIC TR All TR Ran T R G e n l TR_Gen2 TR Gen3 p>0.08 (0.6%) p«0.003 (15.7%) p«0.003 (16.3%) p«0.003 (33.6%) p<0.003 (15.5%) p«0.003 (34.4%) p«0.003 (27.0%) p<0.08 (9.8%) p«0.003 (27.7%) p<0.003 (8.7%) p<0.003 (9.3%) p«0.003 (27.9%) p<0.08 (10.6%) p«0.003 (28.7%) T A B L E 4.3 A V E R A G E T R U E POSITIVE RATES (ACROSS EIGHT SUBJECTS) A T FIXED F A L S E POSITIVE RATES OF 1% A N D 2% FOR THE TRAINING-DATA GENERATION METHODS Training-data generation T R G e n l TR_Gen2 TR_Gen3 T R T E I C TR_3TEIC TR_A11 T R R a n method Average TP rate (%) at fixed 37.1 31.9 37.6 27.3 28.0 34.2 28.2 FP=1% Average TP rate (%) at fixed 62.2 53.6 61.9 50.8 53.7 58.4 52.3 FP=2% 124 False Positive Rate (%) Figure 4.5 Average ROC curves (for false positives less than 2%, as we are interested in false positives less than 2%) across eight subjects for the seven training-data generation methods ( T R G e n l (dashed line with rectangles), TR_Gen2 (dashed line with plus signs), TR_Gen3 (solid line with circles), TR_TEIC (solid line with stars), TR_3TEIC (solid line with lozenges), TR_A11 (solid line with crosses), TR_Ran (dashed line with triangles)) 4.5. Discussion Methods that generate training-data for self-paced brain interfaces are proposed and evaluated. Experimental results from this study on eight subjects show that the training-data generated by two of the proposed methods yield 9.3-34.4% larger A R O I - R O C C than alternative methods which use time of the expected intended control (TEIC) as a time reference to generate the training-data. Comparing the TP rates, the proposed methods improve the average performance from 50.8%-58.4% to about 62% at FP rate of 2%. 125 A s shown in Table 4.1, the two of three proposed methods resulted in significantly better performances than the other alternative methods of training-data generation for all subjects except for SCI3. For this specific subject, our three proposed methods yielded slightly less performance, although the performance was not significantly different than the four alternate methods. Based on the results shown in Table 4.1, TR_Gen2 generated better results than the T R T E I C , T R _ 3 T E I C , TR_Ran. methods. However, this method did not perform as well as T R G e n l and TR_Gen3 methods. A s mentioned before, the TR_Gen3 method uses the average of the features in the IC cluster as the representative feature of the IC class. However, TR_Gen2 uses all the features in the IC cluster as IC training-data. Thus, one reason for TR_Gen2 having lower performance than TR_Gen3 could have resulted from it containing more features that are less likely to belong to the IC class and thus this may have shifted the decision boundary of the classifier towards the N C class. To test whether the use of more IC class features can result in lower system performance, we also applied a method similar to the T R _ G e n l method, except that the three features that are more l ikely to belong to the IC class are chosen from each trial as IC training-data instead of the single best feature. Using this training-data resulted in slightly worse performance than T R G e n l . In three of the eight tested subjects, the performance of the system significantly improves by using the proposed methods. According to the results of the eight tested subjects, i f such methods are used, we believe that the performance of the system w i l l not decrease compared to other alternative methods, and in some subjects significant improvements may be achieved. The proposed training-data generation methods were evaluated on the L F - A S D ' s 1-NN classifier which was trained by L V Q . They yielded superior results than the four alternative methods of training-data generation. A s the comparison was evaluated on the same systems with only training-data being different, this clearly indicated that the quality of the training-data significantly affects system performance. This leads us to believe that the proposed algorithms would also generate superior quality training-data when other classifiers are used, although this w i l l have to be properly verified in a future study. 126 A s mentioned in Section 4.2, before applying our algorithms, two requirements must be satisfied: (1) the use of an E R W (expected response window) is needed, and (2) a large set of features for the N C class (approximately 4 minutes of N C state data) is needed to accurately estimate the probability density distribution of N C features. The N C class data can be collected in separate sub-sessions or from the periods between the IC trials. A s such, these two requirements do not significantly constrain most approaches to system training. We used the top H=75% of N C class features as the N C class training-data. It should be noted that we tried the H=50% and H=25% and the results did not change significantly in our experiments. However, a value of H=75% is more desirable as it would represent the more variability in the N C class features (if any) and enable the classifier to handle this variability. The reason for introducing a variable H in our algorithm was to eliminate the features that are less likely related to the N C class and might be related to artifacts or other brain states. A s shown in Table 4.1 and Fig . 4.5, the training-data generation methods that select the most probable IC feature in the E R W result in better performance of the BI system than the ones that select or use the features around the T E I C such as T R T E I C or T R 3 T E I C . This implies that the T E I C is not the exact time of an actual imagined movement which is usually true as confirmed in (Bashashati et al 2004). One may suggest using a synchronized paradigm to train and set up a self-paced BI system in which T E I C may be known quite accurately. Although T E I C might be known more accurately, but there still is variability in T E I C in response to cue, and synchronizing cues may introduce different brain states (evoked or otherwise) that would not translate well when used in a self-paced setting. It should be noted that the system generates almost 100% true positive rates at higher false positive rates, e.g. FP>4%. However in our experience, relatively high false positive rates cause excessive frustration in subjects. On the other hand, i f we fix the false positive rate at 1%), true positive rates drop to less than 40%. This true positive rate is slightly better than 1 successful attempt out of every 3 attempts, a level that may or may not be usable given the application. For example, a continuous cursor control application may cause less frustration with low true positive rates compared to a menu-driven control application. For cursor control, the subject simply needs to attempt three successive commands to move a certain direction and error correction is a natural part of the control paradigm. However, with a 127 menu-driven system, a repetitive series of erroneous commands may be unduly frustrating while waiting for the correct window of opportunity to re-present. The proposed methods basically generate higher quality training-data from a population of available fuzzy training-data. In other words, these methods select a subset of training-data that has a higher probability of being real events. A s such, the proposed methods are directly applicable to other BI designs, including synchronized B i s and other neuroscience applications where covert tasks (e.g. imagined movements) are involved. In a recent study (Bashashati et al 2004), one of these proposed methods was used to generate higher quality ensemble averages of movement-related potentials related to attempted (or imagined) movements of people with SCI. The methods introduced here were tested on a 2-class brain interface. However, the methods can be expanded to more classes of IC. In the meantime, this study was performed offline; thus, an online study is needed to confirm the results in a more real-world application simulation. 4.6. References Bashashati A , Fatourechi M , Ward R and Birch G 2006 User Customization of the Feature Generator of an asynchronous Brain Interface Annals of Biomedical Engineering 3 4 1051-60. Bashashati A , Mason S, Ward R and Birch G 2004 A n Automatic Method to Generate Ensemble Averages of Movement-Related Potentials for Individuals with Spinal Cord Injuries Proc. of the 26th Inter. Conf. of the IEEE Eng. in Medicine and Biology Science (San Francisco: USA) pp. 4529-4532. Birch G E , Bozorgzadeh Z and Mason S G 2002 Initial on-line evaluation of the L F - A S D brain-computer interface with able-bodied and spinal-cord subjects using imagined voluntary motor potentials IEEE Trans. Neural Sci. Rehab. Eng. 11 219. Birch G E , Lawrence P D and Hare R D 1993 Single trial processing of event related potentials using outlier information IEEE Trans. Biomed. Eng. 4 0 59-73. 128 Borisoff J F, Mason S G , Bashashati A and Birch G E 2004 Brain Computer Interface (BCI) Design for Asynchronous Control Applications: Improvements to the L F - A S D Asynchronous Brain Switch IEEE Trans. Biomed. Eng. 5 1 985-92. Chia-Tang C, Kun-Shan C, Jong-Sen L 2003 The use of fully polarimetric information for the fuzzy neural classification of S A R images I E E E Trans Geoscience and Remote Sensing 4 1 2089-100. Cacoullos T 1966 Estimation of a multivariate density Annals of the Institute of Statistical Math. 18 179-89. Glavinovic M i l 996 Comparison of parzen density and frequency histogram as estimators of probability density functions Pflugers Arch 4 3 3 174-9. Graimann B , Huggins J E, Levine S P and Pfurtscheller G 2004 Toward a direct brain interface based on human subdural recordings and wavelet-packet analysis IEEE Trans Biomed. Eng. 5 1 954-62. Kersten P R, Lee J S, Ainsworth T L 2005 Unsupervised classification of polarimetric synthetic aperture Radar images using fuzzy clustering and E M clustering IEEE Trans Geoscience and Remote Sensing 4 3 519-27. Kohonen T 1990 The self-organizing map Proc. of IEEE 7 8 1464-80. Levine S P, Huggins J E , Bement S L , Kushwaha R, Schuh L A , Rohde M M , Passaro E A , Ross D A , Elisevich K V , and Smith R J 2000 A direct brain interface based on event-related potentials IEEE Trans. Rehabil. Eng. 8 180-5. Mason S G , Bashashati A , Fatourechi M and Birch G E Developing a Framework for Brain-Interface Technology Design," available online at http://www.braininterface.org/. Mason S G and Birch G E 2000 A brain-controlled switch for asynchronous control applications IEEE Trans. Biomed. Eng. 4 7 1297-307. Mason S G and Birch G E 2003 A general framework for brain-computer interface design IEEE Trans. Neural Syst. Rehabil. Eng. 11 70-85. 129 Mason S G and Birch G E 2005 Temporal Control Paradigms for Direct Brain Interfaces -Rethinking the Definition of Asynchronous and Synchronous Proc. HCI Inter. Conf. (Las Vegas, USA). Mil l an J R and Mourino J 2003 Asynchronous B C I and local neural classifiers: A n overview of the adaptive brain interface project IEEE Trans. Neural Syst. Rehabil. Eng. 11 159-61. Nicolelis M A 2003 Brain-machine interfaces to restore motor function and probe neural circuits Nat. Rev. Neurosci. 4 417- 22. Parzen E 1962 On estimation of a probability density function and mode Annals Math. Statistics 3 3 1065-76. Scherer R, Muller G R, Neuper C, Graimann B , Pfurtscheller G 2004 A n asynchronously controlled EEG-based virtual keyboard: improvement of the spelling rate IEEE Trans. Biomed. Eng. 5 1 979-84. Townsend G , Graimann B , Pfurtscheller G 2004 Continuous E E G classification during motor imagery—simulation of an asynchronous B C I IEEE Trans. Neural Syst. Rehabil. Eng. 12 258-65. Vaughan T M et al. 2003 Guest editorial brain-computer interface technology: a review of the second international meeting IEEE Trans. Neural Syst. and Rehabil. Eng. 11 94-109. Wolpaw J R, Birbaumer N , McFarland D J, Pfurtscheller G and Vaughan T M 2002 Brain-computer interfaces for communication and control Clin. Neurophysiol. 1 1 3 767-91. Yom-Tov E and Inbar G F 2003 Detection of movement-related potentials from the electro-encephalogram for possible use in a brain-computer interface Med. Biol. Eng. Comput. 4 1 85-93. 130 Chapter 5 User Customization of the Feature Generator of an 18 Asynchronous Brain Interface 5.1. Introduction Direct Brain Interface (BI) systems provide an alternative communication and control solution for individuals with severe motor disabilities, bypassing impaired interface pathways. Cortical activity associated with an intentional control command is mapped directly to application-specific control signals. Various devices, such as neural prostheses, can then be controlled by cognitive processes only. For a review of the field, see (Mason and Birch 2003, Nicolelis 2003, Vaughan et al 2003, Woplaw et al 2002). There are many BI transducer designs presented in the literature. However, few have been designed specifically for asynchronous control. The concept of asynchronous control of BI systems was introduced in (Mason and Birch 2000). Once the system is on, in an asynchronous BI the users affect the BI transducer output when they want by intentionally changing their brain state. Between periods of intentional control (IC), the user is in a no-control (NC) state - they may be idle, daydreaming, thinking about a problem or lunch, or performing some other action, but they are not trying to control the B I transducer. This form of intermittent control has been termed "asynchronous control" (Mason and Birch 2000). To operate in this paradigm, B I transducers are designed to respond only when there is intentional user control and to remain inactive when the user is in an N C state. B y contrast, synchronous B i s are operational only during specific periods determined by the system (not the user). Although asynchronous (or intermittent) control is the most natural mode of interaction, it has received relatively little attention; recognized in (Woplaw et al 2002), it deserves more. Only a few B I transducers (Birch et al 1993, Levine et al 2000, Mason and Birch 2000, Mi l l an and Mourino 2003, Scherer et al 2004, Townsend et al 2004, Yom-Tov and Inbar A version of this chapter has been published. Bashashati A., Fatourechi M. , Ward R., and Birch G. (2006) User Customization of the Feature Generator of an asynchronous Brain Interface Annals of Biomedical Engineering 34 1051 -60. This work has also been resulted to another publication: Fatourechi M , Bashashati A., Birch G.E., and Ward R.K. (2006) The Design of an Automatically User Customized Asynchronous Brain Interface System IEE Journal of Medical & Biological Engineering and ComputingAA 1093-104. 131 2003) have been specifically designed and tested for asynchronous control. Each produces a multi-state discrete output, and as such wi l l be referred to as "asynchronous brain switches" in the remainder of this paper. The Low Frequency-Asynchronous Switch Design ( L F - A S D ) was first introduced as a two-state B I system for asynchronous control applications (Mason and Birch 2000). It recognizes scalp potentials related to single-trial Movement Related Potentials (MRPs) in the E E G signal. The feature generator of the L F - A S D generates six-dimensional features from six bipolar E E G channels to detect M R P s in the ongoing E E G signal (Mason and Birch 2000). However, to detect the desired M R P pattern in the E E G , design parameters of the feature generator need to be properly adjusted. These parameters were originally chosen based on the E E G data of one subject and since then have been used for all subjects (Mason and Birch 2000). A s the E E G characteristics during intended movements may vary from one subject to another, design parameters need to be customized for each subject. Without such tuning, the system may detect non-MRP patterns in the ongoing E E G . Although there are few user-customized B I systems, customization of BI systems has been reported in several recent papers. These studies focused on user customization of common spatial patterns (CSPs) of the Mu/Beta rhythms of E E G (Blanchard and Blankertz 2004), user-customization of band pass filters to select the proper frequency band (Pregenzer and Pfurtscheller 1999, X u et al 2004), channel selection using support vector machines (Lai et al 2004) , automatic selection of power-band features for each subject using a genetic algorithm (GA) (Scherer et al 2004), and adjusting the parameters of the energy normalization block of the L F - A S D (Fatourechi et al 2005). These papers established that customization of the parameters of a B I system may lead to performance improvements in some subjects. In (Fatourechi et al 2005), we showed improvements in the system's performance for the two able-bodied subjects studied. This improvement motivated us to further pursue customization of the feature generator's parameters for each subject. Eight subjects were studied, four of whom were disabled. To customize parameters for each subject, the desired pattern associated with a specific movement was first determined through ensemble averaging of the E E G data related to the movement. The parameters of the feature generator were then 132 manually estimated. The feature generator's output and the performance of the customized BI are compared to those associated with the original design. We also show that huge variations in performance result when the classifier was trained on trials from different sessions (refer to Section 5.4.2). We use an off-line method of evaluation of the L F - A S D based on the stratified cross-validation approach (Witten and Frank 2000) to reduce huge variations in system performance across the cross-validation sets. We have shown that a robust performance measure of the system can only be obtained by evaluating its performance on one cross-validation set instead of evaluating the performance of the system on all the cross-validation sets. 5.2. Background Customizing the feature generator of the L F - A S D requires detailed knowledge of the L F -A S D ' s design (Mason and Birch 2000), which is briefly reviewed in this section. Fig. 5.1 shows the block diagram of the most recent version of the L o w Frequency Asynchronous Switch Design (the L F - A S D ) (Borisoff et al 2004). This design uses features extracted from six bipolar E E G channels (defined with reference to the International 10-20 System at F1-FC1 , F z - F C z , F2-FC2, F C 1 - C 1 , F C z - C z , and FC2-C2) . After amplification, all six E E G channels are normalized with an Energy Normalization Transform (ENT) (Yu et al 2002). A wavelet-like function is applied as the feature generator. The Karhunen-Loeve Transform ( K L T ) maps the six-dimensional feature space produced by the Feature Generator to a two-dimensional space. A one-nearest neighbor (1-NN) classifier is used as the feature classifier. The system's classification accuracy is further improved by using a moving average and a debounce block to reduce the number of false switch activations (for details, see (Mason and Birch 2000) (Borisoff et al 2004, Mason and Birch 2000)). Overall, the system classifies input patterns as either No Control (NC) or Intentional Control (IC). 133 electrode array Feature Extractor LF-ASD ENT |-| Feature |_j KLT Generator Feature Translator 1-NN Classifier moving average debounce Codebook generation mechanism Figure 5.1 Components of the Low Frequency Asynchronous Switch Design (the LF-ASD) (from (Borisoff et al 2004)).ENT: energy normalization transform, K L T : Karhounen-Loeve transform, 1-N N : one nearest neighbor 5.2.1. LF-ASD Feature Generator The desired bipolar E E G pattern associated with the M R P s is similar to that shown in Fig. 5.2 (Mason and Birch 2000), where the approximate time of the attempted movement is t=n. The feature extractor of the L F - A S D is designed to generate large feature values when such patterns exist in the E E G . > time t=n Figure 5.2 Desired pattern of the bipolar EEG during movement. Ej(n): amplitude difference between local maximum before the movement and local minimum after the movement, E/n): amplitude difference between local maximum before the movement and local minimum before the movement A s Fig. 5.2 shows, the elemental features £,•(«) and E -(n) are defined as the difference of a filtered signal (e(n)) at two points in time, and are calculated in equations (1) and (2) (respectively for more details see (Mason and Birch 2000)). The filtered signal is measured from a pair of bipolar electrodes and is filtered at 1-4 H z using a finite impulse response (FIR) filter based on a Hamming window. There are six such pairs and six such signals. 134 Ei(n) = e(n-ai)-e{n) (1) Ej (n) = e(n - al•) - e(n - at - a j) (2) In the rest of this paper, we use the general term "delay parameters" of the feature generator when referring to at and aj in equations (1) and (2). The delay terms are initially estimated from the ensemble averages based on the minimum peak near the trigger (at time t=n in Fig. 5.2), the first local maximum, and the local minimum before the trigger as illustrated in Fig. 5.2. The trigger point is defined as the point around which the movement is performed. Compound features are defined in equation (3) by pairing elemental features (E^Ej) to emphasize the samples in which two large elemental features appear concurrently. For robustness, the compound features are maximized over a window as follows. Gijin) = max{gij(n-S),gij(n-7),...,gij(n-l),giJ(n)\ (4) This procedure is repeated for each channel. The resulting feature vector is an equally weighted six-dimensional vector, with each dimension reflecting the value of the feature (Gy (n)) in each channel. 5.3. Data Recording The off-line data used in this study were collected from subjects positioned 150 cm in front of a computer monitor. The E E G signal was recorded from six bipolar electrode pairs positioned over the supplementary motor area and the primary motor cortex, as stated in Section 5.2. Electro-oculographic (EOG) activity was measured as the potential difference between two electrodes, placed at the corner of and below the right eye. The ocular artifact was considered present when the potential difference between the E O G electrodes exceeded ±25 p V . A l l signals were sampled at 128 Hz . The subjects used in this study consisted of four men with high-level spinal cord injuries (SCI) and three able-bodied men and one able-bodied female (subject A B 3 ) . Each of the subjects gave written consent prior to participating, according to the guidelines of the gijiri) =Ej(n)xEj(n) = 0 otherwise if £ , ( « ) X £ ( H ) > 0 (3) 135 Behavioral Research Ethics Board ( B R E B ) of the University of British Columbia. A l l subjects were right handed and between 31 and 57 years of age. None of the SCI subjects had residual sensation or motor function in their hands. Data were collected while the subjects performed a guided task. At random intervals of 5.6 to 7.0 seconds (mean of 6.7 seconds), a 2 cm white circle was displayed on the subject's monitor for % second. The subject was asked to attempt to move his right index finger one second after the cue appeared. The one-second delay was used to avoid visually evoked potential effects from the cue. The one second time after the cue is denoted by "time of the expected attempted movement" ( T E M ) . Note that this is the time when the subject is expected to attempt the movement, but this time may vary from subject to subject and from trial to trial. Both subject groups used the same neurological mechanism to drive the brain switch: an attempted finger flexion. This resulted in attempted (i.e., no physical) finger movements in subjects with high-level SCI and actual finger flexion in able-bodied subjects (see (Birch et al 2002) for more details). For each subject, an average of 80 trials was conducted every day for six days. 5.4. Methods 5.4.1. Delay Parameter Customization To determine the desired bipolar pattern associated with the movement for each subject, the signals (within a window centered at T E M ) for each of the six bipolar E E G channels were ensemble averaged. Ensemble averaging enhances the signal-to-noise ratio, and the resulting waveform exhibits the desired pattern that the L F - A S D aims to detect. After the ensemble averages of each subject were generated, the delay parameters were measured manually from the E E G averages. For example, the waveform of the ensemble average of the E E G for subject A B 2 in bipolar channel F2-FC2 was generated as shown in Fig. 5.3. A s our goal is to detect a pattern similar to the average waveform shown in Fig. 5.2, we choose the delay parameters based on the minimum peak "right after T E M " , a maximum peak "before T E M " and a minimum peak "before the location of the chosen maximum peak" from the ensemble average waveform. Based on the location of the peaks (minimum peak at t=0.43s ( M i n i ) , maximum peak at t=1.32s ( M a x l ) , and minimum peak right after T E M at t=1.97s (Min2)), the delay parameters are calculated as aj=Min2-Maxl=0.65s and aj=Maxl-Mini-0.89s. 136 Note that the attempted movement is performed around t-1.8s. A s shown in Fig. 5.3, there are several local minimum peaks (e.g., at t=0.15s or t=1.13s) just before the maximum peak at t=1.32s. We evaluated the system's performance considering these mentioned local minimum peaks. The best performance, however, was achieved at a chosen local minimum of t=0.43s. Time (Seconds) Figure 5.3 The ensemble averages of the normalized EEG in bipolar channel F2-FC2 for subject AB2. Time of the expected movement (TEM) is at t=l .8 s. For simplicity (and given that the ensemble average patterns, time-locked to T E M , are relatively the same for all six channels), equal delay parameters were used for all six bipolar E E G channels. It should be emphasized that our main goal in this paper is to show that the delay parameter customization for each subject is useful. Ensemble averages of the E E G for each subject were generated from the training data, excluding the E O G contaminated trials. The delay parameters were then calculated manually (see Table 5.1 for values). For the subjects whose ensemble average waveforms showed 137 several minimum and maximum peaks, the same procedure as explained for subject A B 2 was performed. Delays resulting in the best performance are reported in Table 5.1. T A B L E 5.1 D E L A Y P A R A M E T E R S (MEASURED IN SECONDS) ESTIMATED F R O M THE E N S E M B L E A V E R A G E S OF THE BIPOLAR E E G SIGNALS (F2-FC2) G E N E R A T E D F R O M THE TRAINING D A T A SET. NOTE T H A T THE S A M E D E L A Y P A R A M E T E R S A R E USED FOR A L L SIX BIPOLAR C H A N N E L S . Subject AB1 0.74 0.68 AB2 0.65 0.89 AB3 0.29 0.16 AB4 1 0.34 s e n 0.88 0.77 SCI2 0.74 0.41 SCI3 0.30 0.5 SCI4 0.70 0.54 5.4.2. Cross-validation and Stratified Cross-validation A s mentioned in Section 5.3, the data from each subject were collected over six days, i.e., six sessions. In our previous work (Birch et al 2002, Borisoff et al 2004, Mason and Birch 2000), the first day's E E G recordings were used to train the classifier, and data from the subsequent five days were used to evaluate the BI system. However, to determine whether the first session's data is representative of future data, we carried out the following tests. To assess the sensitivity of the classifier's performance to different training sets, we tested the system's performance using different training and test sets. Specifically, the classifier was trained based on the data of one of the days (sessions) and the system's performance was evaluated based on the data of the remaining sessions. This process was repeated for every session and averaged over the six values, in a procedure known as cross-validation (Witten 138 and Frank 2000). Table 5.2 shows the results of this analysis for all subjects. In Table 5.2, TSj represents the case in which the data of the ith session were used for training and the data of the other sessions for testing. T A B L E 5.2 T R U E POSITIVE (TP) RATES (%) A T F A L S E POSITIVES (FP) OF 2% FOR DIFFERENT TRAINING A N D TEST SETS. TS, REPRESENTS THE C A S E IN WHICH THE D A T A OF THE I™ SESSION WERE USED FOR TRAINING A N D THE OTHER SESSIONS FOR TESTING. Subject Performance (TP(%) when FP=2%) TS, TS2 TS3 TS4 TS5 TS6 Average Standard deviation ABI 69.7 64.4 46.7 73.9 45.5 82.9 63.9 15.0 AB2 75.5 68.5 57.1 74.5 62.5 82.9 70.2 9.4 AB3 68.3 68.9 65.2 72.0 77.8 53.9 67.6 7.3 AB4 77.0 78.6 73.3 87.2 85.1 75.0 89.5 5.1 SCI1 67.3 70.9 55.8 66.0 56.8 65.8 63.7 5.5 SCI2 65.6 47.7 42.6 49.0 46.8 72.7 54.4 11.1 SCI3 49.9 75.8 75.0 69.1 57.1 65.2 66.0 9.5 SCI4 59.8 54.8 51.0 41.2 51.2 57.1 53.0 6.1 A s Table 5.2 shows, the system's performance varies widely depending on which session was used for training the classifier. Specifically, the average true positives rates varied with standard deviations from 5.1% to 15% across subjects. Using the data of the first session to train the classifier and then testing it on data collected subsequently is more suitable for real-world applications. However, our analysis showed that the system's performance depends highly on the quality of first session's data. We were therefore motivated to find a way of reducing the dependency of the system's performance on the training set. In this paper, an Af-fold stratified cross-validation is used for evaluating the system's performance (Witten and Frank 2000). In stratified cross-validation, the trials are randomly divided into N groups of equal trial numbers, with approximately the same frequency of classes. A s in conventional cross-validation schemes, the classifier is trained based on the data of one of the N groups and the system's performance is evaluated based on the data of the remaining groups. This process is repeated for each group. The average of these N performance measures gives the overall performance of the system. In this study we used a five-fold stratified cross-validation. To randomly pick each cross-validation set, we divided 139 the E E G into 7s adjacent windows around each trial. Each 7s window of E E G contained an attempted movement trial as well as periods of N o Control (NC) state. Finally, cross-validation sets were generated by randomly dividing the total available 7s windows into five equal groups. The classifier was then trained based on the data of one of the five groups and the system's performance evaluated based on the data of the remaining groups. This process was repeated for every group, resulting in five performance measures. The average of these five measures represents the performance of the system. 5.5. Results In Figs. 4 and 5, the waveforms of the two-dimensional output of the feature generator and the waveforms of the ensemble averages centered at T E M are shown for one subject for the two cases: (a) our proposed customized delay parameters of the feature generator and (b) the original setting of the delay parameters. A s Fig. 5.4 shows, the new delay parameters generated stronger feature values at times when we expected them to be high, i.e., the time of the attempted movement (TEM) . This time is represented in these figures by the vertical line at t=1.8s. The feature values around the T E M were maximum in value compared to feature values at other times, when there was no movement, or an attempted one. Notice that the original settings of the delay parameters generated features at T E M that were not significantly different from the features at other times. For example, in Fig . 5.5 the features around t-1.9s and t=0.9s were not very different, while the features around t=1.9s were associated with movement. These observations suggest that with the new delay parameter settings of the feature generator, the system can generate more robust features and can better detect the desired M R P pattern. This is not the case with the original parameter settings. 140 1.2 1.6 1.8 Time (Samples) Figure 5.4 2-dimensional output of the feature generator for normalized bipolar EEG channel F2-FC2 for subject AB2 for the customized delay parameters (solid line: feature values of the first dimension, dashed line: feature values of the second dimension, and solid line with cross: bipolar EEG). Note that the vertical line at t=l .8 s corresponds to the time of the attempted movement (TEM). 141 - 1 u L 0.4 0.8 1.2 1.6 Time (Samples) 1.8 2.4 2.8 Figure 5.5 2-dimensional output of the feature generator for normalized bipolar EEG channel F2-FC2 for subject AB2 for the original delay parameters (solid line: feature values of the first dimension, dashed line: feature values of the second dimension, and solid line with cross: bipolar EEG). Note that the vertical line at t=l .8 s corresponds to the time of the attempted movement (TEM). The performance o f a two-state asynchronous B I system can be evaluated using two measures: (1) the percentage of correct activations during IC states (true positives, or "TPs") and (2) the percentage of false switch activations during N C states (false positives, or "FPs"). A TP was identified i f the BI system was activated at any time within a window of 0.5 seconds before to 1 second after the T E M , called the "TP window". This method is similar to those used by others (Levine et al 2000, Townsend et al 2004, Yom-Tov and Inbar 2003). False positives were assessed in the periods before the system cued subjects to perform the movement and after subjects were expected to perform it (see (Borisoff et al 2004) for more details). We did not evaluate the period during which ocular artifacts occurred. For all subjects, the five-fold stratified cross-validation algorithm was run ten times for each delay parameter set. Adjusting the parameters of the feature generator improved the system's 142 performance in four subjects, from whom three were able-bodied. However, for the rest of the subjects, the performance of the system did not improve with the use of customized delay parameters. Table 5.3 shows: (1) the average true positive rates (at fixed false positive rates of 2%), with standard deviations across the 10 runs for both original and customized delay settings for the subjects, and (2) the significance of the improvements using 'two sample T-test' with significance level of Alpha=0.1. T A B L E 5.3 P E R F O R M A N C E OF THE LF-ASD TOGETHER WITH S T A N D A R D DEVIATIONS ACROSS 10 RUNS OF STRATIFIED CROSS-VALIDATION WITH ORIGINAL A N D N E W D E L A Y P A R A M E T E R S . Subject Performance (TP (%) in FP=2%) Original Customized Improvement Significance parameters parameters level of improvement AB1 65.0 ± 1.4 67.8± 1.4 +2.8 p<0.005 AB2 67.2 ± 1.6 74.0 ± 1.7 +6.8 p<0.005 AB3 62.0+ 1.9 64.0+ 1.3 +2.0 p<0.03 AB4 76.5 ± 1.5 73.1 ± 1.8 - -sen 61.1 ±2.2 63.5± 1.9 +2.4 p<0.08 SCI2 71.2± 2.2 68.0±2.7 - -SCD 52.6±2.3 52.8 ±2.4 +0.2 -SCI4 55.9±3.0 52.9± 3.1 -Table 5.4 shows the average standard deviations (STDs) across the five cross-validation sets to be low compared to values associated with the evaluation procedure (as shown in Table 5.2). The low S T D (~0.5%) of performance across the five cross-validation sets implies that the TP rate for all five cross-validation sets is relatively the same. Consequently, the TP rate for one cross-validation set could show the overall TP rate of the system. 143 T A B L E 5.4. A V E R A G E S T A N D A R D DEVIATION (STD) OF TP RATES (AT FIXED FP=2%) ACROSS E A C H CROSS-VALIDATION SET. Subject ABI AB2 AB3 AB4 SCIl SCI2 SCD SCI4 Average standard deviation (%) 054 055 042 053 057 066 O50 1.28 Figs. 6-9 show the R O C (receiver operating characteristic) curves for four of the subjects (subjects A B I , A B 2 , A B 3 , S C I l ) for the two designs using (1) the original delay parameter values and (2) the new customized delay values for the feature generator. A s Figs. 6-9 show, the new method generated a better TP rate at most of the FP rates. False positive rate (%) Figure 5.6 ROC curve for subject A B I (solid line with circles: ROC curve for the customized BI system, solid line: ROC curve for the original BI system). Note that, for clarity, only false positives below 4% are shown. 144 100 )l I I I I I I I I 0 0.5 1 1.5 2 2.5 3 3.5 4 False positive rate (%) Figure 5.7 ROC curve for subject AB2 (solid line with circles: ROC curve for the customized BI system, solid line: ROC curve for the original BI system). Note that, for clarity, only false positives below 4% are shown. 145 100 nl i i i i i i i I 0 0.5 1 1.5 2 2.5 3 3.5 4 False positive rate (%) Figure 5.8 ROC curve for subject AB3 (solid line with circles: ROC curve for the customized BI system, solid line: ROC curve for the original BI system). Note that, for clarity, only false positives below 4% are shown. 146 100 ) l I I I I I I I I 0 0.5 1 1.5 2 2.5 3 3.5 4 False positive rate (%) Figure 5.9 ROC curve for subject SCI1 (solid line with circles: ROC curve for the customized BI system, solid line: ROC curve for the original BI system). Note that, for clarity, only false positives below 4% are shown. 5.6. Conclusions In this paper, we presented a method for estimating the parameters of the feature generator of our current asynchronous BI system, the L F - A S D . The error characteristics of the new design were shown to be better than those of the original design for four subjects. For these subjects, the true positive rates increased by 2% to 6.8% at false positive rates of 2%. We attribute the improvements to the adjusted delay parameters of the feature generator block. It is highly important to point out that with the new customized parameters, the system detected the desired subject-specific M R P pattern associated with movement. This was achieved by choosing a suitable set o f delay parameters to result in more consistent and stronger feature values during movement. Although the system's performance for four subjects did not increase, the system did detect the desired bipolar M R P pattern, which may not be the case with the original setting. In a BI system, it is very important to ensure that 'only' specific predefined movement attempts 147 activate the output and not any other mental activity. Although the performance of the system with the original delay parameters was slightly higher than that of the customized one, we believe that some patterns had been activating our original B I that were not related to the subject-specific bipolar M R P pattern we were interested in. In Fig . 5.4, which shows the ensemble averages of the E E G of subject A B 2 , the old delay setting generated features around movement that were not very different from feature values at other times; this may have caused performance decrease of the system in subject A B 2 . The opposite can also happen, i.e., some features can be generated in the TP window that activate the output but may not be related to movement. In such a case, the system is activated by E E G patterns that may not be attributed to the desired bipolar M R P pattern. Since the system aims at detecting movements from spontaneous E E G , we cannot determine whether the activations of the system are related to movement attempts or to other brain activity. With the new delay parameter values, one can expect activations of the BI system that are more likely due to movement attempts than other patterns in the E E G . A good original delay parameter value or poor manual customization might also account for the system's performance not improving. These results are not surprising, as other studies have reported performance degradation after customizing their B I system for some subjects. For example, in (Blanchard and Blankertz 2004) the error rate of one of the three subjects decreased more than 11% after customization of the C S P patterns. In (Pregenzer and Pfurtscheller 1999) selection of subject-specific frequency components of the E E G for a B I system is reported. Although the results for two subjects improved when using the D S L V Q (distinctive sensitive L V Q ) classifier, the performance of the third subject degraded after customization. However, it should be mentioned that it is difficult to directly compare the results from our study and these studies, as the recording equipment, recording and classification protocols, and mental tasks considered are different. In addition, the amount of data involved and the degree of training the subjects received before participating in the BI experiments varies for different studies. Compared to the results of our previous study (Fatourechi et al 2005), which customized the normalization block of our BI design, it appears that customization of the feature generator yielded less improvements than customization of the normalization block did for two of the tested subjects ( A B 2 and A B 3 ) in (Fatourechi et al 2005). However, the results cannot be 148 generalized at this stage as the study reported in (Fatourechi et al 2005) was only performed for two of the subjects. Despite the need to verify the results on a larger subject pool, current study yielded performance improvements in most of the able-bodied subjects rather than in the SCI subjects. Since estimating delay parameters from ensemble averages for SCI subjects proved to be difficult, we might not have been able to estimate a suitable delay setting for these subjects. However, we believe that the new customized system detects the M R P pattern in the E E G , which may not be the case for the original design. The results of the stratified cross-validation showed that the standard deviation of the performances across cross-validation sets was low (as shown in Table 5.4). This implies that there is no need to repeat the analysis for each of the five cross-validation sets. Instead, by randomly choosing a set of trials for training, and evaluating the system on the rest of trials, a robust performance measure of the system can be obtained. Evaluating the performance of our system for a set of parameters using five-fold stratified cross-validation takes about 10 minutes on a Pentium 2.8GHz computer; with the new method, only l / 5 t h o f this time is needed. This new analysis procedure mostly benefits an automated parameter customization method. Using such method, the system's performance must be evaluated (for different parameter settings) hundreds of times. Thus, choosing a robust and time-efficient method for evaluating the system is desirable. The study successfully detected the desired M R P pattern and decreased the error rates of our BI design for 50% of tested subjects. This decreased error rate demonstrates the need for a method that automatically adjusts the delay parameters for each subject. A s mentioned in Section 5.4.1, manually estimating the delay parameter values may not yield better performances when there are several minimum or maximum peaks around T E M in the E E G ensemble averages. In such cases, it may be necessary to choose several delay settings and select the one that yields better performance. Using an automated method of adjusting the delay parameters may result in the best delay setting, further increase the system's performance, and remove the subjective bias in delay parameter adjustments. Our future work wi l l specifically explore customization of the L F - A S D parameters in an automated framework. 149 5.7. References Birch G E , Bozorgzadeh Z and Mason S G 2002 Initial on-line evaluation of the L F - A S D brain-computer interface with able-bodied and spinal-cord subjects using imagined voluntary motor potentials Trans. Neural Syst. Rehabil. Eng. 11 219-24. Birch G E , Lawrence P D and Hare R D 1993 Single trial processing of event related potentials using outlier information IEEE Trans. Biomed. Eng. 40 59-73. Blanchard G , Blankertz B 2004 B C I Competition 2003, Dataset Ha: spatial patterns o f self-controlled brain rhythm modulations IEEE Trans. Biomed. Eng. 5 1 1062 - 6 . Borisoff J F, Mason S G , Bashashati A and Birch G E 2004 Brain Computer Interface (BCI) Design for Asynchronous Control Applications: Improvements to the L F - A S D Asynchronous Brain Switch IEEE Trans. Biomed. Eng. 5 1 985-92. Fatourechi M , Bashashati A , Ward R K and Birch G E 2005 A Hybrid Genetic Algorithm Approach for Improving the Performance of the L F - A S D Brain Computer Interface Proc. of the IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (Philadelphia, USA) pp 345-8. Levine S P, Huggins J E , Bement S L , Kushwaha R, Schuh L A , Rohde M M , Passaro E A , Ross D A , Elisevich K V and Smith R J 2000 A direct brain interface based on event-related potentials IEEE Trans. Rehabil. Eng. 8 180-5. Mason S G and Birch G E 2000 A brain-controlled switch for asynchronous control applications IEEE Trans. Biomed. Eng. 47 1297-307. Mason S G and Birch G E 2003 A general framework for brain-computer interface design IEEE Trans. Neural Syst. Rehabil. Eng. 11 70-85. M i l l a n J R and Mourino J 2003 Asynchronous B C I and local neural classifiers: A n overview of the adaptive brain interface project IEEE Trans. Neural Syst. Rehabil. Eng. 11 159-61. Lai T N , Schroder M , Hinterberger T, Weston J, Bogden M , Birbaumer N and Scholkpof B 2004 Support vector channel selection in B C I IEEE Trans. Biomed. Eng. 5 1 1003-10. Nicolelis M A 2003 Brain-machine interfaces to restore motor function and probe neural circuits Nat. Rev. Neurosci. 4 417- 22. 150 Pregenzer M and Pfurtscheller G 1999 Frequency component selection for an EEG-based brain to computer interface IEEE Trans. Rehabil. Eng. 7 413-9. Scherer R, Muller G R, Neuper C, Graimann B and Pfurtscheller G , " A n asynchronously controlled EEG-based virtual keyboard: improvement of the spelling rate IEEE Trans. Biomed. Eng. 5 1 1979-84. Townsend G , Graimann B and Pfurtscheller G 2004 Continuous E E G classification during motor imagery—simulation of an asynchronous B C I IEEE Trans. Neural Syst. Rehabil. Eng. 12 258-65. Vaughan T M et al. 2003 Guest editorial brain-computer interface technology: a review of the second international meeting IEEE Trans. Neural Syst. and Rehabil. Eng. 11 94-109. Witten I H and Frank E 2000 Data Min ing (San Diego, C A : Academic Press). Wolpaw J R, Birbaumer N , McFarland D J, Pfurtscheller G and Vaughan T M 2002 Brain-computer interfaces for communication and control Clin. Neurophysiol. 1 1 3 767-91. X u W , Guan C, Siong C E , Ranganatha S, Thulasidas M and Jiankand W 2004 High accuracy classification of E E G signal Proc. of the IEEE Inter. Conf on Pattern Recognition (Cambridge, UK) pp 3 91 -4. Yom-Tov E and Inbar G F 2003 Detection of movement-related potentials from the electro-encephalogram for possible use in a brain-computer interface Med. Biol. Eng. Comput. 4 1 85-93. Y u Z , Mason S G and Birch G E 2002 Enhancing the performance of the L F - A S D brain-computer interface Proc. of the 24th Annual Conf. and the Annual Fall Meeting of the Biomed. Eng. Society (Houston, TX, USA) pp 2443-4. 151 Chapter 6 An Improved Asynchronous Brain Interface: Making Use of the Temporal History of the LF-ASD Feature Vectors 1 9 6.1. Introduction Over the past decade, several research groups have developed direct brain interface (BI) systems as possible alternative communication and control solutions for individuals with severe disabilities. For a review of the field see (Mason and Birch 2002, Nicolelis 2003, Vaughan et al 2003, Wolpaw et al 2002). BI technology aims at mapping the user's cortical activity associated with an intentional control (such as imagined finger movements) directly to application-specific control signals. Thus, control of various devices such as neural prosthetics is made possible by cognitive processes only. In other words, BI systems bypass traditional interface pathways which cannot be used by individuals with severe disabilities. Several different approaches to the design of BI technology based on signals from scalp electrodes, i.e., electroencephalograms (EEG) , or implanted electrodes have been reported for various communications and control applications. A l l these systems can be represented by the common functional model presented in Fig. 6.1 (Mason and Birch 2003, Mason et al 2003, mason et al 2005). The components between user and assistive device can be treated as a single component, a B I transducer, which functions in a manner similar to a physical transducer like a dial or switch. The role of the BI transducer is to translate the user's brain activity into reliable control signals. There are many BI transducer designs presented in the literature. However, few have been designed specifically for asynchronous control as defined in (Mason and Birch 2000). In general, BI systems are designed to operate in either asynchronous or synchronous control paradigms. For an asynchronous BI system, when the system is O N , the user affects the BI transducer output when they want by intentionally changing their brain state. In between periods of this intentional control (IC) the user is in a no control (NC) state —they may be idle, daydreaming, thinking about a problem or lunch, or performing some other action, but they are not trying to control the BI transducer. To operate in the asynchronous paradigm, B I 1 9 A version of this chapter has been published. Bashashati A., Mason S.G., Ward R.K., and Birch G. (2006) An Improved Asynchronous Brain Interface: Making Use of the Temporal History of the LF-ASD Feature Vectors Journal of Neural Engineering 3(2) 87-94 (Invited paper). 152 electrodes I User 1 BI Transducer amp Artifact Feature I^ FeMirre J ; -Processor Extractor Translator 4 Assistive Device Figure 6.1 Functional model of a BI system depicting the principle functional components. More details can be found in (Mason and Birch 2003). transducers are designed to respond only when there is intentional user control and maintain an inactive state output during times when the user is in a N C state. On the other hand, in a synchronous B I system, the allowable times for intentional user control are restricted to periods defined by the system. Thus, in synchronous systems, the B I technology is tested only during intentional user control and the response of the BI system during N C state is not tested. Even though asynchronous (or intermittent) control is the most natural mode of interaction, it has received relatively little attention in the field. A s recognized in (Wolpaw et al 2002), this is an important problem that requires more attention. Only a few BI transducers (Birch et al 1993, Graimann et al 2004, Levine et al 2000, Mason and Birch 2000, Mi l l an and Mourino 2003, Scherer et al 2004, Townsend et al 2004, Yom-Tov and Inbar 2003) have been specifically designed (and tested) for asynchronous control. Each of the proposed transducers produces a multi-state discrete output and as such w i l l be referred to as an asynchronous brain switch in the remainder of this paper. In developing a non-invasive B I system, the L o w Frequency-Asynchronous Switch Design ( L F - A S D ) was first introduced as a BI for asynchronous control applications (Mason and Birch 2000). The L F - A S D seeks to recognize the movement related potentials (MRPs) in the E E G signal. Recent studies with the latest design of the L F - A S D have demonstrated an average true positive (TP) rate of 64.7% for false positive (FP) rates of 2% (Borisoff et al 2004). To aid the presentation, the latest design of the L F - A S D (Borisoff et al 2004) w i l l be referred as L F - A S D - V 4 in the remainder of the paper. Among the proposed asynchronous B I 153 transducers whose offline performances have been reported in terms of true positives and false positives, (Graimann et al 2004, Levine et al 2000, Townsend et al 2004, Yom-Tov and Inbar 2003), the L F - A S D generates less false positives than others. A l l these offline studies reported average false positive rates in the range 6% and 28% while the average true positive rates were between 73% and 94% (Graimann et al 2004, Levine et al 2000, Townsend et al 2004, Yom-Tov and Inbar 2003). However, it should be mentioned that it is difficult to directly compare the results from our study and these studies, as the recording equipment, recording and classification protocols, and mental tasks considered are different. In addition, the amount of data involved and the degree of training the subjects received before participating in the BI experiments varies for different studies. Our experience to date indicates that the error rates of our design are still too high for individuals with high-level spinal cord injuries (our target population) in most practical asynchronous control applications. For practical applications, one needs to focus on low FP rates. From our experience, FP rates above 2% cause excess frustration and distraction in subjects (Birch et al 2002). In the L F - A S D - V 4 design, the output state of the system at time t,, 0{t{), is determined by the values of the feature vectors at time tx, FVV4 (/,). In other words, the output of the system, at time/! is a function of the feature vector value at t imef l 5 \.Q.,0(tx) = f(FVV4(tl)), where /( . ) is a function that maps the feature values to the output of the system. The assumption that the output state at timer], depends only on the value of the features at that time may not be realistic. This is because a movement, however short in duration, would take more than an instant of time. Moreover, this dependency on the exact instant in time makes the system more vulnerable to E E G signal artifacts. Thus, artifacts may easily trigger the output of the system and cause false activations of the output. Artifacts can also mask the features associated with a movement, thus causing the system not to detect such movement attempts. This would consequently decrease the true activation rate. To study this dependency between the output, 0(t), and feature values, FVV4 (t), we analyzed the ensemble averages of the movement related potentials (MRP) . A s these ensemble averages contain movement related information, we found that during a movement preparation and its execution the vector of the features extracted from the E E G ensemble 154 averages move on a path with a specific, subject-dependent shape. A s an example, Fig. 6.2a shows the ensemble averages of the E E G of a subject for a time span of over three seconds. Fig. 6.2b shows the path of the 2-dimensional feature vector associated with the ensemble averages of the E E G signal centered to the finger switch activations. F ig . 6.2b shows that feature vectors move on a specific path during the movement (movement is attempted at t=1.875s). This path is different from that of which there was no movement (t<1.2s). Figure 6.2 (a) Ensemble averages of the EEG of subject ID and (b) the corresponding feature vectors over time. The attempted movement is at t=l ,875s. This phenomenon implied that the past values of the features provided more knowledge about the actual movement attempt than that provided by the value of the features at one time instant. Thus, it was hypothesized that including the past information of the feature vector would improve the switch's error rates and robustness to artifacts. 155 In this paper, we present a method that models the path that the feature vector traverses during specific movement attempts (IC state) as well as the paths traversed during the N o Control (NC) state. The specific path shapes (path templates) are used to identify the specific M R P patterns in the ongoing E E G signal and to trigger the output of the BI system. The performance of the modified design, named L F - A S D - V 5 , is evaluated using E E G recordings of attempted finger movements of individuals with spinal cord injuries (SCI) subjects as well as able-bodied subjects. In Sections 6.2 of this paper, the proposed design of the modified L F - A S D design is presented. Section 6.3 presents the evaluation method used in this study. The results and conclusions are followed in Sections 6.4 and 6.5, respectively. 6.2. Proposed design of the LF-ASD-V5 In the design of the L F - A S D - V 5 , we aim at incorporating the history of the feature path where the features move in the multidimensional space during both IC states (attempted movement) and N C states. In other words, we want to find a relation between the output of the system and its input as written in equations (1) and (2): 0(tx) = g(FVV5(tx)) (1) FVV5(tx) = [FVV4(tx-L) ... FVVA(tx-\) FVV4(tx)} (2) where FVvs(tx) is the feature matrix at time tx, 0{tx) is the output at time tx, [FVV4(tx -L) ... FVYA(tx -1) FF K 4 ( / , ) ]are the values of the original features of the L F -A S D - V 4 in the 'time window' of t - tx -L to t = tx, L is the length of the window, and g(.) is the function that maps the feature values to the output of the system. Note that the feature matrix FVvs(tx)represents the path that the original feature vectors traverse during time. Basically, the new feature space captures the paths of the feature vectors and was referred to as "path space" in this study. Our goal is to find the representative paths of feature vectors FVy4, that correspond to IC or N C states. Conceptually these representative paths of feature vectors represent the paths that the features move through during N C and IC states. These representative paths of both state classes form the "path templates". 156 Fig. 6.3 shows the design of the L F - A S D - V 5 . L F _ A S D - V 5 has the same overall structure of the L F - A S D - V 4 design except that now the feature extraction block contains an additional buffer. This block generates the new features, FVV5, as described in equations (1) and (2). A s the M R P pattern of each subject may vary in duration, the buffer length (L) is defined separately for each subject. Like the L F - A S D - V 4 , this design uses features extracted from the 0-4Hz band in six bipolar E E G channels. After amplification, all six E E G channels are normalized with an Energy Normalization Transform (ENT). Then a low-pass-filter is used to decrease the interference with the features in the high-frequency band. A wavelet-like function is applied as the feature generator to each of the six bipolar E E G channels (For more details see (Mason and Birch 2000)). The resulting feature vector is a six-dimensional vector, with each dimension reflecting the value of the feature in each channel. The Karhunen-Loeve Transform ( K L T ) component is used to reduce the 6-dimensional feature vector to a 2-dimensional feature vector (FVV4, as shown in Fig. 6.3). A 1-NN classifier is used as the feature classifier. Finally, a moving average and a debounce block are used to further improve the classification accuracy of the system by reducing the number of false switch activations (for details, see (Mason and Birch 2000, Borisoff et al 2004). The system classifies the input patterns, at every 1/16th of a second, to one of the two classes, No Control (NC) or Intentional Control (IC) brain states. 6.2.1. Path Template Generation The procedure used to find the specific path templates related to the movement (IC) and no movement (NC) states is shown in Fig. 6.4. These path templates are generated using the training data. To generate these path templates which form the codebooks in the new feature space, the k-means algorithm (Kohonen 1990) with three vectors per class state is used to generate initial clustering of each class separately. This is followed by Learning Vector Quantization (LVQ3) (Kohonen 1990) to find the final codebook {path template) in the path space. In the L V Q algorithm, the learning rate, a, is set to 15/(length of training data) and s to 0.25. The algorithm is stopped after it runs 2000 times or i f the template does not change significantly according to a predefined threshold. To generate the training data for the IC state, a window of feature vectors ( F V V 4 ) is selected around the time of expected (attempted) 157 movement, T E M . (Note, details about T E M are given in Section 6.3). In other words, for each attempted movement, one feature vector at t=TEM is picked for the training set. To generate the training data for the N C state, we randomly picked features in the path space (FVV5(t)) from the first session's data corresponding to a N C state that did not contain any eye-blink artifacts. B O O e l e c t r o d e a r r a y amp ENT Feature Extractor FV V LF-ASD Feature Generator KLT B u f f e r FV V5 Feature Translator 1-NN Classifier moving average debounce Path template generation mechanism Figure 6.3 Components of the LF-ASD-V5 transducer, where ENT = Energy Normalization Transform, K L T = Karhunen-Loeve Transform, and 1 - N N = 1 -Nearest Neighbour. FV(t) t = T-L t = T llllll Willi M M W Output Class label at time t =T Time FVV4(T-L) ... FVV4(T)]=FVV5(t = T) feature buffering independent K-Means clustering of each class LVQ3 training database of pre-coded features Path templates for ' ICandNC classes Figure 6.4 Path template generation procedure (also known as classifier training) 6.3. Evaluation Method The data used in this study were collected from subjects positioned 150 cm in front of a computer monitor. The E E G signal was recorded from six bipolar electrode pairs positioned 158 over the supplementary motor area and the primary motor cortex (defined with reference to the International 10-20 System at F1-FC1 , Fz -FCz , F2-FC2, F C 1 - C 1 , F C z - C z , and F C 2 -C2). Electro-oculographic (EOG) activity was measured as the potential difference between two electrodes, placed at the corner and below the right eye. Eye-blink artifacts were considered present when the difference between the E O G electrodes exceeded ±25 p V . A l l signals were amplified by a Grass Model 8-18C E E G amplifier, filtered to a pass-band between 0.1 to 30Hz and sampled at 128Hz by a P C equipped with a 12-bit analog to digital converter embedded on a Data Translation 2801A data acquisition board. The subjects participated in this study consisted of four subjects with a high-level spinal cord injury (SCI) and four able-bodied subjects. A l l subjects were male (except subject K T ) , right handed between 31 and 57 years old. A l l SCI subjects had no residual sensation or motor function in the hands. A l l the subjects were given written consent according to the Behavioral Research Ethics Board ( B R E B ) of the University of British Columbia. The data were collected from the subjects while performing a guided task in 2-minute sub-sessions. These sub-sessions contain both the N C and IC state periods as shown in Fig. 6.5. A t random intervals of 5.6 to 7.0 seconds (mean of 6.7 seconds), a 2cm white circle was displayed on the subject's monitor for lA second, prompting them to attempt a movement. In response to this cue, the subject tried to activate the brain switch by attempting to move his right index finger one second after the cue appeared. The one-second delay was used to avoid visual evoked potential effects from the activity. The time, one second after the cue, is called "time of expected (attempted) movement ( T E M ) " . A s the time to perform the movement attempt depends on a user's response, the movement attempt is not necessarily performed at T E M and may change from subject to subject and attempt to attempt. Both subject groups used the same neurological mechanism to drive the brain switch: an attempted right index finger flexion. This resulted in no movement in subjects with high level SCI, and an actual finger flexion in able-bodied subjects. For each subject, an average of 80 trials was collected every day for 6 days. 159 Time (s) NC state IC state NC state IC state NC state Figure 6.5 Structure of each 2-minute sub-session of E E G recording, where TEM=time of the expected (attempted) movement Besides recording the E E G data during movement attempts, in each session, the E E G data were also recorded for several 2-minutes periods while the person was in a specific no control (NC) state. The sub-sessions that contain only N C state E E G data are gathered in different N C states such as attentive eyes opened while looking at a picture on the monitor or doing a search task, and attentive eyes closed. The reason to record this type of data was to evaluate the performance of the BI design in different N C state periods. The proposed design is evaluated on all the available data, i.e., the sub-sessions that contain mixed N C and IC state data and the sub-sessions that include only N C state periods. The E E G data of the mixed N C and IC states contain movement attempts approximately every 6.7 seconds separated by periods of N C state. The reasons behind the addition of the periods of pure N C state data were to: 1) more thoroughly evaluate the performance of the system on different types of N C state data, and 2) approach the real-world paradigm where a person performs control with longer periods of N C state. The ability of the subjects to control the BI system was evaluated by 1) percentage of correct activations during IC states (true positives, TPs) and 2) percentage of false switch activations during N C states (false positives, FPs). A TP was identified i f the BI system was activated at least once in a window 0.5 seconds before and 1 second after the time of the expected movement (TEM) , a method similar to that employed by others (Graimann et al 2004, Levine et al 2000, Townsend et al 2004, Yom-Tov and Inbar 2003). FPs were assessed in the periods before the appearance of a white circle and after the end of the activation window. The testing system employed was the same as was used by (Mason and Birch 2000, Borisoff et al 2004]. This system automatically detected eye-blinks and blocked the output of the system during these times. Table 6.1 details the percentage of the recorded data of the 160 evaluation set that were blocked by the system for each subject. From the numbers in Table 6.1, an average of 22% of the E E G data contained eye-blinks and 7.4% of the trials of the test set are blocked by the system due to the presence of eye-blinks. The first day's recordings were used to train the classifier and those from the subsequent 5 days were used for evaluation. T A B L E 6.1. DETAILS OF THE EEG D A T A OF THE E V A L U A T I O N SET T H A T W E R E B L O C K E D B Y THE TEST S Y S T E M D U E TO E Y E - B L I N K Subject ID CB CS KT AJ LB BK RT Total duration of recorded EEG (minutes) 66 77 61 87 63 81 64 79 Percentage of data with eye-blinks (%) 17 19 15 28 34 22 19 23 Total number of movement attempts 320 411 294 397 324 415 356 322 Number of movement attempts blocked by the 21 16 12 4 43 25 76 10 test system due to eye-blink. Percentage of movement attempts blocked by 6.6 3.9 4.1 1.0 13.3 6.0 21.4 3.1 the test system due to eye-blink. 6.4. Results The performance of the L F _ A S D - V 5 design is summarized via the receiver operating characteristic (ROC) curves. These results are compared to the performance of the latest L F -A S D design ( L F - A S D - V 4 ) (Borisoff et al 2004). For each subject, different buffer lengths (L's) were used to build the classifier and the buffer length that resulted in the best performance is reported here. Specifically, we tried buffer lengths of 1, 3, 5, 7, 9 and 11. Buffer length of one corresponds to the design of previous system, L F - A S D - V 4 . Figs. 6 and 7 show the path templates o f the feature vectors that the classifier generates for the two cases 1) Intentional Control (IC) state, and 2) N o Control (NC) state. In these figures, the path templates for each class are shown for subjects L B and ID. These templates correspond to the buffer lengths that resulted in the best performance for these subjects. A s the figures show, the path templates of the IC state (the paths that the FVv4 features move on during movement attempts) are quite different from those of the N C state path templates. In most cases, the path templates during N C states were templates with small amplitudes. This coincides with the rationale of the original feature extractor design. The original feature 161 extractor of the L F - A S D was designed to produce features with large values when specific movements were performed and features with small values when the subject was in the N C state. A s mentioned in Section 6.2, the algorithm was designed to generate three path templates per class. In some cases, like the one shown in Fig. 6.6 for N C state path templates, the three templates estimates for the N C class were very similar, thus when they are plotted they appear as one template in this figure. The reason is that the data of that class could be represented with less (than three) path templates e 1 > i 0.5 E b -0.1875 -0.125 -0.0625 +0.0625 +0.125 +0.1875 -0.1875 +0.1875 Figure 6.6 Path templates for subject L B for the two cases 1) movement (IC) state (lines with circles) and 2) no movement (NC) state (lines with crosses). Time=0 corresponds to the time of the expected (attempted) movement (TEM). Note that the algorithm estimates three path templates per class. However, as can be seen in this figure, the three path templates of the N C state 162 Time (s) Figure 6.7 Path templates for subject ID for the two cases 1) movement (IC) state (lines with circles) and 2) no movement (NC) state (lines with crosses). Time=0 corresponds to the time of the expected (attempted) movement (TEM). Note that the algorithm estimates three path templates per class. However, as can be seen in this figure, two of the three path templates of the IC state were very similar and are seen as one template. This is also can be seen for N C state path templates. The performance of the L F - A S D - V 5 was compared to the L F - A S D - V 4 design for a debounce period of 16 (as used in previous study (Borisoff et al 2004)). In Tables 6.2 and 6.3, we show the TP rates at fixed FP rates of 1% and 2% for able-bodied and SCI subjects. The results showed that the best performance improvement of 50% was achieved for subject B K when the modified BI system ( L F - A S D - V 5 ) was used. For able-bodied subjects, the average performance of L F - A S D - V 5 was approximately 6.5% better than L F - A S D - V 4 in the false positive ranges of 1% to 2%. On the other hand, L F - A S D - V 5 improved the detection of attempted movements of spinal cord injured subjects by an average of 25% in the false positive ranges of 1% to 2%.The hypothesis that the performances of L F - A S D - V 4 and L F -A S D - V 5 , operating at false positive level of 2%, had equal means (against the alternate 163 hypothesis that L F - A S D - V 5 had higher average performance than L F - A S D - V 4 ) was rejected at significance level of p<0.01 using paired T-test. The results in Tables 6.2 & 6.3 show that the average true positive rates for SCI subjects are 3.2-6.7% lower than the able-bodied subjects. Using the new design, the average true positive rate differences between the two subject groups drop to the 1-3.4% range. T A B L E 6.2 TRUE POSITIVE (TP) RATES A T FIXED F A L S E POSITIVE (FP) RATES OF 1% A N D 2% FOR ABLE-BODIED SUBJECTS. Absolute TP Percentage of TP LF-ASD-V4 LF-ASD-V5 B u f f e r Improvement at Improvement at Subject '• '-length (L) TP (%) at TP (%) at TP (%) at TP (%) at FP=1% FP=2% FP=1% FP=2% FP=1 % FP=2% FP=1% FP=2% ID 7 41.1 64.6 47.6 69.2 i-6.5 +4.6 +16% +7% CB 5 44.5 66.0 42.7 68.2 -1.8 +2.2 -4% ,; +3 CS 1 45.8 67.6 45.8 67.6 0 0 0% 0°.. KT 9 23.6 48.5 28.3 54.0 +4.7 , +5,5 +20% + 11% Average 38.8 61.7 41.1 64.8 , +2.3 +3.1 +8% . +5% T A B L E 6.3 T R U E POSITIVE (TP) RATES A T FIXED F A L S E POSITIVE (FP) RATES OF 1% A N D 2% FOR SCI SUBJECTS. T T , , „ ^ T T . T T , Absolute TP Percentage of TP LF-ASD-V4 LF-ASD-V5 b Buffer Improvement at Improvement at Subject -length (L) TP (%) at TP (%) at TP (%) at TP (%) at FP=1% FP=2% FP=1% FP=2% FP=1% FP=2% FP=1% FP=2% AJ 5 31.7 61.0 43.7 71.1 +12.0 + 10.1 •38% • 17% LB 7 39.3 64.5 44.7 69.9 -5.5 -5.4 • 14 8% BK 9 28.6 53.5 42.9 68.5 • 14.3 .15 0 +50% +28% RT 5 28.7 55.0 36.9 63.1 +8.2 • ' +8.1 +29% + 15% Average 32.1 58.5 42.1 68.2 +10.0 +9.7 +33% +17% 164 Truncated R O C curves for two SCI subjects ( B K and RT) are shown in Figs. 8 and 9 for both L F - A S D - V 4 and our proposed L F - A S D - V 5 design. A s we are interested in lower FP rate levels, only those values of FPs below 5% are shown in the R O C curves. A s the figures show, for most of the FP rate levels, L F - A S D - V 5 generated a better TP rate than L F - A S D -V 4 . 2 2.5 3 FP Rate (%) 3.5 4.5 Figure 6.8 Truncated ROC curves for SCI subject B K for LF-ASD-V4 system (Dashed line) and our proposed LF-ASD-V5 design (Solid line with circles). 165 QI i i i i i i i i i I 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 F P Rate (%) Figure 6.9 Truncated ROC curves for SCI subject RT for LF-ASD-V4 system (Dashed line) and our proposed LF-ASD-V5 design (Solid line with circles). 6.5. Conclusions We have introduced a new design of an asynchronous BI by utilizing the history of feature vectors in time. The error characteristics of the proposed L F - A S D - V 5 design were better than the previous design ( L F - A S D - V 4 ) with true positive rates increases of up to 50% for false positive rates in the 1-2% range. We have demonstrated that utilizing the knowledge of the path of feature vectors improved the performance of the L F - A S D , but this idea may be useful for other BI transducer designs and pattern recognition problems. Results showed that the performances of the SCI subjects were improved by approximately 25%) on average, while the average improvements for able-bodied subjects were 6.5%. These results demonstrated that the proposed method mostly benefited the SCI subjects rather than the able-bodied ones. One reason behind the higher improvements for SCI subjects might be that the able-bodied subjects generated M R P patterns with large amplitudes and thus the 166 previous design could detect them without considering the path of features. One of our recent studies (Mason et al 2004) showed that SCI subjects generated bipolar M R P patterns whose amplitudes were smaller than the ones of able-bodied subjects. A s in the new design, the paths of the features are taken into account, we are incorporating more information during the movement and thus the trials that were missing before can be detected and the improvements are higher for SCI subjects. While the performance o f the subjects does differ significantly between L F - A S D - V 4 and L F -A S D - V 5 , it does not differ between able-bodied and disabled subjects in L F - A S D - V 5 . This suggests L F - A S D - V 5 gives Consistent results for both subject groups unlike L F - A S D - V 4 , in which able-bodied subjects performed better than SCI subjects. This is an important point that needs to be verified statistically with more subjects. In summary, this work has succeeded in decreasing the error rates of our current BI design. Although this forms a substantial decrease in the error rate, practical experience indicates that further system improvements are still desirable. Several ideas emerged from the results that may be studied in future investigations. For example, as mentioned in Section 6.3, the exact time that a subject attempted the movement was not known accurately. A s this time was user dependent, the subject did not necessarily attempt the movement at T E M which is one second after the cue. In this study to train the classifier, the features in the path space at the T E M were selected even though this time may not have been the actual time of the attempted movements. Thus, these features may not have been good representatives of the actual movement attempt. We observed that using these features resulted in no improvement for subject CS . Thus, we carried out another approach to generate the training data to train the classifier. A s mentioned in Section 6.2.1, to generate the training data for the IC state, the features at the T E M were picked. In the new procedure, we picked more features around T E M for training the classifier. Specifically, we picked the features at t=TEM-l, TEM, and TEM+1 and then trained the classifier with the new training set for subject CS . The results showed true positive percentage improvement of approximately 5%. We also applied the same procedure for subject ID, but the results did not improve. Statistical classifiers are dependent for their accuracy on the quality of the training data as much as on the algorithm used for classification. For useful results to be obtained, the 167 training data set must be representative of the whole area to be classified. This preliminary analysis on the data of two subjects implies the need for a method that selects the most suitable training data to train the classifier. We wi l l look into this issue in our future research. In this study, the output of the system was blocked when the eye-blinks were present. Uti l iz ing the path of the features may nullify our need for an eye-blink detector in our BI system. Thus, our future work wi l l explore the system performance and robustness when the output of the system is not blocked due to eye-blinks. This study was an offline evaluation of the proposed L F - A S D - V 5 design; however, online feedback experiments are also needed to confirm the findings of this study. The use of different feature extraction methods, self-learning classification schemes and customization of the L F - A S D Feature Generator parameters are in the scope of our future directions as well . 6.6. References Birch G E , Bozorgzadeh Z and Mason S G 2002 Initial on-line evaluation of the L F - A S D brain-computer interface with able-bodied and spinal-cord subjects using imagined voluntary motor potentials IEEE Trans. Neural Syst. Rehab. Eng. 10 219-224. Birch G E , Lawrence P D and Hare R D 1993 Single trial processing o f event related potentials using outlier information IEEE Trans. Biomed. Eng. 40 59-73. Borisoff J F, Mason S G , Bashashati A and Birch G E 2004 Brain Computer Interface (BCI) Design for Asynchronous Control Applications: Improvements to the L F - A S D Asynchronous Brain Switch IEEE Trans. Biomed. Eng. 5 1 985-92. Graimann B , Huggins J E , Levine S P and Pfurtscheller G 2004 Toward a direct brain interface based on human subdural recordings and wavelet-packet analysis IEEE Trans. Biomed. Eng. 5 1 954-62. Kohonen T 1990 The self-organizing map Proc. of IEEE 78 1464-1480. Levine S P, Huggins J E , Bement S L , Kushwaha R, Schuh L A , Rohde M M , Passaro E A , Ross D A , Elisevich K V and R. J. Smith 2000 A direct brain interface based on event-related potentials IEEE Trans. Rehab. Eng. 8 180-5. 168 Mason S G , Bashashati A , Fatourechi M and Birch G E Developing a Framework for Brain-Interface Technology Design available online at http://www.braininterface.org/. Mason S G and Birch G E 2000 A Brain-Controlled Switch for Asynchronous Control Applications IEEE Trans. Biomed. Eng. 4 7 1297-307. Mason S G and Birch G E 2003 A general framework for brain-computer interface design IEEE Trans. Neural Syst. Rehab. Eng. 11 70-85. Mason S G , Bohringer R, Borisoff J F and Birch G E 2004 Real-time control of a video game with a direct brain—computer interface Clin. Neurophysiol. 2 1 404-8. Mason S G , Moore M M and Birch G E 2003 Designing pointing devices using brain-computer interface technology Proc. of the Ist Int. IEEE EMBS Conf. on Neural Engineering (Capri, Italy) pp 610-13. M i l l a n J R and Mourino J 2003 Asynchronous B C I and local neural classifiers: A n overview of the adaptive brain interface project IEEE Trans. Neural Syst. Rehab. Eng. 11 159-61. Nicolelis M A 2003 Brain-machine interfaces to restore motor function and probe neural circuits Nat. Rev. Neurosci. 4 417- 22. Scherer R, Muller G R, Neuper C, Graimann B and Pfurtscheller G 2004 A n asynchronously controlled EEG-based virtual keyboard: improvement of the spelling rate IEEE Trans. Biomed. Eng. 51 1979-84. Townsend G , Graimann B and Pfurtscheller G 2004 Continuous E E G classification during motor imagery—simulation of an asynchronous B C I IEEE Trans. Neural Syst. Rehab. Eng. 12 258-265. Vaughan T M 2003 Guest editorial brain-computer interface technology: a review of the second international meeting IEEE Trans. Neural Syst. Rehab. Eng. 11 94-109. Wolpaw J R, Birbaumer N , McFarland D J, Pfurtscheller G and Vaughan T M 2002 Brain-computer interfaces for communication and control Clin. Neurophysiol. 1 1 3 767-91. Yom-Tov E and Inbar G F 2003 Detection of movement-related potentials from the electro-encephalogram for possible use in a brain-computer interface Med. Biol. Eng. Comput. 4 1 85-93. 169 Chapter 7 Effects of Eye Blinks on a Self-paced Brain Interface Design20 7.1. Introduction Many Brain Interface (BI) transducer designs have been presented in the literature (for a review of the field, see (Wolpaw et al., 2002; Vaughan, 2003; Mason et al., 2006; Nicolelis, 2003)). Few of them, however, have been designed specifically for self-paced control. The concept of self-paced control of brain interface systems was introduced in (Mason and Birch, 2000) as "asynchronous control". In a self-paced brain interface, the users affect the BI transducer output whenever they want by intentionally changing their brain state. Between periods of intentional control (IC), users are said to be in a no-control (NC) state - they may be idle, daydreaming, thinking about a problem or lunch, or performing any other action, but not trying to control the BI transducer. BI transducers are thus designed to respond only when there is intentional user control and to remain inactive when the user is in a no-control state. In contrast, most B i s operate only during specific periods determined by the system (not the user). This operating paradigm is referred to as synchronous or synchronized control (Mason and Birch, 2000). Although self-paced control is the most natural mode of interaction, it has received less attention. Only a few BI transducers (Mason and Birch, 2000; Levine et al., 2000; Yom-Tov and Inbar, 2003; Birch et al., 1993; Mi l l an and Mourino, 2003; Scherer et al., 2004; Graimann et al., 2004; Townsend et al., 2004; Borisoff et a l , 2004) have been specifically designed and tested for self-paced control. A s recognized by Wolpaw et al (2002), self-paced systems address a problem important for practical applications, i.e. detection of user commands without the timing cues provided by structured trials. In developing a non-invasive brain interface system, the L o w Frequency-Asynchronous Switch Design ( L F - A S D ) was first introduced as a brain interface for self-paced control applications (Mason and Birch, 2000). The L F - A S D seeks to recognize the movement related potentials (MRPs) in the E E G signal. Recent studies with the latest design of the L F - A S D 2 0 A version of this chapter has been published. With kind permission of Springer Science and Business Media. Bashashati A., Noureddin B., Ward R., Lawrence P., and Birch G. (2007) Effects of Eye Blinks on the Performance of a Self-paced Brain Interface, Clinical Neurophysiology 118 1639-47. 170 have demonstrated an average true positive (TP) rate of 66.5% for false positive (FP) rates of 2% (Bashashati et al., 2006 b). To aid the presentation, the latest two designs of the L F - A S D (Bashashati et al., 2006 a; Bashashati et a l , 2006 b) w i l l be referred to as L F - A S D - V 4 and L F - A S D - V 5 , respectively, in this paper. Among the proposed asynchronous BI transducers whose offline performances have been reported in terms of true positives and false positives (Levine et al., 2000; Yom-Tov and Inbar, 2003; Graimann et al., 2004; Townsend et al., 2004), the L F - A S D generates less false positives. A l l these offline studies have reported average false positive rates in the range of 6% to 28% while the average true positive rates were between 73% and 94% (Levine et al 2000, Yom-Tov and Inbar 2003, Graimann et al 2004, Townsend et al 2004). It is difficult however to directly compare the results of the different studies, as the recording equipment, recording and classification protocols, decision rate and interval of time during which false positives could occur, and mental tasks considered are different. In addition, the amount of data involved and the degree of training the subjects received before participating in the BI experiments also vary between studies. In the L F - A S D - V 4 version, which is similar to the original design of the L F - A S D (Mason and Birch 2000), the output state of the system at time th is only determined by the values of the feature vectors at that time. In contrast, the later design of the L F - A S D ( L F - A S D - V 5 ) was introduced to include the past information of the feature vectors in deciding whether or not there is a movement attempt at time f/. Using this information has been shown to improve the BI ' s error rate (Bashashati et al 2006b). (Bashashati et al 2006a) showed that in order to detect the desired M R P pattern, some design parameters of the feature generator should be adjusted for each subject These parameter adjustments were not done for the L F - A S D - V 5 version. Previously, the testing system employed was the same as that used by (Mason and Birch 2000, Borisoff et al 2004). This system automatically detected eye-blinks and did not evaluate the system during these times. In the original version o f the L F - A S D it was believed that eye-blinks generate excessive false activations of the system, and hence E E G data contaminated with eye-blinks were not evaluated. A s the L F - A S D looks for a specific waveform (template) in the E E G signal to detect the presence of movements, eye movements can also easily generate such a template in the E E G signal and cause false activation of the output 171 Table 7.1 shows the details of eye-blink artifacts in our pre-recorded data that were used in this and previous studies (Bashashati et al., 2006 a; Bashashati et al., 2006 b). A s this table shows, 15-34% of the N C state E E G data were contaminated with eye-blinks and 1-21.4% of the trials were rejected due to the presence of eye-blinks (Borisoff et al., 2004; Bashashati et al., 2006 a; Bashashati et al., 2006 b). A s shown in the third row of Table 7.1, a significant amount of data (15-34%) was not evaluated due to the presence of eye-blinks. This reduced the available time that a user could control the B I system. Ideally, a BI system should be operational at any time. This is not the case for this brain interface system, thus reducing its usability. A t the same time, according to the testing system employed, further devices are needed to record the electro-oculogram (EOG), which is used to detect eye-blinks. From the above, it is clear that a strategy that can handle eye-blink artifacts without the need for rejecting them or the need for E O G recording is desired. T A B L E 7.1 DETAILS OF THE EEG D A T A OF THE E V A L U A T I O N SET T H A T W E R E B L O C K E D B Y THE TEST S Y S T E M D U E TO E Y E - B L I N K S . Subject ID CB CS KT AJ LB BK RT Total duration of recorded EEG (minutes) 66 77 61 87 63 81 64 79 Percentage of data contaminated with eye- 17 19 15 28 34 22 19 23 blinks (%) Total number of movement attempts 320 411 294 397 324 415 356 322 Number of movement attempts blocked by the 21 16 12 4 43 25 76 10 test system due to eye-blinks Percentage of movement attempts blocked by 6.6 3.9 4.1 1.0 13.3 6.0 21.4 3.1 the test system due to eye-blinks (%) Since the L F - A S D - V 5 design considers the history of features in decision making and improves the performance of the system, we thought it might handle the E E G data contaminated with eye-blink artifacts better than the previous versions. In such a case, we no longer need to block the decision of the system during eye-blink artifacts. This paper evaluates the performance of the L F - A S D - V 4 (Bashashati et al., 2006 a) and the latest self-paced brain interface design, the L F - A S D - V 5 , (Bashashati et al., 2006 b) when eye-blinks are not excluded from data. In the meantime, the design parameters of the feature generator in (Bashashati et al., 2006 b) are customized according to (Bashashati et al., 2006 a) for each subject. 172 This study measures the performance of L F - A S D - V 5 when eye-blinks are not excluded from the analysis and compares the performance of this design with L F - A S D - V 4 . The performance of the system is evaluated using E E G recordings of attempted finger movements of subjects with spinal cord injuries (SCI) as well as able-bodied subjects who were recorded previously in BI experiments. In Section 7.2 of this paper, a brief description of the self-paced brain interface designs (LF-A S D - V 4 and L F - A S D - V 5 ) , the description of the experiments, and the evaluation procedure are presented. The results and discussions are followed in Sections 7.3 and 7.4, respectively. 7.2. Methods 7.2.1. Brain Interface Designs 7.2.1.1. LF-ASD-V4 Fig . 7.1 shows the block diagram of the L F - A S D - V 4 design, which is similar to the original design of the self-paced brain interface (Mason and Birch, 2000). This design uses features extracted from six bipolar E E G channels (defined with reference to the International 10-20 System at F1-FC1 , F z - F C z , F2-FC2, F C 1 - C 1 , F C z - C z , and FC2-C2) . After amplification, all six E E G channels are normalized with an Energy Normalization Transform (ENT) (Borisoff et al., 2004). A wavelet-like function is applied as the feature generator. The important design parameters of the feature generator are parameters that determine the shape of the subject-specific movement-related potential pattern that needs to be detected and are referred to as "delay parameters", as shown in Fig . 7.2 as a i , aj. The Karhunen-Loeve Transform maps the six-dimensional feature space produced by the Feature Generator to a two-dimensional space. A one-nearest neighbour (1-NN) classifier is used as the feature classifier. The system's classification accuracy is further improved by using a moving average and a debounce block to reduce the number of false switch activations (for details, see (Mason and Birch, 2000; Borisoff et al., 2004)). Overall, every 1/16th of a second, the system classifies input patterns as either no-control (NC) or intentional control (IC). 173 electrode array amp Feature Extractor LF-ASD ENT Feature G enerator F V v F e eture Tr ansl ator 1-NN moving debounce Classifier average Codebook generation mechanism Figure 7.1 Components of the LF-ASD-V4 transducer (Borisoff et al., 2004), where ENT = Energy Normalization Transform, K L T = Karhunen-Loeve Transform, 1 - N N = 1 -Nearest Neighbour, amp = amplifier, and F V V 4 = extracted feature vector. Ej(n) / \ 1 \ Ei(n) f \ t=n Figure 7.2 Desired pattern of the bipolar E E G during movement, where e(n) = amplitude of the signal, Et(n) = amplitude difference between local maximum before the movement and local minimum after the movement, E/n) = amplitude difference between local maximum before the movement and local minimum before the movement, t=n is the time of the expected movement attempt, and otj and aj are the delay parameters. The delay parameters of the feature generator in (Mason and Birch, 2000) were set to be the same for all subjects. In the L F _ A S D - V 4 design, these parameters were customized separately for each subject to increase the confidence that the system is triggered by the desired M R P pattern and not any other unwanted brain activity. 7.2.1.2. L F - A S D - V 5 L F - A S D - V 5 aimed at including the past values of the features in decision making at any specific time. Fig. 7.3 shows the design of the L F - A S D - V 5 (Bashashati et a l , 2006 b). FVV4(tx) is the value of the original features of the L F - A S D - V 4 , L is the length of the buffer, and 174 FVV5(tx)represents the path that the original feature vectors traverse during time. This path represents the history of the original L F - A S D features in the multidimensional. A s shown in Fig. 7.3, the L F - A S D - V 5 has the same overall structure of the L F - A S D - V 4 design (see Section 7.2.1.1) except for the feature extraction block which contains an additional buffer that captures the trajectory of the feature vector in the multi-dimensional space. A s the movement-related potential pattern of each subject may vary in duration, the buffer length (L) is defined separately for each subject. Overall, every 1/16th of a second, the system classifies input patterns as either no- control or intentional control. For more details refer to (Bashashati et al., 2006 b). In (Bashashati et al., 2006 b) the original parameters of the feature extraction block were used for all subjects. However, according to the findings of (Bashashati et al., 2006 a), these parameters should be adjusted for each subject. A s such, in this paper, the parameters of the feature extraction block are customized according to (Bashashati et al., 2006 a). electrode array amp Feature Extractor LF-ASD Feature G enerator ENT KLT Buffer F V v s F eature Translator 1-NN Classifier moving U ounce average Path tempi ate generation mechanism Figure 7.3 Components of the LF-ASD-V5 transducer (Bashashati et al., 2006 a), where ENT = Energy Normalization Transform, K L T = Karhunen-Loeve Transform, 1-NN = 1-Nearest Neighbour, amp = amplifier, F V V 4 = extracted feature vector of LF-ASD-V4, and F V V 5 = extracted feature vector ofLF-ASD-V4. 7.2.2. Experiments The data used in this study were collected from subjects seated 150 cm in front of a computer monitor. The E E G signal was recorded from six bipolar electrode pairs positioned over the supplementary motor area and the primary motor cortex (defined with reference to the International 10-20 System at F1-FC1, Fz -FCz , F2-FC2, F C 1 - C 1 , F C z - C z , and FC2-C2) . Electro-oculographic (EOG) activity was measured as the potential difference between two electrodes, placed at the corner and below the right eye. Eye-blink artifacts were considered present when the difference between the E O G electrodes exceeded ±25 p V . A l l signals were amplified by an E E G amplifier, filtered to a pass-band between 0.1 to 30Hz and sampled at 175 128Hz by a P C equipped with a 12-bit analog to digital converter embedded on a data acquisition board. The subjects participated in this study consisted of four subjects with a high-level spinal cord injury and four able-bodied subjects. A l l subjects were male (except subject K T ) , right handed between 31 and 57 years old. A l l SCI subjects had no residual sensation or motor function in the hands. A l l the subjects provided written consent according to the Behavioral Research Ethics Board ( B R E B ) of the University of British Columbia. The data were collected from the subjects while performing a guided task in 2-minute sessions. These sessions contained both the no-control and intentional control state periods as shown in Fig. 7.4. A t random intervals of 5.6 to 7.0 seconds (mean of 6.7 seconds), a 2cm white circle was displayed on the subject's monitor for % second, prompting them to attempt a movement. In response to this cue, the subject tried to activate the brain switch by attempting to move his right index finger one second after the cue appeared. The one-second delay was used to avoid visual evoked potential effects from the activity. The time, one second after the cue, is called "time of expected (attempted) movement ( T E M ) " . A s the time to perform the movement attempt depends on the user's response, the movement attempt is not necessarily performed at T E M and may change from subject to subject and attempt to attempt. Both subject groups used the same neurological mechanism to drive the brain switch: an attempted right index finger flexion. This resulted in no movement in subjects with high level spinal cord injury, and an actual finger flexion in able-bodied subjects. For each subject, an average of 80 trials was collected every day for 6 days. TEM TEM —I — Time (s) NC state IC state NC state IC state NC state Figure 7.4 Structure of each 2-minute sub-session of E E G recording, where T E M = time of the expected (attempted) movement, N C = no-control, and IC = intentional control. Besides recording the E E G data during movement attempts, the E E G data were also recorded for several 2-minute periods while the person was in a specific no-control state. These 176 sessions contained only no-control state E E G data and were gathered for different no-control states such as attentive eyes opened while looking at a picture on the monitor or doing a search task, and attentive eyes closed. The reason for recording this type of data was to evaluate the performance of the brain interface system for different no-control state periods. 7.2.3. Evaluat ion The designs are evaluated on all the recorded data, i.e., the 2 minute sessions that contain mixed no-control (NC) and intentional control (IC) state data and the 2 minute sessions that include only N C state periods. The E E G data of the mixed N C and IC states contain movement attempts approximately every 6.7 seconds separated by periods of no-control state. The reasons behind the addition of the periods of pure no-control state data were: 1) to more thoroughly evaluate the performance of the system on different types of no-control state data, and 2) to approach the real-world paradigm where intentional control states are separated by longer periods of no-control state. The ability of a subject to control the brain interface system was evaluated by 1) the percentage of correct activations during IC states (true positives, TPs) and 2) the percentage of false switch activations during N C states (false positives, FPs). A state was identified as true positive i f the brain interface system was activated at least once in a T E M window starting at 0.5 seconds before the time of the expected movement ( T E M ) and ending at 1 second after it, a method similar to that employed in other studies (Yom-Tov and Inbar, 2003; Birch et al., 1993; Graimann et al., 2004; Townsend et a l , 2004). A n y activations before the appearance of a white circle and 1 second after the T E M were considered as false positives. Artifact-free E E G data of the first day's recordings were used for training the 1-NN classifier. The E E G data of the subsequent 5 days recordings were used for evaluating the performance o f the brain interface system in each of the following cases (refer to Fig. 7.5): Case 1: Data during ocular artifact presence (OAi, O A 2 and OA3 periods) are not included at all in the analysis. The total length of data (T) is effectively reduced. Case 2: Data during O A i , OA2 and O A 3 are included in the analysis, but the output is always set to inactive, even i f the L F - A S D is activated (e.g., t2 and t 3). A l l the data (T) is considered, but valid L F - A S D activations may be incorrectly suppressed (e.g. t3). In this 177 case, in fact, the output is frozen and neither a hit nor a miss is registered during artifact presence. Case 3: Data during O A i , OA2 and OA3 are included i