UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Face perception : the relationship between identity and expression processing Fox, Christopher James 2008

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata

Download

Media
ubc_2008_fall_fox_christopher.pdf [ 5.8MB ]
[if-you-see-this-DO-NOT-CLICK]
[if-you-see-this-DO-NOT-CLICK]
Metadata
JSON: 1.0066446.json
JSON-LD: 1.0066446+ld.json
RDF/XML (Pretty): 1.0066446.xml
RDF/JSON: 1.0066446+rdf.json
Turtle: 1.0066446+rdf-turtle.txt
N-Triples: 1.0066446+rdf-ntriples.txt
Original Record: 1.0066446 +original-record.json
Full Text
1.0066446.txt
Citation
1.0066446.ris

Full Text

FACE PERCEPTION: THE RELATIONSHIP BETWEEN IDENTITY AND EXPRESSION PROCESSING   by   Christopher James Fox   B.Sc., The University of British Columbia, 2004     A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF   DOCTOR OF PHILOSOPHY   in   The Faculty of Graduate Studies   (Neuroscience)    THE UNIVERSITY OF BRITISH COLUMBIA  (Vancouver)     June 2008     ? Christopher James Fox, 2008   iiABSTRACT  Current models of face perception suggest independent processing of identity and expression, though this distinction is still unclear. Using converging methods of psychophysics and functional magnetic resonance imaging (fMRI) in healthy and patient populations we assessed the relationship between these two perceptual processes. First, using perceptual aftereffects, we explored the neural representations underlying identity and expression. The expression aftereffect only partially transferred across different identities, suggesting adaptation within identity-invariant and identity-dependent expression representations. Contrarily, the identity aftereffect fully transferred across different expressions. This asymmetry cannot be explained through low-level adaptation. The identity-dependent component of the expression aftereffect relies on adaptation to a coherent expression, not low-level features, in the adapting face. Thus adaptation generating the expression aftereffect must occur within high-level representations of facial expression. Second, using fMRI adaptation, we examined identity and expression sensitivity in healthy controls. The fusiform face area and posterior superior temporal sulcus showed sensitivity for both identity and expression changes.  Independent sensitivity for identity and expression changes was observed in the precuneus and middle superior temporal sulcus respectively. Finally, we explored identity and expression perception in a neuropsychological population. Selective identity impairments were associated with inferior occipitotemporal damage, not necessarily affecting the occipital or fusiform face areas. Impaired expression perception was associated with superior temporal sulcus damage, and also with deficits in the integration of identity and expression. In summary, psychophysics, neuroimaging and neuropsychological methods all provide converging evidence for the independent processing of identity and expression within the face network. However, these same methods also supply converging evidence for a partial dependence of these two perceptual processes: in the expression aftereffect, the functional sensitivities of the FFA and pSTS, and identity deficits   iiiobserved in a patient with primarily impaired expression perception and a spared inferotemporal cortex.  Thus, future models of face perception must incorporate representations or regions which independently process identity or expression as well as those which are involved in the perception of both identity and expression.    ivTABLE OF CONTENTS  ABSTRACT...................................................................................................................................ii  TABLE OF CONTENTS ............................................................................................................ iv  LIST OF TABLES....................................................................................................................... ix  LIST OF FIGURES...................................................................................................................... x  ACKNOWLEDGMENTS........................................................................................................... xi  DEDICATION.............................................................................................................................xii  CO-AUTHORSHIP STATEMENT .........................................................................................xiii  1   INTRODUCTION.................................................................................................................... 1  1.1  The face processing network .......................................................................................... 1 1.2  Identity and expression processing within the face network .......................................... 3 1.3 Psychophysics................................................................................................................. 6 1.4 Functional imaging.........................................................................................................9 1.5 Neuropsychology..........................................................................................................12 1.6   Hypotheses.................................................................................................................... 13 1.7 References..................................................................................................................... 16  2   WHAT IS ADAPTED IN FACE ADAPTATION? THE NEURAL REPRESENTATIONS OF EXPRESSION IN THE HUMAN VISUAL SYSTEM............. 19  2.1 Introduction................................................................................................................... 19 2.2  Methods........................................................................................................................ 21 2.2.1 Subjects........................................................................................................................ 21 2.2.2 Stimuli, Experiment 1.................................................................................................. 22 2.2.3 Stimuli, Experiment 2.................................................................................................. 23 2.2.4 Ratings ......................................................................................................................... 24 2.2.5 Apparatus..................................................................................................................... 24 2.2.6 Procedure ..................................................................................................................... 25 2.2.7 Analysis........................................................................................................................ 28 2.3   Results........................................................................................................................... 29 2.3.1 Experiment 1: An identity-independent representation of expression......................... 29 2.3.2 Experiment 2: An identity-dependent representation of expression............................ 33 2.4   Discussion..................................................................................................................... 36 2.5 References..................................................................................................................... 42  3   IT DOESN?T MATTER HOW YOU FEEL. THE FACIAL IDENTITY AFTEREFFECT IS INVARIANT TO CHANGES IN FACIAL EXPRESSION ................ 44  3.1 Introduction................................................................................................................... 44 3.2 Experiment 1................................................................................................................. 46   v3.2.1 Method......................................................................................................................... 47 3.2.1.1 Subjects................................................................................................................. 47 3.2.1.2 Stimuli................................................................................................................... 48 3.2.1.3 Apparatus.............................................................................................................. 50 3.2.1.4 Procedure .............................................................................................................. 50 3.2.1.5 Analysis................................................................................................................. 52 3.2.2 Results.......................................................................................................................... 52 3.2.3 Comment...................................................................................................................... 53 3.3 Experiment 2................................................................................................................. 55 3.3.1 Methods........................................................................................................................ 56 3.3.1.1 Subjects................................................................................................................. 56 3.3.1.2 Stimuli................................................................................................................... 56 3.3.1.3 Apparatus.............................................................................................................. 57 3.3.1.4 Procedure .............................................................................................................. 57 3.3.1.5 Analysis................................................................................................................. 58 3.3.2 Results.......................................................................................................................... 58 3.3.3 Comment...................................................................................................................... 59 3.4 Experiment 3................................................................................................................. 61 3.4.1 Methods........................................................................................................................ 62 3.4.1.1 Subjects................................................................................................................. 62 3.4.1.2 Stimuli................................................................................................................... 63 3.4.1.3 Apparatus.............................................................................................................. 64 3.4.1.4 Procedure .............................................................................................................. 64 3.4.1.5 Analysis................................................................................................................. 65 3.4.1.6 Ideal observer........................................................................................................ 65 3.4.2 Results.......................................................................................................................... 65 3.4.3 Comment...................................................................................................................... 67 3.5 Discussion..................................................................................................................... 67 3.6 References..................................................................................................................... 73  4   FACTORS CONTRIBUTING TO THE ADAPTATION AFTEREFFECTS OF FACIAL EXPRESSION............................................................................................................. 75  4.1 Introduction................................................................................................................... 75 4.2 Methods......................................................................................................................... 78 4.2.1 Subjects........................................................................................................................ 78 4.2.2 Experiment 1................................................................................................................ 78 4.2.2.1 Stimuli................................................................................................................... 78 4.2.2.1.1  Normal-Face condition ................................................................................. 80 4.2.2.1.2 Quartered-Face/Mixed-Expression condition................................................ 80 4.2.2.1.3 Quartered-Face/Consistent-Expression condition ......................................... 83 4.2.2.2 Apparatus.............................................................................................................. 84 4.2.2.3 Procedure .............................................................................................................. 84 4.2.2.4 Analysis................................................................................................................. 86 4.2.3 Experiment 2................................................................................................................ 87 4.2.3.1 Stimuli................................................................................................................... 87 4.2.3.2 Procedure .............................................................................................................. 88 4.2.3.3 Analysis................................................................................................................. 88 4.2.4 Experiment 3................................................................................................................ 88   vi4.2.4.1 Stimuli................................................................................................................... 88 4.2.4.2 Procedure .............................................................................................................. 89 4.2.4.3 Analysis................................................................................................................. 89 4.3 Results........................................................................................................................... 90 4.3.1 Experiment 1................................................................................................................ 90 4.3.2 Experiment 2................................................................................................................ 93 4.3.3 Experiment 3................................................................................................................ 96 4.4 Discussion..................................................................................................................... 98 4.5 References................................................................................................................... 103  5   THE CORRELATES OF SUBJECTIVE PERCEPTION OF IDENTITY AND EXPRESSION IN THE FACE NETWORK: AN FMRI ADAPTATION STUDY............ 105  5.1 Introduction................................................................................................................. 105 5.2 Methods....................................................................................................................... 107 5.2.1 Participants................................................................................................................. 107 5.2.2 Stimuli........................................................................................................................ 108 5.2.3 FMRI data acquisition................................................................................................ 111 5.2.4 Analysis...................................................................................................................... 113 5.2.4.1 Functional localization........................................................................................ 113 5.2.4.2 Adaptation effects............................................................................................... 114 5.3 Results......................................................................................................................... 116 5.3.1 Behavioral data .......................................................................................................... 116 5.3.2 Adaptation ? Core system.......................................................................................... 118 5.3.3 Adaptation ? Beyond the core system ....................................................................... 124 5.4 Discussion................................................................................................................... 126 5.4.1 OFA involvement in the perception of facial structure ............................................. 128 5.4.2 Category effects for identity and expression perception in the FFA and pSTS......... 129 5.4.3 Adaptation in the mSTS and precuneus..................................................................... 130 5.4.4 Implications for models of identity and expression perception................................. 131 5.5 References................................................................................................................... 134  6   DEFINING THE FACE PROCESSING NETWORK: OPTIMIZATION OF THE FUNCTIONAL LOCALIZER IN FMRI ............................................................................... 136  6.1 Introduction................................................................................................................. 136 6.2  Defining face-selective ROIs: Localizers with static or dynamic stimuli .................. 142 6.2.1 Methods...................................................................................................................... 142 6.2.1.1 Participants.......................................................................................................... 142 6.2.1.2 Stimuli................................................................................................................. 142 6.2.1.3 FMRI data acquisition and analysis.................................................................... 143 6.2.1.4 ROI localization and analysis ............................................................................. 145 6.2.2 Results.................................................................................................................147 6.2.2.1 Core System........................................................................................................ 147 6.2.2.2 Extended System................................................................................................. 151 6.3  Defining face-selective ROIs: the statistically optimal cluster size............................ 154 6.3.1 Methods...................................................................................................................... 156 6.3.1.1 Regions-of-interest and analysis......................................................................... 156 6.3.2 Results........................................................................................................................ 158   vii6.4 Discussion................................................................................................................... 163 6.5 References................................................................................................................... 167  7   PERCEPTUAL DEFICITS IN FACIAL IDENTITY AND EXPRESSION PROCESSING........................................................................................................................... 170  7.1 Introduction................................................................................................................. 170 7.2  Experiment 1: Designing the morphed-face discrimination test................................. 173 7.2.1 Methods...................................................................................................................... 173 7.2.1.1 Participants.......................................................................................................... 173 7.2.1.2 Stimuli................................................................................................................. 174 7.2.1.3 Experimental design............................................................................................ 174 7.2.1.4 Procedure ............................................................................................................ 176 7.2.1.5 Analysis............................................................................................................... 177 7.2.2 Results........................................................................................................................ 177 7.3  Experiment 2: Identifying deficits of identity and expression perception in patients 180 7.3.1 Methods...................................................................................................................... 180 7.3.1.1 Subjects............................................................................................................... 180 7.3.1.2 Patient descriptions............................................................................................. 182 7.3.1.3 FMRI:  Localizing the face network................................................................... 186 7.3.1.4 Morphed-face discrimination test ....................................................................... 189 7.3.2 Results........................................................................................................................ 189 7.4 Discussion................................................................................................................... 195 7.5 References................................................................................................................... 202  8   RESIDUAL SENSITIVITY TO IDENTITY AND EXPRESSION CHANGES IN ACQUIRED PROSOPAGNOSIA........................................................................................... 205  8.1 Introduction................................................................................................................. 205 8.2  Experiment 1: Significant adaptation in the single subject......................................... 208 8.2.1 Methods...................................................................................................................... 208 8.2.1.1 Participants.......................................................................................................... 208 8.2.1.2 Stimuli................................................................................................................. 209 8.2.1.3 Experimental design............................................................................................ 211 8.2.1.4 FMRI data acquisition......................................................................................... 211 8.2.1.5 FMRI data analysis ............................................................................................. 213 8.2.2 Results........................................................................................................................ 214 8.3  Experiment 2: Residual sensitivity in a patient population. ....................................... 217 8.3.1 Methods...................................................................................................................... 217 8.3.1.1 Subjects............................................................................................................... 217 8.3.1.2 Patient descriptions............................................................................................. 219 8.3.1.3 FMRI data acquisition and analysis.................................................................... 224 8.3.2 Results........................................................................................................................ 224 8.4 Discussion................................................................................................................... 227 8.4.1 Adaptation in controls................................................................................................ 227 8.4.2 Adaptation in prosopagnosia...................................................................................... 231 8.5 References................................................................................................................... 236     viii9   CONCLUDING CHAPTER ............................................................................................... 238  9.1   Asymmetric dependence in representations of identity and expression..................... 238 9.2  Sensitivity to identity and expression throughout the face network........................... 242 9.3  Tools for testing patient populations........................................................................... 246 9.4  Impairments in identity and expression perception .................................................... 249 9.5   Future directions ......................................................................................................... 255 9.6  Implications for functional recovery........................................................................... 258 9.7 Summary..................................................................................................................... 260 9.8 References................................................................................................................... 262  APPENDIX................................................................................................................................ 267       ixLIST OF TABLES  Table 5.1  Average results from the functional localizer......................................................... 119 Table 6.1  Objects and motions included in the dynamic localizer ......................................... 144 Table 6.2  Core system of face perception as localized by both localizers.............................. 148 Table 6.3  Extended system of face perception as localized by both localizers ...................... 152 Table 7.1  Results from the neuropsychological screening ..................................................... 181 Table 7.2  Core face network localized in the five patients..................................................... 191 Table 8.1  Dynamically localized core system in four patients and controls........................... 215 Table 8.2  Patient results from the battery of face tests........................................................... 218    xLIST OF FIGURES  Figure 2.1  Example of the expression aftereffect trial........................................................... 27 Figure 2.2  Same identity, different identity and verbal expression aftereffects..................... 30 Figure 2.3  Different image, visual/non-facial, and auditory expression aftereffects............. 34 Figure 2.4  Hierarchical representations of facial expression ................................................. 37 Figure 3.1  Example of the identity aftereffect trial................................................................ 51 Figure 3.2  Same expression, different expression and different image identity aftereffects. 54 Figure 3.3  Novel and familiar face identity aftereffects ........................................................ 60 Figure 3.4  Identity and expression discrimination thresholds................................................ 66 Figure 3.5  Asymmetric relationship between identity and expression representations ......... 69 Figure 4.1  Examples of stimuli for examining the expression aftereffect ............................. 79 Figure 4.2  Creation of normal and quartered-face stimuli ..................................................... 81 Figure 4.3  Example of the expression aftereffect trial........................................................... 85 Figure 4.4  Expression aftereffects using quartered-face stimuli............................................ 92 Figure 4.5  Expression aftereffects using thresholded faces ................................................... 95 Figure 4.6  Expression aftereffects using schematic faces...................................................... 97 Figure 5.1   Example morph matrix and adaptation face pairs............................................... 109 Figure 5.2  Behavioral responses recorded during the fMRI session.................................... 117 Figure 5.3  Adaptation effects in the right OFA.................................................................... 120 Figure 5.4  Adaptation effects in the right FFA .................................................................... 121 Figure 5.5  Adaptation effects in the right pSTS................................................................... 123 Figure 5.6  Adaptation effects in the right mSTS.................................................................. 125 Figure 5.7  Adaptation effects in the precuneus.................................................................... 127 Figure 6.1  FMRI localized core system of face perception ................................................. 137 Figure 6.2  FMRI localized extended system of face perception.......................................... 138 Figure 6.3   Comparison of static and dynamic localizers in the core system ....................... 150 Figure 6.4  Comparison of static and dynamic localizers in the extended system................ 153 Figure 6.5  Face selectivity as a function of cluster size in the core system......................... 159 Figure 6.6  Average cluster size with maximal face-selectivity in the core system.............. 161 Figure 6.7  Comparison of different thresholding methods for ROI localization................. 162 Figure 7.1  Sample trials from the morphed-face discrimination test................................... 175 Figure 7.2  Control data from the morphed-face discrimination test.................................... 179 Figure 7.3  Structural MRI of the five patients ..................................................................... 183 Figure 7.4  FMRI images of the core face network as localized in the five patients............ 192 Figure 7.5  Results of the morphed-face discrimination test for the five patients ................ 193 Figure 8.1  Examples of fMRI adaptation trials.................................................................... 210 Figure 8.2  Single subject adaptation effects in controls....................................................... 216 Figure 8.3  Structural MRI of the four prosopagnosic patients............................................. 220 Figure 8.4  Localized core system and adaptation effects in prosopagnosic B-IOT1........... 225 Figure 8.5  Localized core system and adaptation effects in prosopagnosic R-IOT1........... 226 Figure 8.6  Localized core system and adaptation effects in prosopagnosic B-AT1............ 228 Figure 8.7  Localized core system and adaptation effects in prosopagnosic R-AT1............ 229       xiACKNOWLEDGMENTS  I would like to express my deep thanks to Dr. Jason Barton for all the support he has given me throughout my time in his lab. His willingness to let me try, his patience, and valuable guidance have made these years a truly worthwhile experience.  Thanks to all those who have contributed to the work presented in this thesis, Andrea Butler, Dr. Bradley Duchaine, Hashim Hanif, Dr. Giuseppe Iaria, Dr. So Young Moon, and Dr. Ipek Oruc. None of this work would have been possible without each of your significant contributions.   Thanks are also due to Dr. David Katz, who assisted in the initial creation of our neuropsychological test battery, and also provided important training in the administration of neuropsychogical tests.  Thanks are also in order for all the other members of the Human Vision and Eye Movement Laboratory, the staff at the UBC High Field MRI Centre, and all those who participated in these experiments.  I would also like to acknowledge the support and guidance provided by members of my thesis committee, Dr. James Enns, Dr. Dean Foti, and Dr. Deborah Giaschi, as well as members of my comprehensive examination committee, Dr. Todd Handy and Dr. Joanne Matsubara.  Additionally, I would like to recognize the financial support from The National Institutes of Health, The Canadian Institutes of Health Research, and The Michael Smith Foundation for Health Research, which was provided to me personally and additionally to the studies described herein.  Finally, I express my sincerest gratitude to all those patients who gave of their time to contribute to these studies. It is hard for anyone to appreciate the difficulties you experience as a result of your problems with face recognition. It is our shared hope that your efforts will contribute to valuable therapies for yourself and others in the future.   xiiDEDICATION  To those who have sacrificed the most to get me where I am today  My always supportive parents Roy and Hilda Fox  and  My dear wife Amy  Thank-you   xiiiCO-AUTHORSHIP STATEMENT  A number of other authors have contributed to the studies comprising Chapters 2-8 of this thesis.   I had the primary role for experimental design, data collection, analysis and manuscript preparation for the work described in Chapter 2.  I had the primary role for experimental design, data collection, analysis and manuscript preparation for the work described in Chapter 3. Data collection and analysis of discrimination thresholds using the Quest procedure (Experiment 3) was performed by Dr. Ipek Oruc.  I, along with Dr. Ipek Oruc, was involved in the experimental design, analysis, and manuscript preparation of the work described in Chapter 4. The primary responsibility of data collection and initial manuscript preparation fell to Andrea Butler.   I had the primary role for experimental design, data collection, analysis and manuscript preparation for the work described in Chapter 5. Dr. So Young Moon provided important support in the initial experimental design and troubleshooting associated with the fMRI paradigm. Dr. Giuseppe Iaria provided help in the analysis of collected data.  The design of the dynamic localizer, as described in Chapter 6, was primarily performed by Dr. Giuseppe Iaria. Data collection and manuscript preparation was a shared responsibility. Data analysis was primarily my responsibility, and all work related to Part 2 (optimal ROI size) was again my responsibility.  I had the primary role of designing the morphed-face discrimination test described in Chapter 7. Control data was collected and analyzed by Hashim Hanif. Recruitment of patients R-IOT1, and R-AT1 was performed by Dr. Bradley Duchaine, with others recruited by Dr. Jason Barton. FMRI data collection, and neuropsychological testing, was a shared responsibility with Dr. Giuseppe Iaria. All patient analyses and manuscript preparation was primarily my responsibility.  I had the primary role in designing, analyzing and preparing the manuscript presented in Chapter 8. I shared the responsibility for data collection and neuropsychological testing with Dr. Giuseppe Iaria. Again patients R-IOT1 and R-AT1 were recruited by Dr. Bradley Duchaine, with others recruited by Dr. Jason Barton.  Dr. Jason Barton provided a supervisory role in the experimental design, data collection, analysis, and manuscript preparation of all the work described in Chapters 2-8.     11   INTRODUCTION    1.1  The face processing network What?s in a face? A simple question, but as it turns out, a not so simple answer. In fact, a whole wealth of information is contained within every face, ranging from the consistent structural characteristics that enable recognition of a unique facial identity to the rapidly changing and often subtle arrangements of facial features that represent different facial expressions and allow an insight into that person?s emotional state. Today, complex cognitive and anatomic models have been developed to integrate the many different cognitive processes and cortical regions that underlie the extraction and interpretation of facial information (Bruce and Young, 1986; Gobbini and Haxby, 2007; Haxby, Hoffman et al., 2000), yet this has not always been the case.   The first evidence for face perception as a distinct component of the human visual system came from studies of neuropsychological patients who displayed a selective impairment in facial recognition (Bodamer, 1947). Prosopagnosia, as this impairment is now known, cannot be explained by general problems with vision, intelligence, memory or other cognitive functions, and thus appears to be the result of damage to neural regions selectively involved in the perception of faces (Barton, 2003). One candidate region may be located in the inferior temporal cortex, on the lateral fusiform gyrus; an area known as the fusiform face area (FFA) which responds more strongly to faces than objects in functional magnetic resonance imaging (fMRI) studies (Kanwisher, McDermott et al., 1997). Not surprisingly, lesion studies had long pointed to the inferior occipitotemporal cortex as critical for face processing, with prosopagnosia often developing after bilateral (Damasio, 1985; Meadows, 1974) or even unilateral right hemispheric damage to this region (de Renzi, 1986; Landis, Cummings et al., 1986). Again, this finding is echoed in the fMRI literature where more robust face-related activity is commonly seen in the right hemisphere version of the FFA (Kanwisher, McDermott et al., 1997).   2  However, to complicate the story, a number of cases of prosopagnosia have been reported with atypical lesions affecting only the anterior temporal lobes (Barton, Press et al., 2002; Evans, Heggs et al., 1995), the left occipitotemporal cortex (Mattson, Levin et al., 2000; Tzavares, Merienne et al., 1973), or even ones in which sparing of the right FFA has been documented (Rossion, Caldara et al., 2003). In addition, different forms of prosopagnosia have also been reported, with some patients exhibiting problems in the formation of a facial percept and others who form an accurate facial percept but exhibit problems associating this percept with facial memory stores (de Renzi, Faglioni et al., 1991). A similar pattern of apperceptive and associative deficits is also seen in other visual agnosias (Lissauer, 1890). Furthermore, beyond prosopagnosia, there are also reports of patients with trouble recognizing facial expressions, but relatively unaffected identity recognition (Adolphs, Damasio et al., 1996; Adolphs, Tranel et al., 1994; Kurucz, Feldmar et al., 1979; Young, Newcombe et al., 1993).    In order to explain the heterogeneity of face perception deficits a cognitive model was developed which included several distinct processes deemed integral to normal face recognition (i.e. ? formation of a facial percept, facial memory stores, facial expression processing etc. (Bruce and Young, 1986)). Like the expansion of the cognitive model, from a unitary process of face perception to one containing several distinct processes working together, the anatomic model of face perception has expanded from a single face-selective cortical region, the FFA, to a network of regions involved in the processes underlying face perception (Gobbini and Haxby, 2007; Haxby, Hoffman et al., 2000; Ishai, Schmidt et al., 2005; Kanwisher, McDermott et al., 1997). In addition to the FFA two other cortical regions have demonstrated face-selectivity (i.e.-more responsive to faces than objects); the occipital face area (OFA), located on the inferior occipital gyrus, and a portion of the posterior superior temporal sulcus (pSTS). Both the OFA and pSTS were identified in the initial report of face-selective cortical regions (Kanwisher, McDermott et al., 1997) and, due to their consistent identification (Ishai, Schmidt et al., 2005),   3have been incorporated along with the FFA as the ?core system? of face processing (Haxby, Hoffman et al., 2000).    In addition to the core system are a number of cortical regions, not exclusively activated by faces, but which do play an important role in processes related to facial recognition; the ?extended system? of face processing (Haxby, Hoffman et al., 2000). Included in the extended system are the anterior temporal lobes which store semantic memories and names associated with faces (Douville, Woodard et al., 2005; Gobbini and Haxby, 2007; Snowden, Thompson et al., 2004; Thompson, Graham et al., 2004; Tsukiura, Fujii et al., 2002; Tsukiura, Mochizuki-Kawai et al., 2006), the amygdalae which are involved in the emotional response and in the storage of emotion memory (Adolphs, Damasio et al., 1996; Brierley, Medford et al., 2004), the precuneus which is involved in the discrimination of familiar and novel faces (and other stimuli) (Gobbini and Haxby, 2006; Kosaka, Omori et al., 2003), and the anterior paracingulate and inferior frontal gyri which may be involved in interpreting the intentions of others (Gobbini and Haxby, 2007). Also, while not explicitly modeled in the extended system (Gobbini and Haxby, 2007; Haxby, Hoffman et al., 2000), there is also evidence for a secondary region on the superior temporal sulcus, more anterior than the classically modeled pSTS, which is also selectively activated by faces (Winston, Henson et al., 2004). Working together, regions of the core and extended systems enable the accurate face recognition which we so often take for granted.       1.2  Identity and expression processing within the face network One of the most striking features of both cognitive and anatomic models of face perception is the parallel nature of processing. With a number of patient studies reporting selective deficits in identity and expression perception (Bodamer, 1947; Kurucz, Feldmar et al., 1979; Meadows, 1974), Bruce and Young (1986) suggested that identity and expression perception may in fact proceed independently. This proposal has since been incorporated into current anatomic models,   4wherein the FFA has been attributed the perception of temporally invariant facial structure (i.e.- facial identity) and the pSTS the perception of dynamically changing aspects of the face (i.e.- facial expression) (Gobbini and Haxby, 2007; Haxby, Hoffman et al., 2000). Indeed, further neuropsychological reports of brain damaged patients (Adolphs, Damasio et al., 1996; Adolphs, Tranel et al., 1994; Barton, 2003; Duchaine, Parker et al., 2003; Young, Newcombe et al., 1993) have continued to provide potential evidence for a double dissociation between these perceptual processes, though the anatomical distinction in causal lesions is not always clear. In addition, the more advanced method of fMRI adaptation, a technique which examines neural sensitivity to facial changes, has been used to demonstrate FFA sensitivity for identity changes (Andrews and Ewbank, 2004; Rotshtein, Henson et al., 2005), and pSTS sensitivity for dynamic facial changes such as facial viewpoint (Andrews and Ewbank, 2004), eye gaze (Pelphrey, Singerman et al., 2003), and eye and mouth movements (Puce, Allison et al., 1998).   However, the evidence against a complete independence of identity and expression perception is growing. A number of psychophysical studies have demonstrated interference patterns whereby irrelevant changes in facial identity affect performance on tests of expression discrimination (Ganel, Goshen-Gottstein et al., 2004; Schweinberger, Burton et al., 1999; Schweinberger and Soukup, 1998), a finding which suggests, at some level, shared resources for identity and expression perception. Another study, in which celebrity faces were identified, showed that responses were speeded when celebrities displayed a characteristic facial expression of slight happiness, an effect not replicated in novel faces (Kaufmann and Schweinberger, 2004). This finding raises the possibility that identity representations of familiar faces may in fact be encoded with a characteristic facial expression. Additionally, fMRI studies assessing the functional characteristics of regions comprising the face network have provided further evidence against complete independence in the processing of facial identity and expression. Within the FFA, a region thought to independently process facial identity (Haxby, Hoffman et al., 2000),   5expressive faces produce a more robust response than neutral faces (Ishai, Schmidt et al., 2005; Vuilleumier, Armony et al., 2001), and the FFA even displays a sensitivity for changes in facial expression (Ganel, Valyear et al., 2005). Within the pSTS, a region thought to independently process facial expression (Haxby, Hoffman et al., 2000), sensitivity to facial expression changes has been demonstrated, but the same study also shows even more pronounced sensitivity to changes in facial identity within the same region (Winston, Henson et al., 2004). These data begin to question the independent model of identity perception in the FFA and expression perception in the pSTS.   Another piece of evidence which disputes the complete independence of identity and expression processing comes from a principle component analysis of facial images (Calder, Burton et al., 2001). In this analysis, the important components, or facial characteristics, required for identity and expression discriminations are determined. Certain components were identified which were only important for identity discriminations, and others which were only important for expression discriminations, a result which may appear to support the independent model (Calder, Burton et al., 2001). However, in addition to these identity- or expression-selective components were a number of components important for both identity and expression discriminations (Calder, Burton et al., 2001). This result has led to the proposal of a ?relative segregation? in identity and expression processing, wherein Calder and Young (2005), suggest that a similar pattern may be seen in the cortical face network, with some regions independently involved in the processing of identity or expression, but with additional regions critical for both aspects of face perception. The present thesis will examine the relationship of identity and expression perception and, using converging methods of psychophysics and fMRI in healthy controls and neuropsychological patient populations, determine whether this relationship is more suited to an independent model of identity and expression perception (Haxby, Hoffman et al.,   62000) or to a ?relative segregation? model (Calder and Young, 2005) wherein identity and expression perception exhibit some degree of interdependence.    1.3  Psychophysics The first method we used to probe the relationship between identity and expression perception, was that of visual adaptation and perceptual aftereffects. Visual adaptation is the process by which neural representations that encode certain aspects of a stimulus adapt, or reduce their activity, in response to prolonged exposure to the stimulus. Following visual adaptation, when presented with an ambiguous stimulus, the subject now displays a perceptual bias away from the adapting stimulus; the perceptual aftereffect. The most familiar example of a visual aftereffect is that generated by adaptation to color (Allan, Siegel et al., 1997; Nieman, Hayashi et al., 2005), wherein adapted photoreceptors bias the perception of an ambiguous white background towards the color opposite of that just adapted (i.e.- yellow perception after blue adaptation). Though the color aftereffect is retinally-based, there have been demonstrations of cortically-based aftereffects such as those following adaptation to facial identity (Leopold, O'Toole et al., 2001) or facial expression (Webster, Kaping et al., 2004). In this form of visual adaptation a face is presented for a few seconds, during which time neural representations encoding that face adapt (i.e.-reduce, sharpen, or facilitate their activity; see (Grill-Spector, Henson et al., 2006)), and this is followed by the brief presentation (<1s) of an ambiguous test face, created by morphing two or more facial images, on which subjects are required to make a judgment (Leopold, Rhodes et al., 2005). The perception of the ambiguous test face, like that of the white background in color adaptation, is now biased away from the adapted face. For example, after adapting to a disgusted face, the subsequent viewing of a 50/50% morph between a disgusted and surprised face would most likely result in the perception of surprise in this face; adapting to a surprised face would conversely shift the perception of the same ambiguous morph face towards disgust (Webster,   7Kaping et al., 2004). The nature of the adapting representation can be probed by manipulating the relationship between adapting and test face.    In previous reports of face aftereffects, adapting faces have been used to create the ambiguous test faces (Leopold, O'Toole et al., 2001; Webster, Kaping et al., 2004). Such a procedure makes it impossible to determine the nature of the adapting representation. For instance, one might explain the aftereffects in terms of adaptation to low-level aspects of the image such as feature orientation or shape. As these aspects of the image will be morphed along with any changes in identity and expression there is no way to parse the effects of low-level adaptation from those of high-level adaptation in a neural representation of the face itself. One way to determine whether the aftereffects do in fact originate in the adaptation of face representations is by changing the adapting stimulus. For instance if one can still observe an expression aftereffect (as described above) when the adapting face is a different identity than the test face, then any low-level explanations can be ruled out. Adaptation must be occurring in a high-level representation of facial expression. Furthermore, the nature of such a representation would be a general representation of expression, which is not image- or identity-specific, as the aftereffect is shown to transfer across different identities. Another important question would be whether we could see evidence for an expression representation which is identity- but not image-specific. Again, manipulating the adapting stimulus can provide a means for addressing this question. One can compare the aftereffect in conditions where adapting and test stimuli are congruent in terms of identity and image, where they are congruent in terms of identity but not image, and where they are incongruent in both identity and image. The differential pattern of aftereffects seen in these conditions can delineate the nature of neural representations underlying the expression aftereffect, and determine whether there is evidence for identity-dependence within representations of facial expression.    8  In Chapter 2 we perform just such an experiment, and examine the nature of expression representations underlying the expression aftereffect. Central to the present thesis, we explore the interaction between facial identity and expression on the expression aftereffect, and seek for any evidence of shared resources for these two perceptual processes. In addition, we also examine the extent to which a general representation of expression can be activated by non-face stimuli. We use emotional words, emotional non-face images, and emotional voices as adapting stimuli.  The ability of any of these stimuli to induce an expression aftereffect on the subsequently viewed test faces would suggest that the adapting representation is not face-specific but is in fact a more general representation of expression or emotion.   In Chapter 3 we perform an analogous experiment to examine the nature of identity representations underlying the identity aftereffect. Similar to the expression aftereffect, the identity aftereffect can be measured by adapting to a particular face and performing an identity discrimination on a morphed face with ambiguous identity (Leopold, O'Toole et al., 2001). As in the experiment described in Chapter 2 we can vary the adapting stimuli to create conditions where adapting and test stimuli are congruent in terms of expression and image, where they are congruent in terms of expression but not image, and where they are incongruent in both expression and image. Importantly, evidence for an identity aftereffect which is expression- but not image-specific would suggest a shared representation, encoding both identity and expression, within the human visual system. Another interesting component of facial identity is the difference in representations of novel and familiar identities, with representations of familiar identities potentially more robust and similar to representations of facial expressions. In fact there have been reports of interactions between identity and expression for familiar but not novel faces (Ganel, Goshen-Gottstein et al., 2004; Kaufmann and Schweinberger, 2004). Therefore we will examine the identity aftereffect, and any interaction with facial expression, using both novel and familiar faces.   9  Finally, in Chapter 4, we will perform a more detailed analysis of the expression aftereffect. As described above, one method to discriminate low-level and high-level adaptation is to determine whether an aftereffect can transfer across different images of the same person (i.e.- identity- but not image-specific). However, a criticism of this method can still be the high similarity between different pictures of the same person. Thus, the influence of low-level adaptation, while diminished, may not be completely ruled out. Another way to address this issue is by designing adapting stimuli which share all low-level image properties with the test stimuli (as in the congruent identity and image condition), but which either contain a coherent facial expression or do not. Coherence in facial expression can be disrupted by jumbling facial features or by displaying multiple expressions in the same adapting stimuli (i.e.- right half of face is angry, while left half of face is afraid). Adaptation in the coherent expression condition, but not in the incoherent expression condition would provide strong evidence that the aftereffect is not the result of low-level feature adaptation, but is in fact adaptation of a neural representation of the face itself. Such a distinction is important to ensure that any interaction between facial identity and expression occurs at the level of the face-representation and not at point earlier in the visual system.    1.4  Functional imaging The second method we used to probe the relationship between identity and expression processing is that of fMRI, which can be a powerful tool for assessing the sensitivity of neural populations to different aspects of the facial stimulus (Andrews and Ewbank, 2004; Rotshtein, Henson et al., 2005; Winston, Henson et al., 2004). As described above (Section 1.3), prolonged exposure to a visual stimulus will result in reduced neural activity within regions encoding that stimulus. This reduction in activity to prolonged, or simply repeated stimuli, can be measured as a decrease in the blood oxygen level dependent (BOLD) signal in regions encoding that stimulus, a process   10known as fMRI adaptation (Grill-Spector, Henson et al., 2006). The repetition of identical stimuli will result in general adaptation throughout the visual system and face network as all aspects of the visual stimulus have been repeated (Winston, Henson et al., 2004). However, selectivity within the face-network can be assessed by repeating only certain aspects of the facial stimulus in the second presentation. For example, the second face image could have the same identity but different expression to the first, resulting in adaptation in regions encoding facial identity, but a release from adaptation in regions encoding facial expression. This method has been used to demonstrate selective adaptation to identity in the FFA, and adaptation to both identity and expression in the pSTS (Winston, Henson et al., 2004). Another fMRI adaptation study demonstrated differential sensitivities in the occipital and fusiform face areas, with the OFA showing a release from adaptation to any structural change in a face, whereas the FFA required a change across a categorical boundary between identities to initiate a release from adaptation (Rotshtein, Henson et al., 2005). Thus the FFA is not only sensitive to changes in identity, but is also involved in the categorical perception of identity (Rotshtein, Henson et al., 2005).   Using a similar method, in Chapter 5, we sought to examine the categorical perception of identity and expression throughout the face network. Although there has been some evidence for sensitivity to identity (Andrews and Ewbank, 2004; Rotshtein, Henson et al., 2005) and expression changes (Ganel, Valyear et al., 2005) in the FFA, and also the pSTS (Winston, Henson et al., 2004), there is little evidence for their role in the categorical perception of these changes. Furthermore, the majority of these and other studies require subjects to perform irrelevant tasks (Rotshtein, Henson et al., 2005; Winston, Henson et al., 2004), yet there is evidence that these regions are modulated by attention (Palermo and Rhodes, 2007), with some effects even driven by the subject?s perception, not the stimulus itself (Grill-Spector, Knouf et al., 2004). Thus, we had subjects perform relevant tasks of identity and expression discrimination   11during fMRI adaptation scans. Because of this design, we were able to analyze each region in the face network and determine its role in the discrimination of facial identity and expression. Such a method provides a stronger link between neural activity and behavior and can provide more evidence for any potential region involved in the perception of both identity and expression.   Following this adaptation study in healthy subjects, our goal was to use fMRI to characterize the face network in patients with lesions (see below). One important shift when transitioning from studies in healthy subjects to studies in patients with varying lesions is the need to develop a reliable means of localizing the face network in single subjects rather than a group of subjects. Because prior methods did not consistently reveal all core regions of the face network in all healthy subjects, interpreting the absence of activation in a patient is problematic. Therefore, we performed a preliminary study in healthy subjects aimed at improving the sensitivity and reliability of the localizer used to identify regions of the face network.   The identification of face processing regions was initially achieved by contrasting activity when viewing static images of faces with activity when viewing static images of objects (Kanwisher, McDermott et al., 1997). Today, many studies employ a functional localizer that uses the same face minus object contrast to identify regions-of-interest (ROI) in the face network and then perform experimental analyses within these localized ROIs (Andrews and Ewbank, 2004; Schiltz, Sorger et al., 2006; Yovel and Kanwisher, 2005). However, as noted in the original study (Kanwisher, McDermott et al., 1997) and subsequent studies (Ishai, Schmidt et al., 2005) including our own (Chapter 5), the standard static localizer is not fully reliable in identifying face-related ROIs across subjects. As a result, in Chapter 6, we design a new functional localizer that uses dynamic stimuli as opposed to the traditional use of static stimuli, resulting in a more ?life-like? or natural functional localizer. It was our hypothesis that the use of dynamic stimuli would result in a more robust and reliable functional localizer. A reliable functional localizer is imperative for examining patient populations, as it decreases the   12possibility of failed localizations and subsequently false claims of functional damage to the face network.      1.5  Neuropsychology The final method we used to probe the relationship between identity and expression processing was the lesion study. Patient populations provide important data for the construction of any cognitive model, and indeed the potential double dissociation in identity and expression impairments has helped to shape the independent processing of current models (Bruce and Young, 1986; Haxby, Hoffman et al., 2000). However, the anatomical heterogeneity of causal lesions in identity (Barton, 2003; Meadows, 1974; Takahashi, Kawamura et al., 1995) and expression (Adolphs, Damasio et al., 1996; Kurucz, Soni et al., 1980; Young, Newcombe et al., 1993) deficits makes it difficult to associate perceptual impairments with specific anatomical substrates. Furthermore, there is an inherent discrepancy in the difficulty of tests of identity and expression perception with identity tests requiring the knowledge of many different identities (Barton, Cherkasova et al., 2001), while expression tests only require the knowledge of a few different expressions (Kurucz and Feldmar, 1979).    In Chapter 7 we used morphed faces to design a test of identity and expression perception balanced for level of difficulty. The balanced nature of our morphed-face discrimination test makes it an appropriate tool for examining potential dissociations of identity and expression deficits in patient populations. We recruited five patients for this study: two with inferior occipitotemporal lesions likely affecting the right OFA and FFA; two with anterior temporal lesions likely sparing the core system of face processing; and one with a lesion of the superior temporal sulcus that potentially affected the right pSTS. Patients performed a neuropsychological screening and battery of face tests to determine the selectivity of their deficits, and were then given the morphed-face discrimination test to examine any potential dissociation. Results from   13these tests were correlated with data from the fMRI localizers, which identified any functional damage and/or spared cortical regions (Chapter 6).   Finally, in Chapter 8, we examine spared regions of the face network, in four patients with acquired prosopagnosia. We looked for evidence of any residual sensitivity to identity or expression changes in these spared regions. Relatively few studies have asked this question, with one study reporting an absence of identity sensitivity in the FFA of a prosopagnosic (Rossion, Caldara et al., 2003), and another reporting its presence in the FFA of four patients with the congenital form (i.e.- no visible brain damage) of prosopagnosia (Avidan, Hasson et al., 2005). However, both of these studies are faced with the problem of determining statistical significance in the single case, with one group visually comparing their patient data to patterns of sensitivity seen in controls (Rossion, Caldara et al., 2003) and the other averaging across their small patient sample (Avidan, Hasson et al., 2005). Neither of these methods is optimal, and as a result we designed a method for assessing statistically significant sensitivity in the single subject by averaging multiple adaptation scans within the same individual. This method was piloted in a small group of controls and proved effective. Next, we presented our single subject adaptation method to the four prosopagnosic patients, and probed any spared regions of the face network for residual sensitivity to identity and/or expression changes.    1.6   Hypotheses In the present thesis, we employ a number of different techniques to assess the relationship between identity and expression processing within the human visual system. These complementary methods of psychophysics, functional imaging, and neuropsychology can provide converging evidence, thereby strengthening any conclusions made concerning the relationship between identity and expression processing, beyond that which is possible through the use of one method alone.    14  First, in Chapters 2-4, we used psychophysics to examine the neural representations underlying identity and expression perception. We hypothesized the existence of a neural representation of expression that generalizes across different identities, and used the expression aftereffect to explore this possibility in Chapter 2. In Chapter 3 we used the identity aftereffect to explore a similarly hypothesized neural representations of facial identity (i.e.- one that generalizes across different facial expressions). In Chapter 4 we returned to the expression aftereffect, hypothesizing it to be a result of adaptation within a high-level neural representation of facial expression; one which is dependent upon a coherent expression in the adapting face.   Second, in Chapters 5-6, we used functional imaging to explore the cortical network underlying identity and expression perception. Using fMRI adaptation we examined the role of a number of regions in the face network in the perception of identity and expression. In Chapter 5, we tested the hypothesis of the OFA as an input to the face network with no role in the perception of identity or expression (Haxby, Hoffman et al., 2000). Additionally we tested the hypothesis of independent identity perception in the FFA, and independent expression perception in the pSTS (Haxby, Hoffman et al., 2000) by examining the sensitivity to identity and expression changes within these two cortical regions. In Chapter 6, we developed a functional localizer using dynamic stimuli and hypothesized more sensitive and reliable localization of the face network when using this dynamic localizer.    Third, in Chapters 7-8, we used neuropsychological populations to examine the relationship between identity and expression perception in the damaged brain. We designed a balanced test of identity and expression perception, and presented this test to five neuropsychological patients in Chapter 7. By correlating performance on this test with results from fMRI localization of the face network we were able to test the hypothesis of independent identity deficits following damage to the right FFA and independent expression deficits following damage the right pSTS, a double dissociation (Haxby, Hoffman et al., 2000). In   15Chapter 8 we looked for residual sensitivity in the spared face network of prosopagnosic patients. Again we tested the hypothesis of independent identity processing in the FFA and independent expression processing in the pSTS (Haxby, Hoffman et al., 2000) by examining these regions for residual sensitivity to either facial change.    The main thrust of this thesis then is to use these several complimentary methods to study face perception and, in particular, the relationship between identity and expression processing. Our conclusions will be based on the converging evidence, garnered by these various techniques, for either an independent (Haxby, Hoffman et al., 2000) or ?relative segregation? (Calder and Young, 2005) model of identity and expression processing.  16  1.7 References ADOLPHS R, DAMASIO H, TRANEL D, and DAMASIO AR. Cortical systems for the recognition of emotion in facial expressions. J Neurosci, 16: 7678-7687, 1996. ADOLPHS R, TRANEL D, DAMASIO H, and DAMASIO A. Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala. Nature, 372: 669-72, 1994. ALLAN LG, SIEGEL S, KULATUNGA-MORUZI C, EISSENBERG T, and CHAPMAN CA. Isoluminance and contingent color aftereffects. Percept Psychophys, 59: 1327-34, 1997. ANDREWS TJ and EWBANK MP. Distinct representations for facial identity and changeable aspects of faces in the human temporal lobe. Neuroimage, 23: 905-13, 2004. AVIDAN G, HASSON U, MALACH R, and BEHRMANN M. Detailed exploration of face-related processing in congenital prosopagnosia: 2. Functional neuroimaging findings. J Cogn Neurosci, 17: 1150-67, 2005. BARTON JJ. Disorders of face perception and recognition. Neurol Clin, 21: 521-48, 2003. BARTON JJ, CHERKASOVA M, and O'CONNOR M. Covert recognition in acquired and developmental prosopagnosia. Neurology, 57: 1161-8, 2001. BARTON JJS, PRESS DZ, KEENAN JP, and O'CONNOR M. Lesions of the fusiform face area impair perception of facial configuration in prosopagnosia. Neurology, 58: 71-78, 2002. BODAMER J. Die prosopagnosie. Arch Psychiat Nerv, 179: 6?54, 1947. BRIERLEY B, MEDFORD N, SHAW P, and DAVID AS. Emotional memory and perception in temporal lobectomy patients with amygdala damage. J Neurol Neurosurg Psychiatry, 75: 593-9, 2004. BRUCE V and YOUNG A. Understanding face recognition. Br J Psychol, 77 ( Pt 3): 305-27, 1986. CALDER AJ, BURTON AM, MILLER P, YOUNG AW, and AKAMATSU S. A principal component analysis of facial expressions. Vision Res, 41: 1179-208, 2001. CALDER AJ and YOUNG AW. Understanding the recognition of facial identity and facial expression. Nat Rev Neurosci, 6: 641-51, 2005. DAMASIO A. Prosopagnosia. Trends in Neurosci: 132-5, 1985. DE RENZI E. Prosopagnosia in two patients with ct scan evidence of damage confined to the right hemisphere. Neuropsychologia, 24: 385-9, 1986. DE RENZI E, FAGLIONI P, GROSSI D, and NICHELLI P. Apperceptive and associative forms of prosopagnosia. Cortex, 27: 213-21, 1991. DOUVILLE K, WOODARD JL, SEIDENBERG M, MILLER SK, LEVERONI CL, NIELSON KA, FRANCZAK M, ANTUONO P, and RAO SM. Medial temporal lobe activity for recognition of recent and remote famous names: An event-related fmri study. Neuropsychologia, 43: 693-703, 2005. DUCHAINE BC, PARKER H, and NAKAYAMA K. Normal recognition of emotion in a prosopagnosic. Perception, 32: 827-38, 2003. EVANS J, HEGGS A, ANTOUN N, and HODGES J. Progressive prosopagnosia associated with selective right temporal lobe atrophy. Brain, 118: 1-13, 1995. GANEL T, GOSHEN-GOTTSTEIN Y, and GANEL T. Effects of familiarity on the perceptual integrality of the identity and expression of faces: The parallel-route hypothesis revisited. J Exp Psychol Hum Percept Perform, 30: 583-97, 2004. GANEL T, VALYEAR KF, GOSHEN-GOTTSTEIN Y, and GOODALE MA. The involvement of the "Fusiform face area" In processing facial expression. Neuropsychologia, 43: 1645-1654, 2005. GOBBINI MI and HAXBY JV. Neural response to the visual familiarity of faces. Brain Res Bull, 71: 76-82, 2006.   17GOBBINI MI and HAXBY JV. Neural systems for recognition of familiar faces. Neuropsychologia, 45: 32-41, 2007. GRILL-SPECTOR K, HENSON R, and MARTIN A. Repetition and the brain: Neural models of stimulus-specific effects. Trends Cogn Sci, 10: 14-23, 2006. GRILL-SPECTOR K, KNOUF N, and KANWISHER N. The fusiform face area subserves face perception, not generic within-category identification. Nat Neurosci, 7: 555-562, 2004. HAXBY JV, HOFFMAN EA, and GOBBINI MI. The distributed human neural system for face perception. Trends Cogn Sci, 4: 223-233, 2000. ISHAI A, SCHMIDT CF, and BOESIGER P. Face perception is mediated by a distributed cortical network. Brain Res Bull, 67: 87-93, 2005. KANWISHER N, MCDERMOTT J, and CHUN MM. The fusiform face area: A module in human extrastriate cortex specialized for face perception. J Neurosci, 17: 4302-11, 1997. KAUFMANN JM and SCHWEINBERGER SR. Expression influences the recognition of familiar faces. Perception, 33: 399-408, 2004. KOSAKA H, OMORI M, IIDAKA T, MURATA T, SHIMOYAMA T, OKADA T, SADATO N, YONEKURA Y, and WADA Y. Neural substrates participating in acquisition of facial familiarity: An fmri study. Neuroimage, 20: 1734-42, 2003. KURUCZ J and FELDMAR G. Prosopo-affective agnosia as a symptom of cerebral organic disease. J Am Geriatr Soc, 27: 225-30, 1979. KURUCZ J, FELDMAR G, and WERNER W. Prosopo-affective agnosia associated with chronic organic brain syndrome. J Am Geriatr Soc, 27: 91-5, 1979. KURUCZ J, SONI A, FELDMAR G, and SLADE W. Prosopo-affective agnosia and computerized tomography findings in patients with cerebral disorders. Journal of the American Geriatrics Society, 28: 475-478, 1980. LANDIS T, CUMMINGS J, CHRISTEN L, BOGEN J, and IMBOF H-G. Are unilateral right posterior cerebral lesions sufficient to cause prosopagnosia ? Clinical and radiological findings in six additional patients. Cortex, 22: 243-52, 1986. LEOPOLD DA, O'TOOLE AJ, VETTER T, and BLANZ V. Prototype-referenced shape encoding revealed by high-level aftereffects. Nat Neurosci, 4: 89-94, 2001. LEOPOLD DA, RHODES G, MULLER K-M, and JEFFERY L. The dynamics of visual adaptation to faces. Proceedings of the Royal Society B, 272: 897-904, 2005. LISSAUER H. Ein fall von seelenblindheit nebst einim beitrage zur theori derselben. Archiv fur Psychiatrie und Nervenkrankheiten, 21: 222-270., 1890. MATTSON AJ, LEVIN HS, and GRAFMAN J. A case of prosopagnosia following moderate closed head injury with left hemisphere focal lesion. Cortex, 36: 125-37, 2000. MEADOWS JC. The anatomical basis of prosopagnosia. J Neurol Neurosurg Psychiatry, 37: 489-501, 1974. NIEMAN DR, HAYASHI R, ANDERSEN RA, and SHIMOJO S. Gaze direction modulates visual aftereffects in depth and color. Vision Res, 45: 2885-94, 2005. PALERMO R and RHODES G. Are you always on my mind? A review of how face perception and attention interact. Neuropsychologia, 45: 75-92, 2007. PELPHREY KA, SINGERMAN JD, ALLISON T, and MCCARTHY G. Brain activation evoked by perception of gaze shifts: The influence of context. Neuropsychologia, 41: 156-70, 2003. PUCE A, ALLISON T, BENTIN S, GORE JC, and MCCARTHY G. Temporal cortex activation in humans viewing eye and mouth movements. J Neurosci, 18: 2188-99, 1998. ROSSION B, CALDARA R, SEGHIER M, SCHULLER A-M, LAZEYRAS F, and MAYER E. A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing. Brain, 126: 2381-2395, 2003.   18ROTSHTEIN P, HENSON RN, TREVES A, DRIVER J, and DOLAN RJ. Morphing marilyn into maggie dissociates physical and identity face representations in the brain. Nat Neurosci, 8: 107-13, 2005. SCHILTZ C, SORGER B, CALDARA R, AHMED F, MAYER E, GOEBEL R, and ROSSION B. Impaired face discrimination in acquired prosopagnosia is associated with abnormal response to individual faces in the right middle fusiform gyrus. Cereb Cortex, 16: 574-86, 2006. SCHWEINBERGER SR, BURTON AM, and KELLY SW. Asymmetric dependencies in perceiving identity and emotion: Experiments with morphed faces. Percept Psychophys, 61: 1102-1115, 1999. SCHWEINBERGER SR and SOUKUP GR. Asymmetric relationships among perceptions of facial identity, emotion, and facial speech. J Exp Psychol Hum Percept Perform, 24: 1748-65, 1998. SNOWDEN JS, THOMPSON JC, and NEARY D. Knowledge of famous faces and names in semantic dementia. Brain, 127: 860-72, 2004. TAKAHASHI N, KAWAMURA M, HIRAYAMA K, SHIOTA J, and ISONO O. Prosopagnosia: A clinical and anatomical study of four patients. Cortex, 31: 317-29, 1995. THOMPSON SA, GRAHAM KS, WILLIAMS G, PATTERSON K, KAPUR N, and HODGES JR. Dissociating person-specific from general semantic knowledge: Roles of the left and right temporal lobes. Neuropsychologia, 42: 359-70, 2004. TSUKIURA T, FUJII T, FUKATSU R, OTSUKI T, OKUDA J, UMETSU A, SUZUKI K, TABUCHI M, YANAGAWA I, NAGASAKA T, KAWASHIMA R, FUKUDA H, TAKAHASHI S, and YAMADORI A. Neural basis of the retrieval of people's names: Evidence from brain-damaged patients and fmri. J Cogn Neurosci, 14: 922-37, 2002. TSUKIURA T, MOCHIZUKI-KAWAI H, and FUJII T. Dissociable roles of the bilateral anterior temporal lobe in face-name associations: An event-related fmri study. Neuroimage, 30: 617-26, 2006. TZAVARES A, MERIENNE L, and MASURE MC. Prosopagnosie, amnesie, et troubles du langage par lesion temporale gauche chez un sujet gaucher. Encephale, 62: 382-94, 1973. VUILLEUMIER P, ARMONY JL, DRIVER J, and DOLAN RJ. Effects of attention and emotion on face processing in the human brain: An event-related fmri study. Neuron, 30: 829-41, 2001. WEBSTER MA, KAPING D, MIZOKAMI Y, and DUHAMEL P. Adaptation to natural facial categories. Nature, 428: 557-61, 2004. WINSTON JS, HENSON RN, FINE-GOULDEN MR, and DOLAN RJ. Fmri-adaptation reveals dissociable neural representations of identity and expression in face perception. J Neurophysiol, 92: 1830-9, 2004. YOUNG AW, NEWCOMBE F, DE HAAN EH, SMALL M, and HAY DC. Face perception after brain injury. Selective impairments affecting identity and expression. Brain, 116 ( Pt 4): 941-59, 1993. YOVEL G and KANWISHER N. The neural basis of the behavioral face-inversion effect. Curr Biol, 15: 2256-2262, 2005.     192   WHAT IS ADAPTED IN FACE ADAPTATION? THE NEURAL      REPRESENTATIONS OF EXPRESSION IN THE HUMAN VISUAL SYSTEM1      2.1  Introduction Facial expression is an important vehicle for social communication. The perception and interpretation of facial expression provides us with clues about the emotional state of those with whom we interact. Disordered perception of facial expression is a feature of neurological disorders such as autism and Asperger syndrome, and may contribute to the social disruption experienced by patients with these diagnoses (Hefter, Manoach et al., 2005). Understanding the neural representation of facial expression is important to advancing our knowledge of how the human visual system organizes and extracts socially relevant perceptual signals.   Current concepts of facial recognition suggest parallel processing of facial identity and facial expression in both cognitive and anatomic models, based largely on human functional imaging experiments that supplement earlier neurophysiological data from monkeys (Bruce and Young, 1986; Haxby, Hoffman et al., 2000; Eifuku, De Souza et al., 2004; Andrews and Ewbank, 2004). Processing of facial identity may be a specific role of the fusiform face area, located in the inferior occipitotemporal cortex (Barton, 2003; Grill-Spector, Knouf et al., 2004; Haxby, Hoffman et al., 2000), whereas facial expression may be preferentially processed in the superior temporal sulcus, located in the lateral occipitotemporal cortex (Haxby, Hoffman et al., 2000). The superior temporal sulcus appears to be involved in recognizing the changeable aspects of the face (Haxby, Hoffman et al., 2000), such as direction of gaze (Pelphrey, Singerman et al., 2003), mouth movements (Callan, Jones et al., 2004; Puce, Allison et al., 1998),  as well as expression (Winston, Henson et al., 2004). In addition, fMRI shows that                                                  1 A version of this chapter has been published. Fox C.J. and Barton J.J.S (2007). What is adapted in face adaptation? The neural representations of expression in the human visual system. Brain Research, 1127: 80-89.  ? 2006 Elsevier B.V.   20activity in the superior temporal sulcus is selectively increased when attention is directed towards emotion in facial images (Narumoto, Okada et al., 2001).   While these data suggest that expression may have a specific neuroanatomic substrate in the superior temporal sulcus, they are less clear on the nature of the representations contained within that substrate. Recent work using adaptation paradigms and aftereffects have suggested a means of exploring the neural representations of faces. Previous reports have shown that aftereffects (biased perceptions following sensory adaptation to a stimulus) exist not only for photoreceptor-based phenomena such as color (Allan, Siegel et al., 1997; Nieman, Hayashi et al., 2005), but also for cortically based phenomena such as motion (Seiffert, Somers et al., 2003; Snowden and Milne, 1997), tilt in two dimensions (Adams and Mamassian, 2002), slant in three dimensions (Domini, Adams et al., 2001), and, more recently, for faces (Leopold, O'Toole et al., 2001; Webster, Kaping et al., 2004; Yamashita, Hardy et al., 2005).  One of these reports has documented aftereffects specific to facial identity (Leopold, O'Toole et al., 2001). When shown a series of morphed faces that varied between a target face and its ?anti-face? (one with the opposite structural characteristics to the target face), subjects were more likely to perceive the identity of the target face in an ambiguous morphed image after they had been exposed to the ?anti-face?. Another study found similar aftereffects for a variety of facial properties beyond identity, including gender, race, and expression (Webster, Kaping et al., 2004). This second study confirmed that an adaptation paradigm can be a useful tool to probe the neural populations involved in perceiving expression. However, the conclusions that can be made from its results, about the neural representations of expression, are limited because their adapting stimulus was the same image as the one used to generate the morph series. Therefore one cannot determine whether this adaptation is of expression in general, expression in a specific face, or expression in a specific image.   21Our objective was to systematically explore how differences in the adapting stimulus affected the production of aftereffects on expression perception, thereby better defining the neural representations of facial expression. Our initial hypothesis was that there should be a neural representation of expression that generalizes across different facial identities. For facial expression to be a truly useful social cue, it is important to be able to infer similar emotional states from similar expressions on the faces of different people. If so, we predicted that we would find adaptation aftereffects even if the faces of different people were used as the adapting stimuli and the probe stimuli.     2.2   Methods   2.2.1 Subjects   Thirty-eight subjects (23 female) participated in the entire study. All subjects spoke English and did not understand German. In the first experiment twenty-seven subjects (16 female; Age= 30.63yrs, SD= 10.24 yrs) were randomly assigned to one of the three possible expression-pairs, while CJF participated in all three of the expression-pairs in experiment 1, giving 10 subjects for each of the three expression-pairs used in the first experiment. Other than CJF all subjects were na?ve to the purpose of the experiment, and CJF?s results did not differ from the group data. In the second experiment ten different na?ve subjects (7 female; Age= 25.5yrs, SD= 4.88 yrs) participated. All 38 subjects had normal or corrected-to-normal vision, and were able to clearly identify facial expressions and read on-screen text at the testing distance used (57 cm). The protocol was approved by the institutional review boards of Vancouver General Hospital and the University of British Columbia, and all subjects gave informed consent in accordance with the declaration of Helsinki.     222.2.2 Stimuli, Experiment 1 Facial stimuli were obtained from the Karolinska Database of Emotional Faces (KDEF) (Lundqvist, 1998). Research of facial expression suggests six fundamental facial expressions that are reliably recognized across cultures, which are anger, fear, happiness, sadness, surprise, and disgust (Ekman and Friesen, 1971; Ekman, Sorenson et al., 1969). The Karolinska series includes 2 pictures of each individual displaying these six facial expressions. Four individuals were chosen, two female and two male. Two versions of the tests were created, with each version consisting of one of the male and one of the female faces as probe stimuli. For each individual the six facial expressions were paired to create three distinct expression pairs (angry/afraid, happy/sad, disgust/surprise). These pairings were based on a previously reported 3-dimensional model of human emotion (Plutchik?s solid) (Strongman, 1978). This model places anger and fear as opposite emotions, as well as happy and sad. Disgust and surprise are not placed as opposite emotions but are spatially removed from each other within the model. Since opposite emotions are not universally defined or recognized the present examination of aftereffects is different than aftereffects of stimuli with absolute opposites (i.e.-color, tilt, etc.).  For the stimuli used as probes of the aftereffect, we created morphs across the three expression pairs for each Karolinska face, using Fantamorph 3.0 (www.fantamorph.com). Twenty-one images were produced for each morph series, with each picture representing a 5% step within the morph series (i.e.- 0/100, 5/95, 10/90 ?100/0). The thirteen images ranging from 20/80% to 80/20% were used as ambiguous probe stimuli in the final experiment. All facial images were presented in the centre of the screen and spanned a distance of 8.6? horizontally and 11.8? vertically.   Before making judgments on these ambiguous probe stimuli, subjects were first exposed to adapting stimuli (varied by condition) for 5 seconds.  For the same-image condition, the same faces used to generate the morph series were used as adapting stimuli for that particular series.   23For the different-person conditions, the faces used to generate the morph series in one version of the test were used as adapting faces for judgments on the morph series in the other version of the test. Thus in the different-person/same gender condition, the second of the two females was used as the adapting stimulus for the morphed probe stimuli created from the first female, and vice versa. In the different-person/different-gender condition, one of the males was used as the adapting image for one of the female morphed probes, and the other male for the other female probes. This two-version balanced design ensured that an equal number of all the faces were used as adapting stimuli, and the multiple face series were used equally frequently as probes for aftereffects across these three different adapting conditions.  For the verbal condition, word stimuli consisted of the adjective form of the expression (Angry, Afraid, Happy, Sad, Disgusted, and Surprised) in all capital letters, created using Microsoft PowerPoint (www.microsoft.com). These images were re-sized to span the same horizontal distance (8.6?) as the facial images, and placed in the centre of the screen. The letters themselves spanned 1.8? vertically.  2.2.3 Stimuli, Experiment 2 This experiment also used the Karolinska faces (Lundqvist, 1998), but this time only those images representing anger and fear. For the probe stimuli, we used one of the female and one of the male morphed series created for angry/afraid in the first experiment, and also generated another morphed series using the alternate images of the same expression for these two individuals. This was needed to create a balanced design between same-image and same-person/different-image conditions.  The probe stimuli for half of the subjects were the morph series used in the angry/afraid tests in the first experiment. For the other half of the subjects the probes were the morph series created from the alternate images of the same people from the Karolinska series. For adapting   24stimuli the same-image condition again used the same faces that generated the morph series used as probe stimuli in that subject. The same-person/different-image condition used the Karolinska faces that were morphed to generate the alternate morph series, which was not used as a probe in that subject. Thus, across all subjects, each Karolinska face was used with the same frequency as adapting stimuli in the same-image and same-person/different-image conditions, and each morph series was used equally frequently as a probe for the aftereffect.  The visual/non-facial condition used the images of Darwin?s dogs, obtained from the Internet, which were initially drawn by Charles Darwin as a standard representation of a dog displaying anger or fear (Darwin, 1899). These images were presented in the middle of the screen and spanned a distance of 10? horizontally and 6.8? vertically.  The auditory condition used audio files from a battery of emotional sounds that had been rated on a seven point scale for emotional intensity (http://pascal.kgw.tu-berlin.de/emodb). The selected files ranged from ratings of 5.15 to 6.52. These sound files were German, semantically non-emotional sentences which were read with emotional prosody. Three to four of these audio files were linked to create a composite audio file, which lasted 5 seconds and contained both male and female voices.  2.2.4 Ratings All Angry and Afraid adapting stimuli were rated by 6 participants (2 female) who had not been previously exposed to these stimuli. Subjects rated the stimuli on a 10 point scale and were asked how angry or how afraid the stimuli were.  2.2.5 Apparatus All experiments were designed and run using SuperLab 1.71 software (www.cedrus.com). The first experiment was run on a G5 PowerMac with a 20? widescreen display. The second   25experiment was run on a G4 PowerBook with a 17? widescreen display. Audio files were presented with Sennheiser HD500 headphones. These headphones were large full ear-cup headphones which reduced external noises that would interfere with perception of the auditory stimuli.  2.2.6 Procedure Subjects were told that they would be shown a series of faces and that they would have to state which of two possible facial expressions was depicted in each face. They were shown examples of the two (unmorphed) expressions and asked to name the expressions, which they all correctly identified. Next, they saw an example of a morphed image and were told that the faces they would be judging were mixtures of the two expressions, and that they should make their best guess at the displayed facial expression.   Subjects were randomly assigned to one of two versions of the test (each version having both one female probe series and one male probe series), and to one of the expression pairs. Each subject first completed a practice session for the expression pair being tested. The 13 morphed images of the 2 experimental faces (1 male and 1 female) were randomly presented 4 times each, for a total of 104 trials, without any adapting stimulus. Each face was on the screen for 300 ms, and was followed by a screen with a large question mark. This question mark remained until the subject indicated, with a key press, which expression they saw (two-alternative forced-choice decision). Following their decision there was a 500 ms inter-trial interval consisting of a blank screen.  Following the practice condition the experiment began. This consisted of 4 blocks, each with a different adapting condition (i.e. for the first experiment, these conditions were same-image, different-person/same-gender, different-person/different-gender, and verbal). The order of these blocks was randomized for each subject. In each block a trial consisted of a 5 s presentation of   26the adapting stimulus (Figure 2.1), which represented one of the two expressions used to create that particular morph series (i.e. for same-image condition in the angry/afraid sub-test, this would be either the angry face or the afraid face used to create the morphed series). The adapting stimulus was followed by a 50 ms mask (a random arrangement of black and white pixels) to reduce apparent motion effects in the following probe, and then a 300 ms probe image to measure the aftereffect, which was one of the morphed faces (Leopold, O'Toole et al., 2001). A question mark then appeared on the screen and remained until the participant indicated which of the two expressions they saw in the probe image, using a key press. After their decision a 500 ms blank screen served as the inter-trial interval.  Subjects were asked to attend to the adapting stimulus, but not to make a judgment about this stimulus. Following each subtest there was a short rest break, before the next subtest began. Each trial was seen only once. With thirteen degrees of morphing, two different probe series (one male and one female), and two different adapting stimuli (i.e. angry adapting stimulus and afraid adapting stimulus) this created 52 trials for each of the four adapting conditions, and hence 208 trials for each subject.                   27Figure 2.1  Example of the expression aftereffect trial  Example of an adapting trial, with samples of stimuli used in all the different adapting conditions in the first and second experiments. These stimuli were taken from the afraid condition. After the adapting stimulus is shown for 5 seconds, there is a short mask serving as an inter-stimulus interval followed by the probe stimulus. This is an ambiguous image from the morphed series, in this case between angry and afraid. Subjects then make a two-alternative forced-choice decision on the expression displayed in the probe stimulus. (Note: the faces pictured here are taken from the author?s collection, while actual experimental stimuli were taken from the Karolinska Database of Emotional Faces)      282.2.7 Analysis For each adapting condition, we calculated the proportion of responses that were given for one of the choices (e.g. how many times they responded ?afraid?). One score was for adaptation after one of the emotions (e.g. the angry adapting stimulus) and another score for adaptation after viewing the opposite emotion (e.g. the afraid adapting stimulus). The subtraction between these two values gave the difference score, which represents the magnitude of the adaptation effect for that particular adaptation condition. All statistical analyses were run on SPSS 13.0 software (www.spss.com), and significance levels for all tests were set at p < .05. In the first experiment, we used a univariate ANOVA with the difference score as the dependent measure. Emotional axis (3 levels), Adapting Condition (4 levels), Test version (2 levels) and Gender of the Probe Face (2 levels) served as predictors within the General Linear Model. Significant main effects were followed up with post-hoc Bonferroni comparisons with correction for multiple comparisons. A priori two-tailed student?s t-tests were run on each adaptation condition to determine whether that condition resulted in a significant aftereffect.   In the second experiment a univariate ANOVA was also run with the difference score as the dependent measure, and Adapting Condition (4 levels), Test Version (2 levels) and Gender of the Probe Face (2 levels) as the predictors. Post-hoc Tukey HSD t-tests examined significant main effects. A priori t-tests were performed as described above.   Independent samples t-tests were used to compare the same-image conditions from Experiments 1 and 2, the different-person condition of Experiment 1 with the visual/non-facial condition of Experiment 2, as well as for the ratings of emotional intensity in all adapting stimuli.       29  2.3   Results  2.3.1 Experiment 1: An identity-independent representation of expression In the first part of this study, we contrasted the effects of four different adapting conditions on the production of an expression-based aftereffect (Figure 2.1). This was done for three different series of morphed images, one from angry to afraid, one from sad to happy, and one from disgusted to surprised. The first adapting condition consisted of images that were identical to those used to derive the morphed images which served as probes of the aftereffect. This ?same-image? condition also served to replicate the findings of the prior study cited (Webster, Kaping et al., 2004). The second and third conditions used adapting stimuli which were faces of different individuals showing one of the expressions used in the probe series. The second condition had the same gender as the probe (?different-person/same-gender?), while the third condition had a different gender (?different-person/different-gender?). The last condition simply presented a word on the screen, naming one of the expressions. This ?verbal? condition was performed to determine if simply evoking the expression through a verbal semantic association was enough to generate face expression aftereffects. If there is indeed a generalized representation of expression in the human visual system, then we would expect to see aftereffects in the different-person conditions. Strong aftereffects in the verbal condition would suggest an even more general non-visual representation of facial expression.  Our initial analysis tested for main effects of adapting condition, expression-pair (i.e. angry/afraid, sad/happy, disgusted/surprised), gender of the probe face, and test version (see Methods). There was no significant main effect of expression-pair, indicating that similar aftereffects were obtained for all three emotional axes (Figure 2.2A). There were also no significant main effects of the gender of the probe face or the version of the test, and no significant two-way, three-way, or four-way interactions involving these variables. Therefore the    30Figure 2.2  Same identity, different identity and verbal expression aftereffects                                  A) Aftereffects for each of the three expression-pairs used in the first experiment. For each expression-pair there were two adapting runs, one for each of the two opposing expressions. Aftereffects are reported as the difference score between the trials using one adapting stimulus (e.g. angry) and trials using the opposite adapting stimulus (e.g. afraid). B) Pooled aftereffects from all three expression-pairs, again reported as the difference score. Error bars indicate one standard error. Significant aftereffects are denoted with an asterisk (*) and non-significant aftereffects noted (n.s.). SI = same-image, DP/SG = different-person/same-gender, DP/DG = different-person/different-gender, V = verbal. Inset graphs show, for the different adapting conditions, the probability of choosing Expression 2 as a function of the percentage of Expression 2 in the probe stimulus, fitted with sigmoid functions. The solid lines represent the data for Expression 1 (i.e.-angry, happy, disgusted) and the dashed lines represent the data for Expression 2 (i.e.-afraid, sad, surprised) in the expression-pairs.    31data were collapsed across version, gender of probe face, and expression-pair in the following analyses.    There was a significant main effect of adapting condition on aftereffects [F(3,47)=1.656, p<.0001] (Figure 2.2B). As reported in another study (Webster, Kaping et al., 2004), the likelihood of reporting an emotion in an ambiguous probe face was reduced if subjects were adapted to the same image, expressing that emotion, which was used to create the morph series. Our same-image condition generated a similar aftereffect, where judgments of expression in the probe stimuli were biased towards the expression opposite to that displayed in the adapting stimuli [t(59)=12.540, p<.0001]. The magnitude of this aftereffect was sizeable, being a 22.62% (SEM = 1.80%) difference in the probability of expression choice between the two adapting conditions (i.e.- Angry and Afraid). However, we also found that aftereffects were generated using images from persons different from those seen in the probe series of morphed images, as adapting stimuli. Significant shifts in emotional judgments were produced in both the different-person/same-gender condition [6.40%, SEM=1.92%; t(59)=3.339, p=.001] and different-person/different-gender condition [7.06%, SEM=1.44%; t(59)=4.887, p<.0001]. In contrast to the clear aftereffects induced by different-person conditions, the verbal condition did not generate a significant aftereffect on the perception of facial expression [0.05%, SEM=1.44%; t(59)=.038, p=.970].  Post-hoc Bonferroni comparisons between the different adapting conditions showed that the difference between aftereffects of the different-person/same-gender and different-person/different-gender conditions was not significant (? =0.65%; p=1.00), indicating that expression adaptation generalizes across gender. Further comparisons showed that the same-image aftereffect was significantly larger than the aftereffects of all other conditions, including the different-person/same-gender condition (? =16.22%; p<.0001), the different-person/different-gender condition (? =15.56%; p<.0001), and the verbal condition (? =22.57%, p<.0001). The   32insignificant aftereffect of the verbal condition was significantly smaller than the same-image condition, and the different-person/different-gender condition (? =7.00%, p=.027), and showed a trend to being significantly smaller than the different-person/same-gender condition (? =6.35%, p=.059). This first experiment confirmed the existence of a neural representation of facial expression that generalizes across facial identity, and that adaptation can occur across this generalizable representation of expression. As such, this represents what might be termed an abstract ?visual semantic? for expression, where expression is not coded as a particular appearance of a particular face, but as some shared configuration among all faces when a person is experiencing a certain emotional state. However, whether this is a visual semantic that is face-specific or that is generalizable to other visual depictions of emotion is an open question. Although a verbal semantic association was insufficient to generate an aftereffect, this may be attributable to intensity differences in the adapting stimuli. Emotional words (2.92, SEM=0.54) were rated as less intense than emotional faces (5.17, SEM=.27) and this rating difference was significant [t(82)=3.216, p=.002]. Thus the lack of aftereffect to verbal stimuli may be due to an adapting stimulus which was too weak to elicit such an effect. It is therefore still possible to ask whether this semantic representation is purely visual or if it is multi-modal in nature.  Finally, another question is suggested by the observation that the same-image condition generated a much stronger aftereffect than the different-person conditions. The implication is that there must be some neural representation that is adapted in the same-image condition but not in the two different-person conditions. The results of this first experiment do not clarify whether this neural representation is specific to the image or specific to the individual.      332.3.2 Experiment 2: An identity-dependent representation of expression Our second experiment was designed to test these three issues: i) is the ?visual semantic? an abstract facial representation or a more general visual representation; ii) is there a multi-modal expression aftereffect; iii) is the larger same-image aftereffect a result of adaptation to identity or image. In Experiment 1 we showed that there was no difference between the 3 expression-pairs, therefore we restricted Experiment 2 to one expression-pair (angry-afraid). Again four adapting conditions were used. The first was a repeat of the same-image condition. The second was a same-person/different-image condition, in which the identity between adapting and probe stimulus was held constant, but a different image (different to the image used to create the probe series) of that individual displaying one of the morphed expressions was used. The contrast between this and the same-image condition would reveal whether the strong aftereffect generated in the same-image condition is an adaptation to a specific image showing expression or to a specific person showing expression.  The third adapting condition, the visual/non-facial condition, used pictures of anger and fear that did not involve human faces, namely Darwin?s dogs (Darwin, 1899). While it is true that facial information is not completely removed from these images of Darwin?s dogs, it was believed that these images could still provide evidence for a more general visual semantic representation of emotion beyond that for human faces. Finally, the fourth, auditory condition examined whether expressions of emotion heard in neutral German sentences read with emotional prosody could generate cross-modal aftereffects on judgments of facial expression. As in the first experiment, the initial analysis demonstrated that the aftereffect was significantly affected by adapting condition [F(3,15)=1.208, p<.0001] but not by gender of the target face [F(1,15)=.006, p=.595] or test version [F(1,15)=.021, p=.316], and that there were no significant two-way or three-way interactions between these variables. Therefore the data were collapsed across test version and gender of probe face in the following analyses (Figure 2.3).   34Figure 2.3  Different image, visual/non-facial, and auditory expression aftereffects  Aftereffects of adapting conditions used in the second experiment, reported as the difference score between angry and afraid adapting stimuli. Error bars show one standard error. Significant aftereffects are denoted with an asterisk (*), while trends to significance are indicated with a pound sign (#). Non-significant aftereffects are noted (n.s.). SI = same-image, SP/DI = same-person/different-image, V/NF = visual/non-facial, A = auditory. Inset graphs show for the different adapting conditions the probability of choosing ?afraid? as a function of the percentage of the afraid face in the probe stimulus, fitted with sigmoid functions. In each inset graph the solid lines represents the data for the ?angry? condition, the dashed lines represent the data for the ?afraid? condition.         35The same-image condition produced a significant aftereffect [26.54%, SEM=3.80; t(19)=6.976, p<.0001], similar in magnitude to that of the same-image condition  in Experiment 1 [t(38)=.077, p=.939] . Furthermore, adaptation to the same-person/different-image condition resulted in an aftereffect equally as large as the same-image condition [30.38%, SEM=2.98; t(19)=10.203, p<.0001]. The visual/non-facial condition generated a trend to a weak aftereffect [4.23%, SEM= 2.12; t(19)=1.993, p=.061]. There was no aftereffect in the auditory condition [3.85%, SEM=3.23; t(19)=1.191, p=.248]. This lack of an auditory aftereffect could not be attributed to emotionally less intense stimuli in the auditory condition, as the auditory stimuli (7.00, SEM=0.44) were rated as significantly more intense than the face stimuli (5.17, SEM=0.27) [t(82)=-2.662, p=.009].  Post-hoc Bonferroni comparisons between the different adapting conditions showed that the strong aftereffects of the same-image and same-person/different-image conditions were not significantly different (? =3.85%, p=1.00), and the weak or non-existent aftereffects of the visual/non-facial and auditory conditions also did not differ (? =0.38%, p=1.00), due in large part to the large standard deviation of the auditory condition (14.44%).  The visual/non-facial aftereffect was significantly smaller than the aftereffects observed in the same-image condition (? =22.31%; p<.0001) and the same-person/different-image  condition (? =26.15%; p<.0001), and the aftereffect of the auditory condition was also significantly smaller than those in the same-image condition (? =22.69%, p<.0001) and the same-person/different-image  condition (? =26.54%, p<.0001).    Finally, as a further examination of the trend in the visual/non-facial aftereffect, we compared this aftereffect with the different- person aftereffect (collapsed across gender) from the first experiment (angry/afraid expression pair only), using an independent samples t-test. The visual/non-facial aftereffect, while smaller (? =4.81%) was not significantly smaller than the different-person aftereffect [t(58)=1.362, p=.178].  Of note, the emotional faces and the dogs   36received emotional ratings that were not significantly different (5.17, SEM=.27 for faces, 4.58, SEM=.50 for dogs, t(82)=.840, p=.403).       2.4   Discussion The results of these two experiments suggest that at least two neural representations of facial expression exist in the human visual system (Figure 2.4). First, the fact that aftereffects can be generated from the faces of different people confirms our hypothesis that a neural representation of expression that is independent and generalizable across facial identity exists. Deduction about the second neural representation relates to the observation that much larger aftereffects were generated by images of the same person than images of a different person. Our balanced design (see Methods) ensured that the same facial images were used in the same-image conditions and the different-person conditions, both for adaptation and for probes. The pairing of the adapting stimuli and probe images, not the images themselves, generated the same-image or different-person conditions. Thus differences in the aftereffect cannot be attributed to differences in the intensity of emotion in different images, but is a result of the pairing in that particular condition. The larger aftereffects with the same-image condition therefore suggest that there is a second neural representation of facial expression that is not independent of facial identity.   One possibility we considered was that this second identity-dependent component of the heightened aftereffect of the same-image condition was simply the result of adaptation to low-level image properties. We addressed this issue in the second experiment by using the same-person/different-image condition. Our results demonstrated an aftereffect generated by the same-person/different-image stimuli which was equal in magnitude to the aftereffect generated by the same-image condition (Figure 2.3). Thus the identity-dependent component of adaptation seen in the same-image condition may relate to a visual representation of expression specific to the individual portrayed, which generalizes over variations in that expression by that individual. Due    37Figure 2.4  Hierarchical representations of facial expression   A schematic diagram illustrating the two proposed representations of facial expression. Low-level image properties (bottom box) are processed and emerge as an identity-dependent representation of expression (middle pictures). Downstream of this representation of expression, which is linked to identity, is a general ?visual semantic? representation of expression that is not specific to identity, but represents an abstract representation of that facial expression (top black and white figures; left = happy, right = angry). (Note: the faces pictured here are taken from the author?s collection, while actual experimental stimuli were taken from the Karolinska Database of Emotional Faces)      38to the balanced design across subjects of this experiment with regards to the stimuli used for adaptation and the stimuli used for probing the aftereffect, the results again cannot be attributed to variations in the intensity of expression within the different images.    While it is true that in the same-person/different-image condition the physical difference between adapting stimuli and probe image is smaller than in the different-person conditions we posit that the aftereffect seen here is not the result of an aftereffect based on low-level image properties, but is the result of a neural representation which codes for both identity and expression (identity-dependent expression). An aftereffect to low-level features could operate on two levels, retinal and cortical. Retinal adaptation to local luminance values can be ruled out as subjects were not instructed to maintain steady fixation on one point. Cortical low-level adaptation cannot be entirely excluded: for example, given the residual similarity between the two images of the same face used, adaptation to the orientation of local features may in some degree contribute to the magnitude of the identity-dependent adaptation effect. However, there are other lines of evidence that support the hypothesis of a higher-level identity-dependent representation of expression. First, the existence of an identity-dependent representation of expression is also suggested by studies showing interference effects from variations in identity on the speed of classification of expression (Baudouin, Martin et al., 2002; Ganel, Goshen-Gottstein et al., 2004; Schweinberger and Soukup, 1998). Second, a recent study reported that expression can enhance identity recognition in familiar faces, suggesting there may be prototypical expressions linked with particular identities (Kaufmann and Schweinberger, 2004). Also, a principle component analysis study found a large degree of overlap between components which are important for expression discrimination and those which are important for identity discrimination (Calder, Burton et al., 2001). This study also showed that identity discriminations could still be made when using only the components which were selected for expression discriminations and vice versa (Calder, Burton et al., 2001). Thus there is a degree of overlap in   39the information used for these two types of facial discriminations, which would be expected if there were neural representations that processed expression in an identity-dependent manner, or identity in an expression-dependent manner. The potential existence of a visual system designed to include both identity-dependent and identity-independent representations of faces has precedence in monkey studies of viewpoint variation (Perrett, Oram et al., 1991). These show two distinct neural populations that respond to faces in specific angles of view: i) neurons that respond to a particular angle of view but only for a specific identity; and ii) neurons that respond to a particular angle of view irrespective of identity (Perrett, Oram et al., 1991). Furthermore, populations that generalize responses for one stimulus property across other stimulus properties are often portrayed in neural models as a second hierarchical layer that receives converging input from a first layer whose responses also vary with those other properties. Thus, for example, a layer that encodes identity across variations in view point may receive converging input from a layer that encodes identity seen from specific viewpoints (Rosen, 2003). In Figure 2.4, we speculate that our two neural populations might be arranged in a similar fashion, with a population that encodes expression across variations in identity receiving converging input from a population that encodes expression seen in specific individual faces.   Our additional conditions explored the properties of the visual semantic representation that is responsible for the portion of the aftereffect that generalizes across identity, as revealed by the different-person condition. There was a trend to an aftereffect from visual/non-facial stimuli. Whether this was related to expressions on the dog?s faces can be debated. Given the small residual aftereffects with these stimuli, it is not possible to exclude the possibility that the visual semantic representations involved in the different-person condition consists of both a face-semantic and a more modest, more general visual semantic for emotion beyond that for human faces.    40  We found no convincing evidence of cross-modal adaptation from auditory perception or from verbal semantic information. Viewing an emotional word or listening to a prosodic sentence did not affect expression judgment on an ambiguous face. This would suggest that all of the face adaptation effects we found for expression are related to visual processing. However, some caution is required with this conclusion, as the intensity of emotion perceived in the verbal stimuli was weaker than that  perceived in the face stimuli. This emotional intensity explanation for the lack of an aftereffect can not be extended to the auditory condition, as these stimuli were seen as more emotionally intense than the face stimuli. Therefore, despite reports that the superior temporal sulcus is considered a multi-modal region of cortex, combining both visual and auditory information (Sekiyama, Kanno et al., 2003),  our results fail to show integration of emotional information across modalities in the neural representations affected by adaptation.  Similar failures to find cross-modal effects on face perception have been reported in a study that showed that visual distracters reduced a facial identity aftereffect while auditory distracters had no such influence (Moradi, Koch et al., 2005).  Cross-modal aftereffects likely occur at a point of emotional experience and integration rather than perceptual processing. Emotional integration of faces and voices has been demonstrated with fMRI, reportedly occurring in the right amygdala for negative emotions (fear), and the left superior temporal gyrus for positive emotions (happiness) (Pourtois, de Gelder et al., 2005). Other studies have linked emotional experience to activity in frontal and subcortical structures (Phan, Wager et al., 2002; Wicker, Keysers et al., 2003; Yip, Leung et al., 2004), sites downstream of the superior temporal sulcus, which is generally accepted to be the site of facial expression recognition (Allison, Puce et al., 2000; Haxby, Hoffman et al., 2000). Failure of the auditory condition to generate aftereffects in our paradigm may suggest that the neural representations that can be adapted by this method are restricted to visually responsive populations in extra-striate cortex.   41  What are the possible neuroanatomic correlates of the different neural representations of expression which our probe stimuli revealed in the present study? A recent event-related fMRI study examined adaptation to both facial identity and expression (Winston, Henson et al., 2004). Adaptation to identity but not expression was found in the fusiform face area, and adaptation to expression but not identity was found in the middle portion of the superior temporal sulcus. In the posterior portion of the superior temporal sulcus there were large adaptation effects to identity and a smaller adaptation effect to expression. The authors concluded that the posterior superior temporal sulcus and the fusiform face area encoded identity, while expression was encoded more anteriorly than previously believed, in the middle superior temporal sulcus (Winston, Henson et al., 2004). Our results suggest a possible alternate explanation. Our finding of a neural representation of expression that is specific to facial identity, would predict a neural population that shows adaptation effects for both identity and expression, much as they reported for the posterior superior temporal sulcus. On the other hand, the finding of a visual semantic representation which generalizes across facial identities would predict a neural population that shows adaptation effects for expression but not identity, as they reported for the middle superior temporal sulcus. Thus the functional segregation reported in this fMRI study (Winston, Henson et al., 2004) may offer a tantalizing parallel with the behavioral data we report. Further investigations are desirable to address our proposal, that there are distinct identity-specific and general visual semantic neural representations of face expression, and our speculation that these may be reflected in segregated neural populations within the superior temporal sulcus.    42  2.5  References ADAMS WJ and MAMASSIAN P. Common mechanisms for 2d tilt and 3d slant after-effects. Vision Res, 42: 2563-8, 2002. ALLAN LG, SIEGEL S, KULATUNGA-MORUZI C, EISSENBERG T, and CHAPMAN CA. Isoluminance and contingent color aftereffects. Percept Psychophys, 59: 1327-34, 1997. ALLISON T, PUCE A, and MCCARTHY G. Social perception from visual cues: Role of the sts region. Trends Cogn Sci, 4: 267-278, 2000. ANDREWS TJ and EWBANK MP. Distinct representations for facial identity and changeable aspects of faces in the human temporal lobe. Neuroimage, 23: 905-13, 2004. BARTON JJ. Disorders of face perception and recognition. Neurol Clin, 21: 521-48, 2003. BAUDOUIN JY, MARTIN F, TIBERGHIEN G, VERLUT I, and FRANCK N. Selective attention to facial emotion and identity in schizophrenia. Neuropsychologia, 40: 503-11, 2002. BRUCE V and YOUNG A. Understanding face recognition. Br J Psychol, 77 ( Pt 3): 305-27, 1986. CALDER AJ, BURTON AM, MILLER P, YOUNG AW, and AKAMATSU S. A principal component analysis of facial expressions. Vision Res, 41: 1179-208, 2001. CALLAN DE, JONES JA, CALLAN AM, and AKAHANE-YAMADA R. Phonetic perceptual identification by native- and second-language speakers differentially activates brain regions involved with acoustic phonetic processing and those involved with articulatory-auditory/orosensory internal models. Neuroimage, 22: 1182-94, 2004. DARWIN C. The expression of the emotions in man and animals. New York: D. Appleton and Company, 1899. DOMINI F, ADAMS W, and BANKS MS. 3d after-effects are due to shape and not disparity adaptation. Vision Res, 41: 2733-9, 2001. EIFUKU S, DE SOUZA WC, TAMURA R, NISHIJO H, and ONO T. Neuronal correlates of face identification in the monkey anterior temporal cortical areas. J Neurophysiol, 91: 358-71, 2004. EKMAN P and FRIESEN WV. Constants across cultures in the face and emotion. J Pers Soc Psychol, 17: 124-9, 1971. EKMAN P, SORENSON ER, and FRIESEN WV. Pan-cultural elements in facial displays of emotion. Science, 164: 86-8, 1969. GANEL T, GOSHEN-GOTTSTEIN Y, and GANEL T. Effects of familiarity on the perceptual integrality of the identity and expression of faces: The parallel-route hypothesis revisited. J Exp Psychol Hum Percept Perform, 30: 583-97, 2004. GRILL-SPECTOR K, KNOUF N, and KANWISHER N. The fusiform face area subserves face perception, not generic within-category identification. Nat Neurosci, 7: 555-62, 2004. HAXBY JV, HOFFMAN EA, and GOBBINI MI. The distributed human neural system for face perception. Trends Cogn Sci, 4: 223-233, 2000. HEFTER RL, MANOACH DS, and BARTON JJ. Perception of facial expression and facial identity in subjects with social developmental disorders. Neurology, 65: 1620-5, 2005. KAUFMANN JM and SCHWEINBERGER SR. Expression influences the recognition of familiar faces. Perception, 33: 399-408, 2004. LEOPOLD DA, O'TOOLE AJ, VETTER T, and BLANZ V. Prototype-referenced shape encoding revealed by high-level aftereffects. Nat Neurosci, 4: 89-94, 2001. LUNDQVIST D, AND LITTON, J.E. The averaged karolinska directed emotional faces-akdef. Secondary Titl., 1998. MORADI F, KOCH C, and SHIMOJO S. Face adaptation depends on seeing the face. Neuron, 45: 169-75, 2005.   43NARUMOTO J, OKADA T, SADATO N, FUKUI K, and YONEKURA Y. Attention to emotion modulates fmri activity in human right superior temporal sulcus. Brain Res Cogn Brain Res, 12: 225-31, 2001. NIEMAN DR, HAYASHI R, ANDERSEN RA, and SHIMOJO S. Gaze direction modulates visual aftereffects in depth and color. Vision Res, 45: 2885-94, 2005. PELPHREY KA, SINGERMAN JD, ALLISON T, and MCCARTHY G. Brain activation evoked by perception of gaze shifts: The influence of context. Neuropsychologia, 41: 156-70, 2003. PERRETT DI, ORAM MW, HARRIES MH, BEVAN R, HIETANEN JK, BENSON PJ, and THOMAS S. Viewer-centred and object-centred coding of heads in the macaque temporal cortex. Exp Brain Res, 86: 159-73, 1991. PHAN KL, WAGER T, TAYLOR SF, and LIBERZON I. Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in pet and fmri. Neuroimage, 16: 331-48, 2002. POURTOIS G, DE GELDER B, BOL A, and CROMMELINCK M. Perception of facial expressions and voices and of their combination in the human brain. Cortex, 41: 49-59, 2005. PUCE A, ALLISON T, BENTIN S, GORE JC, and MCCARTHY G. Temporal cortex activation in humans viewing eye and mouth movements. J Neurosci, 18: 2188-99, 1998. ROSEN E. Face representation in cortex: Studies using a simple and not so special model. Secondary Titl. Boston: Massachusetts Institute of Technology. Masters of Engineering in Electrical Engineering and Computer Science: 75, 2003. SCHWEINBERGER SR and SOUKUP GR. Asymmetric relationships among perceptions of facial identity, emotion, and facial speech. J Exp Psychol Hum Percept Perform, 24: 1748-65, 1998. SEIFFERT AE, SOMERS DC, DALE AM, and TOOTELL RB. Functional mri studies of human visual motion perception: Texture, luminance, attention and after-effects. Cereb Cortex, 13: 340-9, 2003. SEKIYAMA K, KANNO I, MIURA S, and SUGITA Y. Auditory-visual speech perception examined by fmri and pet. Neurosci Res, 47: 277-87, 2003. SNOWDEN RJ and MILNE AB. Phantom motion after effects--evidence of detectors for the analysis of optic flow. Curr Biol, 7: 717-22, 1997. STRONGMAN K. The psychology of emotion. New York: Wiley and Sons, 1978. WEBSTER MA, KAPING D, MIZOKAMI Y, and DUHAMEL P. Adaptation to natural facial categories. Nature, 428: 557-61, 2004. WICKER B, KEYSERS C, PLAILLY J, ROYET JP, GALLESE V, and RIZZOLATTI G. Both of us disgusted in my insula: The common neural basis of seeing and feeling disgust. Neuron, 40: 655-64, 2003. WINSTON JS, HENSON RN, FINE-GOULDEN MR, and DOLAN RJ. Fmri-adaptation reveals dissociable neural representations of identity and expression in face perception. J Neurophysiol, 92: 1830-9, 2004. YAMASHITA JA, HARDY JL, DE VALOIS KK, and WEBSTER MA. Stimulus selectivity of figural aftereffects for faces. J Exp Psychol Hum Percept Perform, 31: 420-37, 2005. YIP JT, LEUNG KK, LI LS, and LEE TM. The role of sub-cortical brain structures in emotion recognition. Brain Inj, 18: 1209-17, 2004.    443   IT DOESN?T MATTER HOW YOU FEEL. THE FACIAL IDENTITY      AFTEREFFECT IS INVARIANT TO CHANGES IN FACIAL EXPRESSION2    3.1  Introduction Faces are complex stimuli. Not only do they have complicated three-dimensional structures, but they convey a multitude of perceptual data, including information about identity, gender, race, expression, and direction of gaze, among others. Current behavioral and neuroanatomical models have proposed that the processing of these different types of information may occur in at least two streams (Bruce and Young, 1986; Haxby, Hoffman et al., 2000). One stream is dedicated to the extraction of structural cues that support the perception of identity, gender, and race. Such properties are stable over time and therefore it is hypothesized that these dimensions involve neural representations that are invariant to the dynamic elements of faces (Haxby, Hoffman et al., 2000). These dynamic elements may be processed by the other stream, as temporally varying information conveys key data for the perception of expression, gaze direction, and visual speech (Haxby, Hoffman et al., 2000). The proposal that different anatomic structures process different types of information might lead to a prediction that the perception of facial identity and the perception of facial expression are independent. However, there is growing behavioral and anatomic evidence that this is not the case, and that there may be interactions between the two (Calder and Young, 2005; de Gelder, Frissen et al., 2003; Fox and Barton, 2007; Ganel, Valyear et al., 2005; Humphreys, Avidan et al., 2007; Kaufmann and Schweinberger, 2004; Palermo and Rhodes, 2007; Stephan, Breen et al., 2006; Winston, Henson et al., 2004)   Face adaptation is a recently developed method that can be used to probe the neural representations responsible for the perception of these various facial dimensions (Fox and Barton, 2007; Leopold, O'Toole et al., 2001; Webster, Kaping et al., 2004). Prolonged viewing                                                  2 A version of this chapter has been published. Fox C.J., Oruc I., Barton J.J.S (2008). It doesn?t matter how you feel. The facial identity aftereffect is invariant to changes in facial expression. Journal of Vision, 8(3): 11, 1-13. ? 2007 ARVO   45of a particular face causes a perceptual aftereffect in which an average face is now seen as having structural properties opposite to the adapted face (Leopold, O'Toole et al., 2001). Aftereffects have been reported for the facial dimensions of identity, gender, race and expression among others (Fox and Barton, 2007; Leopold, O'Toole et al., 2001; Webster, Kaping et al., 2004). In all cases the perceptual aftereffect biases perception of an ambiguous test face away from the adapting face along the dimension being examined.   In earlier studies, we used adaptation to explore the nature of neural representations of facial expression in the human visual system (Butler, Oruc et al., 2008; Fox and Barton, 2007). We have shown that adaptation in our paradigm is not generated at the level of local image elements such as orientation, shape or curvature, but likely at a higher level of face representation (Butler, Oruc et al., 2008). Furthermore, we have shown that the magnitude of the expression aftereffect is modulated by the identity of the adapting face (Fox and Barton, 2007). When the adapting and test faces are images of the same person, a large expression aftereffect is generated (Fox and Barton, 2007; Webster, Kaping et al., 2004). An expression aftereffect is still produced even with incongruent identities (when the adapting and test faces are of different people), suggesting that at least some of the expression aftereffect can be attributed to an identity-invariant representation of expression (Fox and Barton, 2007). Of note, though, the magnitude of the expression aftereffect when using incongruent identities is less than that produced when adapting and test images are of the same person (Fox and Barton, 2007). This larger adaptation with congruent identities may suggest the existence of another neural representation of facial expression which is specific to the identity of the adapting face (Fox and Barton, 2007). Indeed, the concept of both dependent and independent (or ?invariant?) layers of representation, with the former providing converging input to the latter, is not an uncommon feature of neural network models that simulate human object recognition (Rosen, 2003).   46  This finding of both identity-dependent and identity-invariant components in expression adaptation raises the question of whether a corresponding situation exists for the representation of identity. Thus, the first goal of our study was to determine if there are both expression-dependent and expression-invariant components to identity adaptation. This issue is further complicated, however, by the fact that, unlike the situation with facial expressions, where the majority of subjects have extensive experience with most facial expressions, neural representations of identity may differ in their strength, with novel faces having relatively weak representations and highly familiar faces having strong representations. Hence a second goal of our study was to determine if the degree of facial familiarity modulated the effects of expression on the identity aftereffect.     3.2  Experiment 1 In Experiment 1 we asked whether there is evidence for expression-dependence within neural representations of identity for novel faces. Morph series were created between two anonymous identities with the same expression. Images selected from the middle range of these morph series display a recognizable expression, but are ambiguous in their identity. Subjects adapt to one of the two identities that were used to create the morph series and are then asked to judge which identity an ambiguous morphed test face most resembled. Adaptation to the first identity will increase the probability that these ambiguous test faces will be identified as similar to the second identity, while adaptation to the second identity will decrease the probability that ambiguous test faces will be identified as similar to it. The difference in these two probabilities is the measure of the identity aftereffect. By manipulating the adapting faces, but using the same test faces across experimental conditions, we can determine which aspects of the adapting faces affect the generation of the identity aftereffect.   47  Experiment 1 consisted of three experimental conditions. 1) The congruent-expression condition used, as adapting stimuli, the same images used to construct the morphed test faces. Thus the facial expressions in the adapting faces and the test faces are the same. 2) The congruent-expression/different-image condition used as adapting stimuli different images of the same faces used to create the morphed test stimuli. However, these different images were still of the same individuals with the same expression. If the congruent-expression/different-image condition produces an aftereffect equal to that in the congruent-expression condition, this would ensure that the aftereffect is not due to properties specific to a particular image, but due to a specific face. 3) The incongruent-expression condition used, as adapting stimuli, faces of the same people but with a different expression than that in the images used to create the morphed test faces. A significant aftereffect in this condition would be consistent with an expression-invariant representation of identity. A significant reduction in the aftereffect compared to the congruent-expression condition would be consistent with the existence of a separate expression-dependent representation of identity as well.  3.2.1 Method  3.2.1.1 Subjects Ten subjects (7 female; Age = 29.1 + 5.5 years) participated in Experiment 1. All subjects, excluding one (CJF), were na?ve to the purpose of the experiment. Subjects had normal or corrected-to-normal vision, and could clearly see all faces and read on-screen text at the testing distance of 57cm. The protocol was approved by the institutional review boards of Vancouver General Hospital and the University of British Columbia. All subjects gave informed consent and the experiment was conducted in accordance with the principles of the Declaration of Helsinki.    483.2.1.2 Stimuli Two female photographic subjects (F01 and F22) were selected from the Karolinska Database of Emotional Faces (Lundqvist and Litton, 1998). A and B versions of these individuals displaying anger and fear were used. Background, hair, ears and neck were blacked out using Adobe Photoshop CS2 9.0.2 (www.adobe.com). Facial features and external jaw contour were preserved using this method. Distinguishing marks, such as moles, were removed using the Spot Healing Brush Tool. Images were then cropped to ensure that all faces were centrally located within the image frame. Cropped images were resized and displayed at a standard width of 400 pixels (10.8?). Luminance and contrast were visually adjusted to be comparable across all images.    Images of F01 and F22 with similar expressions (e.g., F01/Angry-A with F22/Angry-A) were paired to create morph series with Fantamorph 3.0 (www.fantamorph.com). A morph series was created for each of the two versions (A and B images) of each facial expression (angry and afraid). Each of the four morph series contained 41 images, with each image representing an equal 2.5% step along the morph series (i.e.- 100/0%, 97.5/2.5%,?,0/100%). The thirteen middle images (65/35% to 35/65%) were used in the experiment as the test faces with ambiguous identity, while the unmorphed original images were used as the adapting faces.   Half of the subjects were assigned the two A-series for morphed test images, and half were assigned the two B-series. The 13 test faces from each of the two assigned morph series (one for angry and one for afraid images) were used as test images in all experimental conditions for that subject. Adapting stimuli were manipulated between experimental conditions:   1) The congruent-expression condition used, as adapting stimuli, the same (unmorphed) images that were used to generate the morphed test images. Thus the A-series group of subjects (1) adapted to the Angry-A images of F01 or F22, before seeing the Angry-A test images that morphed identity between F01 and F22, and (2) adapted to the Afraid-A images of F01 or F22, before seeing the Afraid-A test images of identity morphs between F01 and F22.    49 2) The congruent-expression/different-image condition used as adapting stimuli the unmorphed images used to create the alternative series of morphed test images (which the subject never saw). Thus, the A-series group of subjects (1) adapted to the Angry-B images of F01 or F22, before seeing the Angry-A test images that morphed identity between F01 and F22, and (2) adapted to the Afraid-B images of F01 or F22, before seeing the Afraid-A test images of identity morphs between F01 and F22.  3) The incongruent-expression condition used as adapting stimuli the images used to create the morphed test faces with the other expression. Thus, the A-series group of subjects (1) adapted to the Angry-A images of F01 or F22, before seeing the Afraid-A test images that morphed identity between F01 and F22, and (2) adapted to the Afraid-A images of F01 or F22, before seeing the Angry-A test images of identity morphs between F01 and F22.    As a result, the incongruent-expression condition used the same adapting stimuli and same morphed test faces as the congruent-expression condition. The critical difference is that the pairing of adapting and test stimuli was switched. This aspect of experimental design, controls within subjects for any variation in the strength of the adapting power of specific images. The use of the A series of Angry and Afraid images for half the subjects and the B series of Angry and Afraid images for the other half allowed us to balance across subjects the adapting and test stimuli between the congruent-expression and congruent-expression/different-image conditions.   In the experimental trials, a choice screen was displayed after the presentation of each morphed test face. Each choice screen displayed the two unmorphed identities (F01 and F22) used to create the morph series from which the test face was chosen, with the left/right location of F01 versus F22 randomized across trials. Subjects performed a two-alternative forced-choice task and indicated which identity the morphed test face most resembled with a keypress.      503.2.1.3 Apparatus Experiment 1 was designed using Superlab Pro 2.0.4 (www.cedrus.com), and displayed on an HP Compaq nx9600 notebook with a 17? widescreen monitor. Subjects viewed these stimuli from approximately 57 cm viewing distance and in standard dim room lighting.  3.2.1.4 Procedure To familiarize them with the experimental procedure, subjects were first given a short practice version of the experiment made from two other faces. This practice block consisted of 6 trials, and was repeated if subjects failed to understand the instructions. Following the practice block subjects were shown images of F01 and F22 with neutral expressions. They were told that they would be making judgments on facial images morphed between these two individuals, and that they were to make their best guess as to whom the morphed face most resembled.   The experiment consisted of three blocks, one for each experimental condition, presented in a randomized order to each subject. Each block was comprised of the two morph series assigned to that subject and the 4 adapting stimuli appropriate for that experimental condition.  Each adapting stimulus was seen once before each of its 13 respective test stimuli for a total of 52 trials per block and 156 trials in total. Blocks were separated by a short rest break.   Within each block, a trial began with 5s of adaptation to one of the four possible adapting stimuli. Subjects were told to attend to the face on the screen, but not to fixate on a single location. The adapting stimulus was followed by a 50ms mask (a random arrangement of black and white pixels) to reduce apparent motion effects, and then a 300ms morphed test face. Following the test face, a choice screen was displayed and remained on-screen until subjects indicated their response (Figure 3.1). A 500ms blank screen acted as the inter-trial interval. This trial sequence is identical to the one used in our previous study (Fox and Barton, 2007), with     51Figure 3.1  Example of the identity aftereffect trial                                        An example of one experimental trial. Images shown are taken from the Famous Familiar Congruent-Expression condition in Experiment 2. Each trial began with a 5s presentation of an adapting stimulus. This adaptation was followed with a short mask (50ms) to disrupt any apparent motion effects. An identity-ambiguous test stimulus was then presented for 300ms. This was followed by one of two possible choice screens and the subject was asked to choose the identity which most closely resembled the previously viewed test stimulus. The different pairings of adapting and test stimuli created the various experimental conditions.    52timing parameters based on prior studies of the dynamics of face adaptation (Leopold, Rhodes et al., 2005).  3.2.1.5 Analysis For each adapting stimulus we calculated a response score. This was calculated by assigning a 0 or 1 to the two possible identity choices, and averaging this value across the 13 test stimuli associated with that adapting stimulus (Fox and Barton, 2007). All 13 test stimuli were taken from the mid-range of the morph series, placing them on the slope of the psychophysical sigmoid, and ensuring that they were perceived as having an ambiguous identity. As each of the 13 test stimuli were presented only once in each condition, we were unable to plot a psychophysical curve in each individual subject, and morph level was not considered as a factor for further analysis. Instead the response score was used for all statistical analyses. For illustrative purposes we also calculated the mean difference in response scores between pairs of adapting stimuli (e.g., response score after adapting to F01-Angry minus response score after adapting to F22-Angry), which is an index of the adaptation effect. Response scores were entered into a univariate General Linear Model (GLM) with condition (congruent-expression, congruent-expression different-image, incongruent-expression), adapting-face-identity (F01, F22), and adapting-face-expression (angry, afraid) as fixed factors, and subject as a random factor. Post-hoc linear contrasts were performed to examine any significant effects. All statistical analyses were performed on SPSS 14.0 software (www.spss.com). Significance levels were set at ? <0.05.   3.2.2 Results The GLM revealed a significant main effect of adapting-face-identity (F(1,9) = 24.54; p<0.005) indicating a robust identity aftereffect. Post-hoc linear contrasts showed that significant identity aftereffects were generated in all conditions (congruent-expression: t(19) = 3.71; p<0.005,   53congruent-expression different-image: t(19) = 3.09; p<0.01, incongruent-expression: t(19) = 4.15; p<0.005). Other main effects were not significant. We observed a significant interaction between adapting-face-identity and adapting-face-expression (F(1,9) = 6.72; p<0.05), with afraid faces producing a 10% larger identity aftereffect (Mean Difference Score + SEM; 0.24 + 0.04) than angry faces (0.14 + 0.04). No other interactions were significant. The lack of a significant interaction between condition and adapting-face-identity (F(2,18) = 0.34; p>0.5) indicates that there was no difference in the identity aftereffect elicited by the three experimental conditions (Figure 3.2). Changing the adapting image in the congruent-expression/different-image condition did not reduce the identity aftereffect, indicating that the identity aftereffect is not dependent on the specific image used to create the morphed test faces. The fact that we obtained an aftereffect even though adapting and test stimuli had different expressions in the incongruent-expression condition is consistent with adaptation of an expression-invariant representation of identity. The fact that this aftereffect is not significantly less than the aftereffect in the congruent-expression condition suggests little or no contribution to adaptation from an expression-dependent representation of identity.  3.2.3 Comment Experiment 1 used a very similar methodology to our previous study (Fox and Barton, 2007) which examined the influence of identity on adaptation for facial expression. That earlier study showed that the expression aftereffect was much larger when the identities of adapting and test faces were congruent than when these identities were incongruent (Fox and Barton, 2007). These results suggested a hierarchical structure underlying facial expression perception, with identity-dependent representations of expression providing input to identity-invariant representations of expression (Fox and Barton, 2007), analogous to neural networks that model the emergence of viewpoint-invariance from view-specific representations of faces (Rosen, 2003).   54Figure 3.2  Same expression, different expression and different image identity                         aftereffects                                     Experiment 1. (A) Mean response scores (+ SEM) are presented, with significant differences indicated by asterisks. Significant differences in response score following adaptation to F01 versus adaptation to F22 represent a significant identity-aftereffect for that experimental condition. (B). The mean difference in response scores (a quantitative index of the aftereffect) are presented for each experimental condition. Identity aftereffects are found for all three experimental conditions: it is not affected by a change in the image used for the adapting stimulus, even if the expression in the adapting stimulus is no longer congruent with that of the test stimuli. This suggests that, for novel faces, the identity aftereffect is not image-specific and also invariant across changes in facial expression.    55  The results of the present Experiment 1 are different. These data do not provide evidence of a similar pattern of expression-dependence within representations of facial identity. Aftereffects are not modulated by the congruency of facial expression between the adapting stimuli and the test faces.  At the very least, if such expression-dependent representations of identity do exist, their contribution to adaptation is very weak compared to that of expression-invariant representations.   One possible reason for such weak expression-dependent representations is that the faces we used to probe for identity aftereffects in Experiment 1 were novel to the subjects. Other groups have suggested that expression effects in identity processing may vary with the familiarity of the face (Ganel, Goshen-Gottstein et al., 2004; Kaufmann and Schweinberger, 2004). To test the possibility that expression-modulated aftereffects might emerge with more familiar faces, which may have stronger identity representations than novel faces, we performed a second experiment.     3.3  Experiment 2 In Experiment 2, we used pairs of faces that differed in their level of familiarity. As in Experiment 1, for each face pair we created adaptation trials with congruent-expression and incongruent-expression conditions. The first level of facial familiarity used an unnamed novel face pair. These novel faces were different individuals than those used in Experiment 1 but were also not familiar to the subjects prior to testing: hence it was designed to replicate the findings of Experiment 1. The second level of facial familiarity also consisted of a novel face pair; however, in the days preceding testing subjects were shown these faces, which were given arbitrary names, and asked to memorize them and their names. This named novel pair was thus recently but minimally familiar to subjects. The third level of facial familiarity used a famous familiar face pair, which consisted of two celebrity faces. (Previous work showing that the effects of   56expression on identity recognition tasks vary with familiarity used similar comparisons between novel and celebrity faces (Ganel, Goshen-Gottstein et al., 2004; Kaufmann and Schweinberger, 2004).) The fourth level of facial familiarity used a personally familiar face pair. Some studies suggest that the representations of personally familiar faces may differ from those of celebrities (Herzmann, Schweinberger et al., 2004; Kloth, Dobel et al., 2006), possibly because we experience the faces of those in our daily lives in a wider dynamic range (of viewpoint, expression, gaze, etc) than the faces of people in the news, who may be portrayed in more stereotyped views and situations. By using an array of familiarity levels, this experiment will (1) determine more comprehensively whether expression-dependence of identity representations is mediated by familiarity, and (2) identify the level of familiarity at which it emerges, specifically whether a name, semantic knowledge, or personal experience is key to the formation of expression-dependent representations.   3.3.1 Methods  3.3.1.1 Subjects Twelve subjects participated in Experiment 2 (7 female; Age = 29 + 4.97 years). Eight subjects had previously participated in Experiment 1 (including CJF) and four subjects were newly recruited for Experiment 2.   3.3.1.2 Stimuli Due to the limited availability of celebrity images displaying expressions of anger or fear in viewpoints, lighting and resolution suitable for morphing, we chose to use happy and neutral faces in the present design. Happy faces were defined as frontal-view faces with open-mouth smiles, and neutral faces were defined as frontal-view faces with closed mouths and horizontal   57lips. Each familiarity level consisted of two female faces, and two pictures of each face (one happy and one neutral). Unnamed novel faces were two female faces (F15 and F24) selected from the Karolinska Database of Emotional Faces  (Lundqvist and Litton, 1998). Named novel faces were two different female faces (F08 and F25) selected from the Karolinska Database. Famous familiar faces were two female celebrities (Cameron Diaz and Reese Witherspoon) gathered from the Internet. Personally familiar faces were two female lab members who were familiar to all subjects, having been encountered on a nearly daily basis for at least 3 months by all subjects. Eye color was consistent within face pairs. All faces were processed and sized using Adobe Photoshop CS2 9.0.2 as outlined in Experiment 1. Two morph series were made for each familiarity level. Each morph series was made between the two individuals displaying the same expression resulting in a happy and neutral morph series for each familiarity level. Again, the unmorphed endpoints of each morph series were taken as adapting stimuli while the central 13 morphed images were taken as identity-ambiguous test faces. The congruent-expression conditions consisted of adapting and test stimuli taken from the same morph series. The incongruent-expression conditions consisted of adapting stimuli from one morph series and test stimuli from the other morph series.   3.3.1.3 Apparatus Experiment 2 was designed and presented as described in Experiment 1.  3.3.1.4 Procedure All subjects, as in Experiment 1, first participated in a short practice block to ensure they understood the task. Four experimental blocks (unnamed novel, named novel, famous familiar, personally familiar) were presented to subjects in a random order. Before each block subjects were shown unaltered images of the two identities that would be used in that experimental block.   58They were told that they would be making judgments on morphed faces between these two individuals, and that they were to make their best guess as to whom the morphed face most resembled. Trials were organized as described in Experiment 1.  We combined congruent-expression and incongruent-expression conditions within each block. Each block was comprised of 4 adapting stimuli (2 individuals displaying 2 different expressions) and two morph series (one for each expression). Each adapting stimulus was seen once before each of the 13 test stimuli taken from its morph series (congruent-expression) and once before each of the 13 test stimuli taken from the morph series with the other expression (incongruent-expression). This resulted in 104 trials per block and 416 trials in total. Blocks were separated by a short rest break.  3.3.1.5 Analysis Response and mean difference scores were calculated as described in Experiment 1. Response scores were entered into a univariate GLM with familiarity-level (unnamed novel, named novel, famous familiar, personally familiar), adapting-face-identity (Identity-1, Identity-2), adapting-face-expression (happy, neutral), and expression-congruency (expression-congruent, expression-incongruent) as fixed factors and subject as a random factor. Post-hoc linear contrasts were performed to examine any significant effects, with significance levels set at ? <0.05.  3.3.2 Results The GLM revealed a significant main effect of adapting-face-identity (F(1,11) = 37.06; p<0.001). Linear contrasts showed that both congruent-expression and incongruent-expression conditions produced significant identity aftereffects in all four levels of facial familiarity (p<0.001; all contrasts). A significant main effect of familiarity-level was observed (F(3,33) = 4.08; p<0.05), however this was modified by a significant three-way interaction between   59familiarity-level, adapting-face-expression, and expression-congruency (F(3,33) = 8.42; p<0.001). As this interaction did not involve the factor of adapting-face-identity, it does not indicate a difference in magnitude of the identity aftereffect across these interacting factors. Rather, it indicates different thresholds for the various morph series about which these aftereffects occur. All other main effects and interactions were not significant. While facial familiarity has been shown to increase the magnitude of identity aftereffects (Jiang, Blanz et al., 2007), we only see a trend for an interaction between familiarity-level and adapting-face-identity [F(3,33) = 2.42; p=0.08], with personally familiar faces (Mean difference score + SEM; 0.37+0.04) showing larger identity aftereffects than famous familiar (0.22+0.03), named novel (0.30+0.04), or unnamed novel faces (0.28+0.04).  The two key findings of Experiment 2 were, first, the lack of an interaction between adapting-face-identity and expression-congruency (F(1,11) = 2.91, p>0.1), reproducing the finding of Experiment 1, that identity aftereffects are not affected by expression, and second, the lack of a three-way interaction between adapting-face-identity, familiarity-level and expression-congruency (F(3,33) = 1.97, p>0.1; Figure 3.3). This indicates that the lack of modulation of the identity aftereffect by the congruency of facial expression between adapting and test face was not modulated by the familiarity of the faces involved.   3.3.3 Comment Despite the use of different stimuli, different expression pairs (angry-afraid versus happy-neutral), and different arrangements (randomly mixed versus blocked) of expression-congruent and expression-incongruent trials, the unnamed novel face condition of Experiment 2 replicated the results of Experiment 1, with perceptual aftereffects of a similar magnitude of around 20-25%. This reinforces the conclusion that the identity aftereffect with novel faces is not reduced    60Figure 3.3  Novel and familiar face identity aftereffects                                       Experiment 2. (A) Mean response scores  (+ SEM) are presented, with significant differences indicated by asterisks. (B). The mean difference in response scores (a quantitative index of the aftereffect) are also presented for each experimental condition. Significant identity aftereffects are elicited in each experimental condition. The Unnamed Novel conditions, using different novel faces and different facial expressions, replicate the results reported for Experiment 1, showing that the identity aftereffect for novel faces is invariant to changes in facial expression. This invariance to facial expression is also demonstrated in the three other experimental condition (Named Novel, Famous Familiar, and Personally Familiar) representing different levels of facial familiarity. The magnitude of the identity aftereffect is not modulated by the familiarity of the faces used.   61 when facial expression is changed in the adapting stimuli.  The failure of expression to modify the identity aftereffect was reproduced in all four levels of facial familiarity, suggesting further that identity representations are expression-invariant at all levels of facial familiarity.  When contrasted with the results of our previous study (Fox and Barton, 2007), these findings suggest an interesting asymmetry between representations of facial identity and expression: while expression aftereffects are reduced when identity differs between adapting and test stimuli, suggesting some dependence on identity, identity aftereffects are not affected when expression differs, suggesting complete expression-invariance.    What accounts for this difference? One possibility to consider is the following. It may be that in a representational ?face space?, adaptation of the neural representation for a specific face also causes some partial adaptation of faces that are highly similar and share many characteristics with that adapted face. The question then is whether two images of different expressions in the same person are more similar than two images of different people with the same expression. If so, this might account for why changing expression does not reduce identity aftereffects while changing identity does reduce expression aftereffects. We performed Experiment 3 to determine if faces differing in expression but not identity were more similar than faces differing in identity but not expression.      3.4  Experiment 3 Aftereffects in general are modulated by the similarity between the adapting stimulus and the test stimulus. For example, in the classical size aftereffect, after adapting to a test grating pattern of medium spatial frequency, a higher frequency grating will be perceived as an even higher frequency, and a lower spatial frequency grating will appear to be even lower. However, this effect only occurs when the test pattern is within 2-octaves of the adapting frequency on either   62side. If the test pattern is too dissimilar to the adapting pattern, the aftereffect disappears (Blakemore and Sutton, 1969).     Similarly for face adaptation, one would expect that aftereffects would disappear or become reduced if the adapting and test faces are too dissimilar. Our previous study of the expression aftereffect does in fact show this pattern (Fox and Barton, 2007); the reduced aftereffect seen in the different identity condition may simply be due to increased dissimilarity between adapting and test images. Why then is the magnitude of the identity aftereffect not reduced when adapting and test faces have different expressions compared to when they have the same expression? Are the physical or perceived changes in the same face displaying two different expressions too small to have an effect on adaptation?    We explored this possibility using two parallel routes: first by estimating the perceptual distances between face pairs, and second by estimating the physical distances between them. We measured discrimination thresholds for human observers as an indicator of perceptual distances. We compared the contrast thresholds for discriminating pairs of faces (same identity) showing two different expressions (?expression-set?), to the thresholds for discriminating pairs of faces (same expression) of two different individuals (?identity-set?). To estimate physical distances between face pairs, we measured the discrimination thresholds of an ideal observer using the same sets of stimuli.  3.4.1 Methods  3.4.1.1 Subjects Two subjects (CJF and IO) participated in Experiment 3 (1 female; Age = 30 + 2.8 years). Both subjects were experienced psychophysical observers with normal or corrected-to-normal vision.    633.4.1.2 Stimuli Each stimulus set consisted of 12 image pairs. Importantly, the images comprising the expression-set were the adapting stimuli used in Experiment 1 and 2 of the present study and the images comprising the identity-set were the adapting stimuli used in our previous study (Fox and Barton, 2007). Image pairs were not the two endpoints of a particular morph series, but were corresponding endpoints of two different morph series. For example, the images used in Experiment 1 would be paired as follows: F01/AngryA with F01/AfraidA, F22/AngryA with F22/AfraidA, F01/AngryB with F01/AfraidB, and F22/AngryB with F22/AfraidB. In this way we were able to estimate the level of similarity between images used in the congruent and incongruent conditions, and thereby determine whether the level of similarity could explain the difference between congruent- and incongruent-condition aftereffects seen in these experimental conditions.    Image pairs in the expression-set showed one individual displaying two expressions, either (a) an angry and an afraid expression (as described in Experiment 1), or (b) a neutral and a happy expression (as described in Experiment 2). Image pairs in the identity-set showed two individuals displaying the same expression, either fear, anger, disgust, happiness, sadness or surprise (as described in Fox and Barton (2007).    All stimuli were 512 ?  512 pixel in size, which corresponded to 8.5? ?  8.5? visual angle at the viewing distance of 107 cm. The faces were seen through an oval mask that was 254 ?  360 pixels at the central axes. Thus the face-width was approximately 4.2?.    Stimuli were generated using Matlab 7.0, Adobe Photoshop 6.0 and Adobe Illustrator 10 as follows. Digital images of the face stimuli were first converted to grayscale. Then the luminance values were scaled to a range of 0-1. An oval mask was overlaid on the face images and the luminance value outside the oval was set to 0.5 (mid-gray). The luminance of the face   64image, seen through the mask, was normalized to have mean luminance of 0.5 (mid-gray) and standard deviation of 1, such that all face images had equal starting contrast.  3.4.1.3 Apparatus The experiment was run on a computer equipped with a Cambridge Research Systems VSG 2/3 36 MB frame buffer. Stimuli were displayed on a SONY Trinitron 17? monitor (model GDM-200 PS) at 1024 ?  768 resolution. The stimuli luminance values were linearized using an OptiCAL photometer (Model OP200-E) by Cambridge Research Systems via software that generates and saves a gamma-correction look-up table. Mean luminance was 40 cd/m2. The viewing distance was 107 cm.   3.4.1.4 Procedure On each trial the subject first viewed a 500ms fixation cross, and then one of two possible face images for 150ms. This was followed by a choice screen showing the two possible images, which was displayed until the subject completed the two-alternative forced-choice task. Subjects indicated their response with a keypress. Feedback was provided in the form of a single click for a correct response and a double click for an incorrect response. Trials were blocked, with one image pair tested within each block. The order of the 24 blocks, corresponding to the 24 face pairs, was randomized for each subject.    The experimental procedure was coded in Matlab 7.0 using the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997), and the CRS VSG Toolbox for Matlab. Discrimination thresholds at 82% correct were measured with two interleaved staircases that lasted 40 trials each, using the Quest procedure (Watson and Pelli, 1983).      653.4.1.5 Analysis The discrimination threshold estimates for each face pair were obtained by averaging the individual threshold estimates from the two interleaved staircases (i.e., total of 80 trials per threshold). The discrimination thresholds were then submitted to a repeated measures ANOVA with stimulus set (expression-set, identity-set) as a within subjects factor.    3.4.1.6 Ideal observer We ran an ideal observer simulation of this two-alternative forced-choice discrimination using the same sets of stimuli on which the human observers were tested. On each trial, one of two possible face images, F1 or F2, were chosen at random as target stimulus, S, and a zero mean unit variance Gaussian white noise, N, was added to this image at the appropriate contrast, c, as follows: S = Fi,c +N, i=1 or 2, where Fi,c denotes face image i at contrast c. The contrast was determined by the staircase procedure at each trial. The value of the noise variance was arbitrarily chosen to be 1, as we were not looking for a specific level of threshold, but rather any difference between the thresholds for the expression- and identity-sets. The target stimulus contrast on each trial, as well as the statistics of the noise, were available to the ideal observer. The response of the ideal observer was based on a minimum distance rule:  ? ? 2i,ciF . This is equivalent to Bayesian a posteriori maximization as both face images were selected as target stimulus with equal probability (Tjan, Braje et al., 1995).   3.4.2 Results The repeated measured ANOVA using discrimination thresholds obtained from the two human observers showed no difference between the identity-set and expression-set (F(1,1)=3.61, p>0.3; Figure 3.4). Thus, on a perceptual level, changes in expression were not harder to discriminate    66Figure 3.4  Identity and expression discrimination thresholds                         Results from Experiment 3. Mean discrimination thresholds were calculated by averaging the thresholds obtained from the 12 identity-set pairs and 12 expression-set pairs separately. These mean discrimination thresholds (+ SEM) are plotted for the two human observers and the ideal observer.             67than changes in identity. Similarly, a one-way ANOVA on the discrimination thresholds obtained from the ideal observer shows no difference between the identity-set and expression-set (F(1,22)=2.71, p>0.35; Figure 3.4). Image pairs that differ in identity but not expression are as physically dissimilar as image pairs that differ in expression but not identity.  3.4.3 Comment For the stimuli used in this and our prior report, the differences between images of the same expression in different people were no greater than the differences between images of different expressions in the same individual, in either physical (as measured by the ideal observer) or the human perceptual terms (as measured in the two human subjects). Therefore this does not provide support for a proposal that the former are perceptually closer in face space than the latter, a proposal that might provide a simple explanation of the difference in the degree of invariance of identity versus expression aftereffects.       3.5  Discussion The results of Experiment 1 and 2 are consistent: adaptation to identity transfers fully over changes in expression, regardless of the level of familiarity with the person depicted. This contrasts with our previous work, which showed larger expression aftereffects when adapting and test faces had congruent identities than when adapting and test faces had incongruent identities (Fox and Barton, 2007). While the results for expression adaptation suggested a possible hierarchical construction of expression representations, with identity-dependent representations feeding into more abstract identity-invariant representations of expression (Fox and Barton, 2007), the current results for identity aftereffects suggest that adaptation for identity occurs primarily if not solely in an expression-invariant representation.      68  This suggests an asymmetric construction of identity and expression representations within the human visual system (Figure 3.5). Such an asymmetry in the relationship between identity and expression has also been demonstrated through earlier work using Garner?s interference task; irrelevant changes in facial identity strongly retard the speed of facial expression discriminations, while irrelevant changes in facial expression do not affect the speed of facial identity discriminations (Schweinberger and Soukup, 1998; Baudouin, Martin et al., 2002; Schweinberger, Burton et al., 1999). Other experiments examining the interaction between face gender (another temporally invariant feature of faces) and face expression showed similar asymmetry; gender interfered with an expression discrimination task, but expression did not interfere with a gender discrimination task (Atkinson, Tipples et al., 2005). Furthermore, adaptation studies using functional magnetic resonance imaging have shown that the fusiform face area, postulated to be a key cortical region in the processing of identity, is sensitive to changes in facial identity but not expression, whereas the superior temporal sulcus, postulated to be a key cortical region in the processing of expression, is sensitive to changes in both facial identity and facial expression (Winston, Henson et al., 2004).    What might generate such an asymmetry in aftereffects? One potential explanation could be related to the degree of similarity between congruent and incongruent images. One might expect that two images of different people with the same expression would be more dissimilar than two images of the same person with different expressions. Since adaptation to one face reduces responses to other nearby representations in face space (Anderson and Wilson, 2005; Loffler, Yourganov et al., 2005), it may be that identity aftereffects generalize more across expression changes, which may be closer together in face space, than expression aftereffects generalize across identity changes, which may be farther apart in face space. However, in Experiment 3 we found no support for this. Both human and ideal observers showed no difference in the contrast thresholds for discriminating between facial expressions in the same    69Figure 3.5  Asymmetric relationship between identity and expression representations                             A schematic summary of results for the identity and expression aftereffects (Fox and Barton, 2007). The different pattern of results found in these two studies, using very similar methodologies, suggests an asymmetric construction within neural representations associated with face perception. The results are consistent with neural representations of expression that show both identity-dependent and identity-invariant representations (Fox and Barton, 2007), while the current data provide evidence for only an expression-invariant neural representation of identity.          70person and for discriminating between different facial identities with the same expression. Thus, there is no corresponding asymmetry in either the perceptual or physical similarity of faces differing in identity versus expression, to account for the asymmetry in the dependency of aftereffects.   Beyond perceptual and physical similarity, one may speculate upon other reasons for such an asymmetry in the relationship between expression and identity. Expressions and identity may differ in the range of representations involved. While the human visual system encodes thousands of facial identities, some have argued that the many subtle variations of expression can be reduced to a small, finite number of categories (Ekman and Friesen, 1971; Ekman, Sorenson et al., 1969). A dependent layer of representation may be more likely if the range of modulating inputs is large than if it is small.    Behavioral reasons for this asymmetry can also be advanced. It is important that the perception of face identity is impervious to changes in facial expression, so that one can continue to recognize an individual regardless of their emotional state. However, accurate perception of emotional state may require modification of expression perception by the individual?s identity. The ?structural reference theory? of Ganel and Goshen-Gottstein (Ganel, Goshen-Gottstein et al., 2004) proposes that certain faces have structural properties that bias towards certain facial expressions. Learning the structure of these faces leads to compensatory modifications of judgments about the individual?s emotional state. In support of this theory, changes in facial configuration have been shown to influence the perception of facial expression (Martinez and Neth, 2007). Therefore, precise perception of facial expression may require referencing to identity-dependent representations of expression, in addition to generalizations made possible by identity-invariant representations.     A modulation of interactions between facial identity and expression by familiarity had been suggested by two earlier studies (Ganel, Goshen-Gottstein et al., 2004; Kaufmann and   71Schweinberger, 2004). Using Garner?s interference task one study replicated the finding that, with novel faces, irrelevant changes in expression had no effect on the speed of identity discriminations, while irrelevant changes in identity slowed expression discriminations (Ganel, Goshen-Gottstein et al., 2004). However, irrelevant changes in expression did increase reaction times for identity discrimination when famous faces were used (Ganel, Goshen-Gottstein et al., 2004). Inspection of their data, though, shows that interference is still asymmetric, with smaller interference effects for expression changes during identity discrimination than for identity changes during expression discrimination (Ganel, Goshen-Gottstein et al., 2004). The second study measured reaction times during an identification task (Kaufmann and Schweinberger, 2004). Images of celebrities were more rapidly identified when they displayed a slightly happy expression, but the effect of expression was not observed with faces only seen in the context of the experiment (Kaufmann and Schweinberger, 2004). The authors suggested that representations of celebrity identities may have an attached stereotypical expression (Kaufmann and Schweinberger, 2004); an expression-dependent representation of identity. Our studies fail to show any significant impact of familiarity on adaptation. It may be the adaptation methods we employ probe slightly different physiologic events than those probed with interference or recognition paradigms. For example, while interference may stem from interactions between representations in the visual system, adaptation effects may probe the variance of those representations.    In summary, our experiments demonstrate expression-invariance of the identity aftereffect, regardless of the level of the observer?s familiarity with the faces used, and suggest that the neural representations underlying the perception of the identities of both novel and famous faces are expression-independent. This contrasts with our earlier work using a similar adaptation paradigm, which provided evidence consistent with both identity-dependent and identity-independent representations of facial expression (Fox and Barton, 2007). Together, these   72data suggest an asymmetric construction of identity and expression representations. Expression-invariant representations of identity can be achieved in some perceptual models (Bronstein, Bronstein et al., 2007), and our results may point to important ways in which the encoded representations of expression and identity differ in the human visual system.   73  3.6  References ANDERSON ND and WILSON HR. The nature of synthetic face adaptation. Vision Res, 45: 1815-28, 2005. ATKINSON AP, TIPPLES J, BURT DM, and YOUNG AW. Asymmetric interference between sex and emotion in face perception. Percept Psychophys, 67: 1199-1213, 2005. BAUDOUIN JY, MARTIN F, TIBERGHIEN G, VERLUT I, and FRANCK N. Selective attention to facial emotion and identity in schizophrenia. Neuropsychologia, 40: 503-11, 2002. BLAKEMORE C and SUTTON P. Size adaptation: A new aftereffect. Science, 166: 245-7, 1969. BRAINARD DH. The psychophysics toolbox. Spat Vis, 10: 433-6, 1997. BRONSTEIN AM, BRONSTEIN MM, and KIMMEL R. Expression-invariant representations of faces. IEEE Trans Image Process, 16: 188-97, 2007. BRUCE V and YOUNG A. Understanding face recognition. Br J Psychol, 77 ( Pt 3): 305-27, 1986. BUTLER A, ORUC I, FOX C, and BARTON J. Factors contributing to the adaptation aftereffects of facial expression. Brain Res, 1191: 116-126, 2008. CALDER AJ and YOUNG AW. Understanding the recognition of facial identity and facial expression. Nat Rev Neurosci, 6: 641-51, 2005. DE GELDER B, FRISSEN I, BARTON J, and HADJIKHANI N. A modulatory role for facial expressions in prosopagnosia. Proc Natl Acad Sci U S A, 100: 13105-10, 2003. EKMAN P and FRIESEN WV. Constants across cultures in the face and emotion. J Pers Soc Psychol, 17: 124-9, 1971. EKMAN P, SORENSON ER, and FRIESEN WV. Pan-cultural elements in facial displays of emotion. Science, 164: 86-8, 1969. FOX CJ and BARTON JJ. What is adapted in face adaptation? The neural representations of expression in the human visual system. Brain Res, 1127: 80-9, 2007. GANEL T, GOSHEN-GOTTSTEIN Y, and GANEL T. Effects of familiarity on the perceptual integrality of the identity and expression of faces: The parallel-route hypothesis revisited. J Exp Psychol Hum Percept Perform, 30: 583-97, 2004. GANEL T, VALYEAR KF, GOSHEN-GOTTSTEIN Y, and GOODALE MA. The involvement of the "Fusiform face area" In processing facial expression. Neuropsychologia, 43: 1645-1654, 2005. HAXBY JV, HOFFMAN EA, and GOBBINI MI. The distributed human neural system for face perception. Trends Cogn Sci, 4: 223-233, 2000. HERZMANN G, SCHWEINBERGER SR, SOMMER W, and JENTZSCH I. What's special about personally familiar faces? A multimodal approach. Psychophysiology, 41: 688-701, 2004. HUMPHREYS K, AVIDAN G, and BEHRMANN M. A detailed investigation of facial expression processing in congenital prosopagnosia as compared to acquired prosopagnosia. Exp Brain Res, 176: 356-373, 2007. JIANG F, BLANZ V, and O'TOOLE AJ. The role of familiarity in three-dimensional view-transferability of face identity adaptation. Vision Res, 47: 525-31, 2007. KAUFMANN JM and SCHWEINBERGER SR. Expression influences the recognition of familiar faces. Perception, 33: 399-408, 2004. KLOTH N, DOBEL C, SCHWEINBERGER SR, ZWITSERLOOD P, BOLTE J, and JUNGHOFER M. Effects of personal familiarity on early neuromagnetic correlates of face perception. Eur J Neurosci, 24: 3317-21, 2006. LEOPOLD DA, O'TOOLE AJ, VETTER T, and BLANZ V. Prototype-referenced shape encoding revealed by high-level aftereffects. Nat Neurosci, 4: 89-94, 2001. LEOPOLD DA, RHODES G, MULLER K-M, and JEFFERY L. The dynamics of visual adaptation to faces. Proceedings of the Royal Society B, 272: 897-904, 2005.   74LOFFLER G, YOURGANOV G, WILKINSON F, and WILSON HR. Fmri evidence for the neural representation of faces. Nat Neurosci, 8: 1386-90, 2005. LUNDQVIST D and LITTON JE. The averaged karolinska directed emotional faces-akdef, 1998. MARTINEZ A and NETH D. Face configuration biases the perception of facial expressions [abstract]. Journal of Vision, 7: 943a, 2007. PALERMO R and RHODES G. Are you always on my mind? A review of how face perception and attention interact. Neuropsychologia, 45: 75-92, 2007. PELLI DG. The videotoolbox software for visual psychophysics: Transforming numbers into movies. Spat Vis, 10: 437-42, 1997. ROSEN E. Face representation in cortex: Studies using a simple and not so special model. Secondary Titl. Boston: Massachusetts Institute of Technology. Masters of Engineering in Electrical Engineering and Computer Science: 75, 2003. SCHWEINBERGER SR, BURTON AM, and KELLY SW. Asymmetric dependencies in perceiving identity and emotion: Experiments with morphed faces. Percept Psychophys, 61: 1102-1115, 1999. SCHWEINBERGER SR and SOUKUP GR. Asymmetric relationships among perceptions of facial identity, emotion, and facial speech. J Exp Psychol Hum Percept Perform, 24: 1748-65, 1998. STEPHAN BC, BREEN N, and CAINE D. The recognition of emotional expression in prosopagnosia: Decoding whole and part faces. J Int Neuropsychol Soc, 12: 884-95, 2006. TJAN BS, BRAJE WL, LEGGE GE, and KERSTEN D. Human efficiency for recognizing 3-d objects in luminance noise. Vision Res, 35: 3053-69, 1995. WATSON AB and PELLI DG. Quest: A bayesian adaptive psychometric method. Percept Psychophys, 33: 113-20, 1983. WEBSTER MA, KAPING D, MIZOKAMI Y, and DUHAMEL P. Adaptation to natural facial categories. Nature, 428: 557-61, 2004. WINSTON JS, HENSON RN, FINE-GOULDEN MR, and DOLAN RJ. Fmri-adaptation reveals dissociable neural representations of identity and expression in face perception. J Neurophysiol, 92: 1830-9, 2004.               754   FACTORS CONTRIBUTING TO THE ADAPTATION AFTEREFFECTS OF      FACIAL EXPRESSION3    4.1  Introduction Adaptation paradigms are commonly used to characterize the mechanisms and neural representations involved in the perception of various visual dimensions, such as color, orientation, and frequency, among others. One classic example is the color aftereffect, where an observer perceives a green after-image square after adapting to a red square for many seconds (e.g. Clifford and Rhodes, 2005). While simple color aftereffects can be explained by adaptation of color-opponency cells in the retina, recent experiments have also shown adaptation aftereffects for high-level visual stimuli such as faces, across such dimensions as identity, gender, race and expression (Leopold, O'Toole et al., 2001; Webster, Kaping et al., 2004; Fox and Barton, 2007).    All of these face adaptation studies have used similar paradigms that involve morphed faces. Essentially, subjects view a particular face during an adaptation period that lasts several seconds and then are shown ambiguous test images that are created by morphing between that face and another; adaptation causes these subjects to respond that the morphed images are less similar to the face they had viewed during the adaptation period. This aftereffect is attributed to a reduction in neural responses evoked by the adapting face (Huber and O'Reilly, 2003). When an ambiguous image is viewed following adaptation, the responses in competing, unadapted representations of other faces are stronger than the response in the adapted representation, leading to a bias of perception towards unadapted stimuli (Leopold, O'Toole et al., 2001).  This conceptualization of the origins of face aftereffects suggests that adaptation studies may be a useful and important means of probing the nature of the neural representations of faces and the                                                  3 A version of this chapter has been published. Butler A., Oruc I., Fox C.J., and Barton J.J.S (2008). Factors contributing to the adaptation aftereffects of facial expression. Brain Research, 1191: 116-126. ? 2007 Elsevier B.V.   76organization of ?face space?, the theoretical multidimensional relationship of facial representations in the human visual system (Rhodes, Brennan et al., 1987; Valentine, 1991), and the impact of phenomena such as learning on these representations (Wilson and Diaconescu, 2006).     However, before such inferences about the neural representations of faces can be made from adaptation studies, one must first consider whether local adaptation to image elements in face stimuli might account for the observed aftereffects. Complex stimuli like faces, after all, are composed of many different local elements, which include, line orientation, curvature, and shape, all properties for which aftereffects have been described (Regan and Hamstra, 1992; Suzuki and Cavanagh, 1998; Adams and Mamassian, 2002; Suzuki, 2003; Blakemore and Sutton, 1969; Gibson and Rander, 1937). If local adaptation to these image elements accounts for a significant portion of face aftereffects, this would significantly constrain the inferences about face space that can be derived from face adaptation studies.    Some studies have argued that if the aftereffect persisted despite changes in stimulus size or location, such invariance would argue against a significant contribution of local adaptation to image elements towards face aftereffects. Thus Zhao & Chubb (2001) found that, while aftereffects to distorted faces were largest when the adapting and test stimuli were of the same size, significant aftereffects could still be obtained even when the two stimuli differed in size by a factor of four. Leopold et al. (2001) found that the magnitude of the aftereffect was not altered even if the location of adapting and test stimuli differed by as much as 6 degrees.    In a recent study of facial expression aftereffects, Fox & Barton (2007) showed that the aftereffects persisted even when adapting and test stimuli were images of different individuals, suggesting that neural representations of expression are identity-invariant. However, because expression aftereffects generated by incongruent identities were smaller than ones generated by congruent identities, a second, identity-dependent representation of expression was also   77postulated (Fox and Barton, 2007). While identity-invariant expression aftereffects are unlikely to be the result of local adaptation to image elements, as the adapting and test stimuli are pictures of different people, identity-dependent expression aftereffects may reflect such local adaptation. To counter this, Fox & Barton (2007) noted that the magnitude of the identity-dependent expression aftereffect was not affected when the adapting stimulus was changed to a different picture of the same expression in the same individual, than the image used to create the morphed test stimuli.    However, different depictions of the same expression by the same person are still highly similar. Thus, this finding (Fox and Barton, 2007) does not definitively exclude a role for local adaptation to image elements in expression aftereffects. For example, one could suppose that adaptation to tilt mediates aftereffects of the expression of anger, since this emotion is usually associated with a downward tilt of the eyebrows. This tilt would still be present in two different images of the same person displaying anger. Likewise, adaptation to the curvature of the mouth could contribute to aftereffects of the expression of happiness across different images of the same person. The goal of the first experiment of this report, then, was to perform a more definitive examination of the role of local adaptation to image elements in the face expression aftereffect.   Beyond the role of local adaptation to image elements, there is also the question of whether adaptation effects originate in representations of facial features or whole-face configurations. While facial features can clearly be recognized and exist independent of the whole-face context, many face-related phenomena are said to depend upon or reflect ?holistic? processing, in which the configuration and inter-relation of features in their natural facial context is critical (Tanaka, Kay et al., 1998; de Heering, Houthuys et al., 2006; Goffaux and Rossion, 2006; Singer and Sheinberg, 2006; Yovel and Duchaine, 2006). To examine the role of both facial features and configuration in face adaptation, we performed two additional experiments,   78one in which features were presented without the normal facial configuration, and another in which the normal facial configuration was preserved but presented with minimal (or impoverished) feature information.    4.2  Methods  4.2.1 Subjects Twelve subjects participated in all three experiments (8 female; Age = 27.0 +10.8 years). All participants had normal or corrected-to-normal vision. The institutional review boards of Vancouver General Hospital and the University of British Columbia approved the protocol and all subjects gave informed consent in accordance with the declaration of Helsinki.   4.2.2 Experiment 1  4.2.2.1 Stimuli We created three types of adapting stimuli to be used in the three conditions (Figure 4.1A), which were the Normal-Face condition, the Quartered-Face/Consistent-Expression condition, and the Quartered-Face/Mixed-Expression condition. All stimuli were created from the facial images of two individuals (one female, F22, and one male, M17) from the Karolinska Database of Emotional Faces (Lundqvist and Litton, 1998) showing one of two expressions (afraid or angry).        79Figure 4.1  Examples of stimuli for examining the expression aftereffect                  (A) Experiment 1 (i) Normal-Face condition, using angry face A, (ii) Quartered-Face/Consistent-Expression condition, with the upper right and lower left quarters from angry face A and the lower right and upper left quarters from angry face B, (iii) Quartered-Face/Mixed-Expression condition, with the upper right and lower left quarters from angry face A and the lower right and upper left quarters from afraid face A. (B) Experiment 2 (i) Intact-Face condition, (ii) Scrambled-Face condition. (C) Experiment 3 (i) Intact-Schematic-Face condition, (ii) Scrambled-Schematic-Face condition.       804.2.2.1.1  Normal-Face condition (i) Adapting stimuli: In the Normal-Face condition, these were unaltered images of F22 and M17, displaying anger or fear (Figure 4.2A). The database has two images of each expression per person, which we labeled as A and B versions. We paired these arbitrarily to create an A pair and a B pair of afraid and angry images.   (ii) Test stimuli: Using these A and B pairs, our second step was to create test stimuli for the experiment by generating a morph series between the two opposite adapting stimuli of the same person, using FantaMorph 3.0 software (www.fantamorph.com). This created 4 morph series, an A and a B series for F22, and an A and a B series for M17 (i.e. angryA/afraidA indicates a morph between angry face A and afraid face A, while angryB/afraidB indicates a morph between angry face B and afraid face B). From these morph series we selected for our test stimuli thirteen morphs that ranged from 35% angry/65% afraid to 65% angry/35% afraid in 2.5% steps. This range was chosen to ensure that the displayed facial expression was ambiguous for all test stimuli.    In the experiment, half of the subjects saw series A, in which the two adapting stimuli were angry face A and afraid face A and the test stimuli were selected from the morph series between angry face A and afraid face A. The other half saw series B, in which the adapting stimuli and all morphed test stimuli were derived from angry face B and afraid face B.   4.2.2.1.2 Quartered-Face/Mixed-Expression condition (i) Adapting stimuli: We took the same unmorphed faces used as adapting stimuli in the Normal-Face condition and divided them into quarters, using Adobe Photoshop CS 8.0 (Adobe Systems, San Jose CA), with the tip of the nose serving as the center point of each face (Figure 4.2B). We then recombined these quarters, so that the upper left and lower right quarters were from the face showing one expression (e.g. afraid face A), while the lower left and upper right quarters were    81Figure 4.2  Creation of normal and quartered-face stimuli                               (A) Normal-Face condition. The A versions of the Angry and Afraid images for F22 are shown. These will be used as adapting stimuli in the Normal-Face condition. A morph series is then generated between these two images, and 13 stimuli with morph mixtures ranging from 65% angry/35% afraid to 35% angry/65% afraid will be used as test stimuli for the adaptation effect. The smaller images shown are three representative images from this morph series: the 65% angry/35% afraid, 50% angry/50% afraid, and 35% angry/65% afraid morphed images. (B) Quartered-Face/Mixed-Expression condition. The images used in the Normal-Face condition are divided into quarters, and recombined into two images (#1 and #2) that are a mixture of angry and afraid quarters, one (#1) in which the upper right and lower left quarters are angry and the lower right and upper left quarters are afraid, and another (#2) that is the reverse. These will be used as adapting stimuli in the Quartered-Face/Mixed-Expression condition. A morph series is then generated between these two images, and 13 stimuli with morph mixtures ranging from 65% stimulus#1/35% stimulus#2 to 35% stimulus#1/65% stimulus#2 will be used as test stimuli for the adaptation effect. Smaller images show the 35%/65%, 50%/50%, and 65%/35% images taken from this series. (Note: the superimposed colors are for illustrative purposes only, to clarify the principle of stimulus design. These range from orange for 100% angry to blue for 100% afraid, with morphs having intermediate values. Stimuli were not colored in the experiment.)    82from the face showing the other expression (e.g. angry face A) (Figure 4.1A.iii). Thus the two adapting stimuli in the Quartered-Face/Mixed Expression condition (Adapting stimulus 1 and Adapting stimulus 2 in Figure 4.2B) were simply a mix of the two adapting stimuli used in the Normal-Face condition (Afraid and Angry stimuli in Figure 4.2A). As subjects were exposed to both adapting stimuli (Afraid and Angry in the Normal-Face condition, and Adapting stimulus 1 and Adapting stimulus 2 in the Quartered-Face/Mixed Expression condition), subjects saw precisely the same set of image quarters during adaptation in each of these experimental conditions, only in a different combination with the other quarters. Hence the local image elements of the faces used as adapting stimuli were preserved across these two conditions, with the only differences being the consistency of the displayed expression in a single adapting image, and any edges introduced at the margins of the facial quarters (for which the Quartered-Face/Consistent-Expression condition served as a control). Quartering rather than halving ensured that both right and left hemifaces as well as upper and lower face halves in themselves contained a hybrid mixture, which was important since there is evidence that attention to faces differs between both the right and left and also the upper and lower halves (Shepherd, Davies et al., 1981; Barton, Radcliffe et al., 2006; Butler, Gilchrist et al., 2005).   (ii) Test stimuli: In the Quartered-Face/Mixed Expression condition, these were created from the quartered images used as adapting stimuli (Figure 4.2B). We generated morph series between Adapting Stimulus 1 and Adapting Stimulus 2, and selected as test stimuli the thirteen morphs that ranged from 35% Adapting stimulus 1/65% Adapting stimulus 2 to 65% Adapting stimulus 1/35% Adapting stimulus 2 in 2.5% steps. Note that this essentially resulted, for example, in a 35:65 morph image having two quarters with a 35% angry/65% afraid mixture and two quarters with a 65% angry/35% afraid mixture. Hence, as with the adapting stimuli, across the entire block the subjects saw exactly the same quarter images during the test phase in both     83the Normal-Face and Quartered-Face/Mixed Expression conditions, merely combined in different ways in the single images.   Again, there were two versions of the test, one using the A images and one using the B images. Those subjects who saw series A images in the Normal-Face condition also saw quartered stimuli generated from the series A images in the Quartered-Face/Mixed Expression condition, while those subjects who saw series B images in the Normal-Face condition saw quartered stimuli generated from the series B images in the Quartered-Face/Mixed Expression condition. Thus the experimental design was balanced within subjects with regard to stimulus components.  4.2.2.1.3 Quartered-Face/Consistent-Expression condition (i) Adapting stimuli: To control for the potential effects of the quartering procedure in the Quartered-Face/Mixed Expression condition, we created a third condition, the Quartered-Face/Consistent-Expression condition. Stimuli in this condition also had edges at the margins of the facial quarters, but differed from the Quartered-Face/Mixed Expression condition in that all quarters had a similar rather than different facial expression. These were created by combining quadrants from the A image and the B image of the same expression by the same person (Figure 4.1A.ii), the same images used in the Normal-Face condition. As the second set of images used to create the adapting stimuli in the Quartered-Face/Consistent-Expression were not seen in the Normal-Face condition by that subject, low-level properties of the faces were not balanced for this condition within a particular subject, although they were balanced across subjects (see Procedure).    (ii) Test stimuli: We next created morphed images from these quartered adapting stimuli. There were again four morph series, two for F22 and two for M17. For each person there was one morph series in which the left-upper and right-lower quadrants were derived from the A   84series of facial images and the right-upper and left-lower quadrants from the B series of facial images, and one morph series with the reverse arrangement. Again, we chose as test stimuli the thirteen morph images ranging from 35/65% to 65/35%.  4.2.2.2 Apparatus Experimental procedure was implemented in SuperLab 1.71 software (www.cedrus.com). A Mac G4 PowerBook with a 17? widescreen display was used in the setting of ambient background lighting, and viewed from a distance of 54 cm. All stimuli were presented in the centre of the screen and subtended 12 deg ?  15 deg of visual angle.  4.2.2.3 Procedure All subjects learned the task through written instructions and a short practice session of ten trials before starting the experiment. Each trial (Figure 4.3) began with a 5-second presentation of an adapting stimulus, which was one of the two endpoint images of a given morph series. This was followed by a 50 ms mask of random black and white pixels. The mask was followed by a 300 ms test stimulus, picked randomly from the series of 13 morphed images corresponding to the method of constant stimuli. A choice screen was then presented, which displayed both adapting images side-by-side, labeled ?1? and ?2?. The displayed position of the two adapting stimuli (left versus right) on the choice screen varied throughout the experiment. This pattern of choice screens was balanced by randomly assigning half of the subjects to the opposite sequence of choice screens. The subject was asked to indicate with a keypress which of the two choice faces the test stimulus most strongly resembled.  A 500 ms blank screen then appeared, followed by the start of the next trial.      85Figure 4.3  Example of the expression aftereffect trial                    Stimuli from the Normal-Face condition of Experiment 1  are used to demonstrate a typical trial. Each trial began with the presentation of one of two possible adapting face stimuli (i.e. either the afraid or angry face). The adapting stimulus was followed by the presentation of a mask. This was followed by the test stimulus, an image formed by morphing between the two possible adapting stimuli. One of two choice screens then followed, which presented the two possible adapting faces. At this point, the subject had to indicate which of the two faces the test image most closely resembled (i.e. either the angry or afraid face).      86  The three different conditions (Normal-Face, Quartered-Face/Consistent-Expression, and Quartered-Face/Mixed-Expression) were tested in separate blocks, with the order randomly determined for each subject. Subjects were also randomly assigned to one of four versions of the experiment. All four versions included both the male (M17) and the female (F22) face as stimuli. Two versions used the A series of images for the Normal-Face and Quartered-Face/Mixed-Expression conditions and the other two used the B series of images. These two experimental versions differed in turn by whether they used the first or the second of the versions of the Quartered-Face/Consistent Expression condition.    Each pairing of a given test with a given adapting stimulus occurred only once. With thirteen test stimuli in each morph series for both the male and the female face, and each stimulus seen twice, once after each of the two non-morphed adapting stimuli (e.g. angry adapting stimulus and afraid adapting stimulus for the Normal-Face or Quartered-Face/Consistent-Expression conditions, and adapting stimulus 1 and adapting stimulus 2 for the Quartered-Face/Mixed-Expression condition), there were a total of 52 trials for each condition, all presented in random order within a block. With three blocks, there were a total of 156 trials.  4.2.2.4 Analysis In the Normal-Face condition, the ?afraid? response was assigned a value of 0, and the ?angry? response was assigned a value of 1. For each adapting stimulus (angry or afraid) we calculated a score that was the proportion of responses where the subject selected the angry face across the full series of 13 test stimuli. As all 13 test stimuli were taken from the mid-range of the morph series and each stimulus was presented only once in each condition, morph level was not considered as a factor for further analysis. This response score, collapsed across all levels of morph, was our behavioral measure for the Normal-Face condition. The response score was   87calculated in a similar manner for the Quartered-Face/Consistent-Expression and the Quartered-Face/Mixed-Expression conditions.   A repeated measures ANOVA was performed with the response score as the dependent measure and significance levels set at p < .05, using JMP IN 5.1 software (SAS Institute, www.jmpin.com). Face-condition (Normal-Face, Quartered-Face/Consistent-Expression, Quartered-Face/Mixed-Expression) and adapting stimulus (image 1, image 2) served as within-subject factors. A difference score was calculated between the response score for the first adapting stimulus and that for the second (opposite) adapting stimulus. This difference score was used for graphical purposes only.   4.2.3 Experiment 2  4.2.3.1 Stimuli We used the same faces as Experiment 1. In this experiment, however, each original image was manipulated in Adobe Photoshop CS 8.0 by applying a number of filters, including grayscale and high-pass, to create black-and-white thresholded images, so that scrambling of the features could be performed without leaving visible segmentation boundaries in the image. The thresholded images were used as adapting stimuli (Figure 4.1B.i) and to generate the morphed test stimuli for the Intact-Face condition, in the same manner as in Experiment 1.   In the Scrambled-Face condition, the features (eyes, mouth, eyebrows and nose) of the non-morphed thresholded images were rearranged in an identical manner for all images, angry and afraid (Figure 4.1B.ii). These served as the adapting and choice stimuli in the trials. The morphing process was then applied to these new scrambled configurations to generate the morphed test stimuli (hence both adapting stimuli and test stimuli were identically scrambled   88arrangements in the Scrambled-Face condition), which at the featural level consisted of physical changes equal in magnitude to the test stimuli in the Intact-Face condition.   4.2.3.2 Procedure  Trials were similar to those in Experiment 1. Each began with a 5-second view of an adapting stimulus, followed by a 50-ms mask, a 300-ms view of a test stimulus, and then the choice screen. There were two blocks of trials, one of 52 trials for the Intact-Face condition, and one of 52 trials for the Scrambled-Face condition. With both blocks, there were a total of 104 trials. Half of the subjects began with the Intact-Face block and the other half with the Scrambled-Face block.   4.2.3.3 Analysis As in Experiment 1, a repeated measures ANOVA was performed with the response score as the dependent measure, and face-condition (Intact-Face, Scrambled-Face) and adapting stimulus (image 1 with angry elements, image 2 with afraid elements) as within-subject factors.   4.2.4 Experiment 3  4.2.4.1 Stimuli We created a new set of stimuli with Adobe Illustrator CS 8.0. Angry and happy expressions consisted of simple black lines, curves and circles. Happy faces were used instead of afraid faces as it was difficult to convey fear in these schematic faces. For the Intact-Schematic-Face condition, we arranged lines in a facial configuration within an oval outline, with tilted eyebrow lines and mouth curves in opposite directions to create the baseline adapting stimuli of angry and happy faces (Figure 4.1C.i). Thirteen test stimuli were created by systematically changing the   89mouth curvature and eyebrow orientation between the 35/65 and the 65/35 points in the happy-afraid transition images in 2.5% steps, analogous to the morphed images in Experiment 1 and 2. Here the 50/50 mid-position between happy and afraid was characterized with flat horizontal lines for the eyebrow and mouth. The adapting and test stimuli for the Scrambled-Schematic-Face condition were created in a similar fashion, but with the line elements in a non-facial configuration within the same oval outline (Figure 4.1C.ii). Again, the adapting stimuli and the test stimuli had identical scrambled arrangements in the Scrambled-Schematic-Face condition.  4.2.4.2 Procedure  Trials were similar to those in Experiments 1 and 2, with a 5-second viewing of one of the two adapting stimuli, followed by the 50-ms mask, the 300-ms test stimulus, and the choice screen showing the two possible extreme stimuli. There were two blocks, one for the Intact-Schematic-Face condition and one for the Scrambled-Schematic-Face condition. In each block all 13 test stimuli were presented twice, once after each of the two adapting stimuli, for a total of 26 trials per block and 52 trials in the experiment.   4.2.4.3 Analysis We ran a repeated measures ANOVA with the response score as the dependent measure, and face-condition (Intact-Schematic-Face, Scrambled-Schematic-Face) and adapting stimulus (image 1 with angry elements, image 2 with happy elements) as within-subject factors.          90  4.3  Results  4.3.1 Experiment 1 To determine whether expression aftereffects result from adaptation of high-level representations of facial expression, or from local adaptation to image elements we created hybrid face stimuli (Figure 4.1A.iii) that lacked a consistent expression but which still possessed the same local elements of the normal faces used to generate an expression aftereffect (Figure 4.1A.i). These faces were created by quartering the original pictures, so that two diagonally opposite quarters (e.g. upper-left and lower-right) were taken from one facial expression (e.g. angry), and the remaining two from a different expression (e.g. afraid) (Figure 4.2B). Our goal was to test whether these quartered stimuli generated aftereffects, and if so, how they compared to the aftereffects obtained by adaptation to faces with consistent facial expressions. As faces no longer contained consistent expressions, subjects did not perform an explicit expression-naming task, but were asked to match the test face to one of the two adapting stimuli displayed on a choice screen (see Figure 4.3). Thus if expression aftereffects are simply due to local adaptation to image elements, then significant aftereffects should be found with both types of stimuli: that is, regardless of the presence or absence of a strong consistent expression. On the other hand, if aftereffects arise solely from adaptation of a high-level representation of facial expression, there should be no measurable aftereffect when adapting to the quartered faces with ambiguous expressions.   Our experiment had three face-conditions. In the Normal-Face condition, whole faces with strong expressions of anger or fear were used as adapting stimuli. In the Quartered-Face/Mixed-Expression condition, the normal faces were quartered and re-arranged to give stimuli with inconsistent expressions, ensuring that subjects were exposed to the same set of image quarters during the adapting phase of this condition as in the Normal-Face condition.   91Quartered stimuli were used as adapting stimuli and were also used to create quartered test stimuli. Lastly, we included a Quartered-Face/Consistent-Expression condition, to control for the edges introduced during the process of quartering. In this condition, hybrid images were constructed from two different pictures of the same expression in the same individual. Again, these quartered stimuli were used for both the adapting and test stimuli.    In all three conditions, subjects adapted to one of two adapting stimuli at the extreme ends of a morph continuum, and then were tested for aftereffects on test stimuli selected from the mid-region of the continuum. The analysis examined for effects of face-condition and adapting stimulus, with aftereffects indicated by the presence of a significant effect of adapting stimuli. The results did show a significant main effect of adapting stimulus (F(1, 127)= 101, p<.0001). More importantly, there was a significant interaction between face-condition and adapting stimulus (F(2, 127) = 21.17, p<.0001). Linear contrasts showed that there was a significant difference in response following adaptation to the two different adapting stimuli in the Normal-Face condition (t = 9.32, p < .0001) and also in the Quartered-Face/Consistent-Expression condition (t = 7.55, p < .0001), but not for the Quartered-Face/Mixed-Expression (t = 0.61, n.s.) (Figure 4.4). Thus there were large aftereffects for the two conditions with adapting stimuli that displayed consistent expressions, but none when the facial expression in the adapting stimuli was inconsistent. Even though the images in the Quartered-Face/Mixed-Expression condition were produced from the same photographs and therefore contained the same local elements as in the Normal-Face condition, the aftereffect was abolished by disrupting the consistency of the displayed facial expression.   These results show that presenting the same local image elements in a way that reduces the coherence of the depicted expression eliminates the aftereffect. The loss of this aftereffect cannot be attributed to a distortion in the image introduced by the quartering process, since a similar mixing of two different images in the Quartered-Face/Consistent-Expression condition    92Figure 4.4  Expression aftereffects using quartered-face stimuli                    Top graph shows the mean frequency of responses that the test stimuli looked like Face 2 after adapting to either Face 1 or Face 2. Bottom graph shows the magnitude of the aftereffect, expressed as the mean difference score, which is calculated from the data in the top graph by subtracting for each subject the frequency of answering face 2 after adapting with face 2 from the frequency of answering face 2 after adapting with face 1 (if there is no aftereffect from previewing images A or B, the frequencies should be the same, and the difference score would be zero). Data are shown for each of the three conditions (Left = Normal-Face, Middle =  Quartered-Face/Consistent-Expression, Right = Quartered-Face/Mixed-Expression) with error bars showing one standard deviation and asterisks denoting significant aftereffects. (For Normal-Face and  Quartered-Face/Consistent-Expression conditions, Face 1 is Afraid and Face 2 is Angry.)      93did not reduce the magnitude of the aftereffect. These findings indicate that the aftereffects in our experiment are not due to local adaptation to image elements such as orientation, curvature or shape. If such local adaptation occurred we would have expected to see at least some aftereffects in the Quartered-Face/Mixed-Expression condition, whereas we found none.    The results of Experiment 1 also raise an interesting question. Within each quartered-image there were facial features in which expression could be identified; however, aftereffects were absent in the Quartered-Face/Mixed-Expression condition. Does this mean that simultaneous adaptation to two different facial expressions disrupts the resultant aftereffect or that adaptation of neural representations of facial expression does not occur at the level of facial features? Our second experiment was designed to address this issue.  4.3.2 Experiment 2 There is a large body of work suggesting that many aspects of face processing may occur holistically, rather than at the level of components or individual features (Tanaka, Kay et al., 1998; de Heering, Houthuys et al., 2006; Goffaux and Rossion, 2006; Singer and Sheinberg, 2006; Yovel and Duchaine, 2006). The lack of an aftereffect in the Quartered-Face/Mixed-Expression condition of Experiment 1 is consistent with this view. However, the ambiguity of the overall expression in the Quartered-Face/Mixed-Expression condition may have obscured aftereffects generated by facial features. To determine if expression aftereffects can occur at the level of facial features, our second experiment compared aftereffects following adaptation to normal faces with those following adaptation to scrambled faces, in which facial configuration is disrupted but the expression remains consistent. If expression is represented at the level of individual features (e.g. eyes, mouth) then at least some aftereffects should be found in the scrambled condition.   94  This experiment had two conditions, both using thresholded facial images that allowed us to scramble images without producing edge artifacts (see Methods). The Intact-Face condition showed thresholded versions of faces with features in a normal arrangement in both the adapting and test stimuli, while the Scrambled-Face condition showed a scrambled arrangement of facial features in both the adapting and test stimuli.   We found a significant main effect of adapting stimulus, with a mean 19% difference in scores between adaptation to the angry versus the afraid images (F(1, 81)= 17.1, p<.0001). However, there was no significant interaction between face-condition and adapting stimulus (F(1, 81)= 0.99, p = n.s.). A priori linear contrasts showed significant aftereffects for both the Intact-Face condition (t= 3.634, p < 0.0005) and the Scrambled-Face condition (t = 2.220, p < 0.03) (Figure 4.5).   The results of Experiment 2 suggest that at least some of the expression aftereffect may be mediated at the level of facial features, since an aftereffect was found in the Scrambled-Face condition. Thus, individual features are able to activate adaptable neural representations of expression even when the normal facial configuration was absent.    While the aftereffect in the Scrambled-Face condition appeared smaller than the aftereffect for the Intact-Face condition, the interaction between face-condition and adapting stimulus was not significant. The lack of interaction might be interpreted as suggesting that the normal facial configuration does not contribute to the expression aftereffect. This would be surprising given the prominent role of configuration in many aspects of face processing. To determine if configuration contributed to the expression aftereffect, we performed a third experiment, in which the perception of expression depended upon the configuration of elements that on their own do not convey a significant expression signal.     95Figure 4.5  Expression aftereffects using thresholded faces  Top graph shows the mean frequency of responses that test stimuli looked ?Angry? after adapting to either Afraid or Angry thresholded faces. Bottom graphs show the magnitude of the aftereffect, expressed as a difference scores, calculated from response scores as in Figure 4.4, with error bars showing one standard deviation and asterisks denoting significant aftereffects. Data on the left are for the Intact-Face condition, data on the right for the Scrambled-Face condition.      964.3.3 Experiment 3 In Experiment 3 we used highly schematic faces made up of line and curve segments and circles. These geometric shapes, which form the individual features of the face, are devoid of expression on their own, but when placed within a facial configuration they combine to depict facial expressions (Figure 4.1C). Thus, in a manner similar to Experiment 2, we compared expression aftereffects in an Intact-Schematic-Face condition to those in a Scrambled-Schematic-Face condition. If facial configuration does not contribute to the expression aftereffect (i.e. the aftereffects are based solely on the representation of individual facial features) then we should not find an expression aftereffect in either condition. If the aftereffects reflect local adaptation to image elements such as tilt and curvature we should find significant and similar aftereffects in both conditions (contrary to the results of Experiment 1). Finally, if adaptation to facial configuration but not local elements contributes to expression aftereffects generated with these schematic images, then we should see significant aftereffects with the intact schematic faces but not with the scrambled ones.   This experiment also had two conditions. In the Intact-Schematic-Face condition, line elements were combined in a face-like arrangement in both the adapting and test stimuli. In the Scrambled-Schematic-Face condition, they were arranged in a non-face-like arrangement for both adapting and test stimuli.   There was no significant main effect of adapting stimulus (F(1,33) = 1.58, n.s.). However, there was a significant interaction between face-condition and adapting stimulus (F(1, 33)= 4.39, p < 0.044). The linear contrasts showed a significant aftereffect in the Intact-Schematic-Face condition (t= 2.372, p= 0.024), but not for the Scrambled-Schematic-Face condition (t = 0.59, n.s.) (Figure 4.6).     97Figure 4.6  Expression aftereffects using schematic faces    Top graph shows the mean frequency of responses that test stimuli looked ?Angry? after adapting to either Happy or Angry schematic faces. Bottom graphs show the magnitude of the aftereffect, expressed as a difference scores, calculated from response scores as in Figure 4.4, with error bars showing one standard deviation and asterisks denoting significant aftereffects. Data on the left are for the Intact-Schematic-Face condition, data on the right for the Scrambled-Schematic-Face condition.      98  The findings in the Scrambled-Schematic-Face condition of Experiment 3 reinforce the conclusion from Experiment 1: that facial expression aftereffects do not originate from local adaptation to image elements like orientation and curvature. The difference between the Scrambled-Face condition in Experiment 2 and the Scrambled-Schematic-Face condition in Experiment 3 are that the features of the highly schematic faces of Experiment 3 do not convey expression when viewed in isolation, being merely tilted lines and curves, whereas expression is still visible in the individual features of the faces in Experiment 2. Thus when expression information is removed from the face parts, adaptation to these face parts does not generate an aftereffect, further reinforcing the conclusion from Experiment 1, that local adaptation to image elements like line orientation and curvature do not mediate the face expression aftereffect. However, when these curved and oriented segments are assembled into a face-like configuration, an expression aftereffect can be found even with these very basic schematic faces. Therefore, while Experiment 2 shows that features alone can generate some of the expression aftereffect, Experiment 3 demonstrates that facial configuration can generate an expression aftereffect. Both experiments show that a perceivable expression is the necessary component to generate an aftereffect. Again this supports the finding reported in Experiment 1 where aftereffects were only seen when a consistent expression was perceived in the adapting face.    4.4  Discussion Previous studies have found aftereffects following adaptation to various aspects of faces, such as identity, race, gender, and expression (Fox and Barton, 2007; Webster, Kaping et al., 2004). This report focused on determining the basis of the facial expression aftereffect. We first considered the possibility that it was derived from local adaptation to image elements within the facial image, such as orientation, curvature and shape. By creating quartered faces with inconsistent expressions, we examined whether this local adaptation could generate an aftereffect in the   99absence of a coherent expressive signal. The result of Experiment 1 was clear: There were no aftereffects when there was no coherent expression. This suggests that neurons coding for local image elements are not the source of the expression aftereffects seen in this adaptation paradigm.  This conclusion was also supported by the lack of aftereffects in the scrambled-schematic-face condition of Experiment 3, once again implying that tilt and curvature aftereffects were not the source of facial expression aftereffects.   One possible reason for a greater aftereffect for images with consistent rather than mixed expressions may be that, not only do consistent images generate a coherent representation of expression, but these coherent representations also have a verbal label or name for the expression that images with mixed expressions lack, and that this may reinforce perception or recall during the task. However, our experiments were designed to minimize the use of a verbal label in all conditions, in that the task required subjects to choose one of two faces on a choice display, a task that does not require any verbal label. Also, in our previous report (Fox and Barton, 2007), the use of verbal labels alone as adapting stimuli did not generate any aftereffect in images of facial expression, indicating that representations at a verbal level do not contribute significantly to face expression aftereffects.   Experiment 2 examined whether local features alone could generate an aftereffect when facial expression was consistent, or if a whole-face configuration was necessary. This was motivated by a large body of literature showing that many aspects of face perception depend on ?holistic? processing (Tanaka, Kay et al., 1998; de Heering, Houthuys et al., 2006; Goffaux and Rossion, 2006; Singer and Sheinberg, 2006; Yovel and Duchaine, 2006). The results showed that it was possible to generate expression aftereffects without a whole-face configuration. However, this result does not imply that facial configuration does not contribute to expression aftereffects: Experiment 3 used schematic faces comprised of simple line elements, that do not resemble a facial feature, or convey a facial expression, when viewed in isolation. This experiment showed   100that these expression-neutral elements generate an expression aftereffect only when arranged into a normal facial configuration. Thus the results of Experiment 2 and 3 suggest the existence of both featural and configural aspects to the neural representations of expression.    Our prior report suggested that identity-dependent representations of expression might exist, because adaptation effects were largest when images of the same person were used as both adapting and test stimuli (Fox and Barton, 2007). We have demonstrated in the current study that identity-dependent expression aftereffects are not the result of local adaptation to image elements of the faces. The existence of an identity-dependent neural representations of expression is also consistent with a number of previous studies (Schweinberger and Soukup, 1998; Schweinberger, Burton et al., 1999; Baudouin, Martin et al., 2002; Ganel, Goshen-Gottstein et al., 2004). For example, a recent study of neural responses in monkeys demonstrated that the responses of cells in the amygdala to facial expression depended upon the identity of the monkey demonstrating that expression (Kuraoka and Nakamura, 2006). In addition, responses to emotion and responses to facial identity were coded in the same phase of the response, consistent with an integrated processing of the two types of information (Kuraoka and Nakamura, 2006). In humans, another potential site for integration of identity and expression information may be the posterior superior temporal sulcus, which showed hemodynamic adaptation to both identity and expression in an event-related fMRI paradigm (Winston, Henson et al., 2004). The superior temporal sulcus and the amygdala are relatively high-level regions in the cortical hierarchy of visual processing (Felleman and Van Essen, 1991) and would be appropriate candidates for the hypothesized locus of neural populations generating the behavioural aftereffects in our experiment.    In contrast, local ?shape dimensions? such as curvature and particularly orientation (Werner and Chalupa, 2004) are represented in the lowest levels of the visual hierarchy, such as the striate cortex (area V1) (Hubel and Wiesel, 1962). While aftereffects for orientation do occur and are known to be based on changes in neural dynamics in V1 (Dragoi, Sharma et al., 2000),   101our experiment suggests that these do not account for the identity-dependent component of the expression aftereffect. This may partly reflect the free-viewing nature of our adaptation period. Aftereffects in strictly retinotopically organized structures like the retina and V1 are highly localized and depend upon controlled fixation during the adaptation period. The absence of a requirement for maintained fixation may account for the lack of adaptation in the quartered-faces with mixed expressions in Experiment 1 and the scrambled line elements of Experiment 3. Allowing our subjects to freely move their eyes during the adaptation period may have allowed us to limit adaptation to representations in higher-level cortical regions for object recognition, many of which are invariant to object location.     Experiments 2 and 3 showed that, on the one hand, the whole-face configuration is able to generate expression aftereffects, and yet, on the other, the configuration is not necessary if the features themselves are sufficiently expressive. This suggests that both holistic facial representations and facial features can generate expression aftereffects. One possible interpretation of these findings is that there are separate feature-based and holistic neural representations of facial expression, both of which can be adapted by their preferred stimuli. However, an alternative interpretation is that facial features alone are able to partially activate holistic representations of facial expressions, and that exposure to a feature collage will result in partial adaptation of these holistic representations. Indeed, neurophysiologic data show that face-responsive cells in monkey temporal cortex do respond partially to isolated features (Perrett, Rolls et al., 1982). If so, one would expect a reduced aftereffect with scrambled faces than with whole faces. The results of Experiment 2 do suggest that the magnitude of the aftereffect for scrambled faces is less than that for whole faces, even though the interaction did not reach significance. Therefore, we consider that this possibility cannot be excluded. Regardless, the results argue against a single representation of expression in a holistic code that can only be accessed by holistic depictions. Rather, either holistic representations can be flexibly activated,   102strongly by whole faces or partially by fragmentary representations, or the system itself is flexible, in that it may contain multiple representations of expression, both feature-based and configurational, consistent with some recent computational models of face processing (Wallraven, Schwaninger et al., 2005).    To conclude, our findings provide further data on the nature of the identity-dependent representation of expression in the human cortex. Our first experiment shows that these expression aftereffects cannot be attributed to well-known aftereffects for low-level shape dimensions such as orientation and curvature. Our data also show that aftereffects can be elicited through adaptation to an expressive face in a normal configuration, or to expressive local features alone, suggesting a degree of flexibility in these representations.           103  4.5  References ADAMS WJ and MAMASSIAN P. Common mechanisms for 2d tilt and 3d slant after-effects. Vision Res, 42: 2563-8, 2002. BARTON JJ, RADCLIFFE N, CHERKASOVA MV, EDELMAN J, and INTRILIGATOR JM. Information processing during face recognition: The effects of familiarity, inversion, and morphing on scanning fixations. Perception, 35: 1089-105, 2006. BAUDOUIN JY, MARTIN F, TIBERGHIEN G, VERLUT I, and FRANCK N. Selective attention to facial emotion and identity in schizophrenia. Neuropsychologia, 40: 503-11, 2002. BLAKEMORE C and SUTTON P. Size adaptation: A new aftereffect. Science, 166: 245-7, 1969. BUTLER S, GILCHRIST ID, BURT DM, PERRETT DI, JONES E, and HARVEY M. Are the perceptual biases found in chimeric face processing reflected in eye-movement patterns? Neuropsychologia, 43: 52-9, 2005. CLIFFORD CWG and RHODES G. Fitting the mind to the world : Adaptation and after-effects in high-level vision. Oxford: Oxford University Press, 2005. DE HEERING A, HOUTHUYS S, and ROSSION B. Holistic face processing is mature at 4 years of age: Evidence from the composite face effect. J Exp Child Psychol, 2006. DRAGOI V, SHARMA J, and SUR M. Adaptation-induced plasticity of orientation tuning in adult visual cortex. Neuron, 28: 287-98, 2000. FELLEMAN DJ and VAN ESSEN DC. Distributed hierarchical processing in the primate cerebral cortex. Cereb Cortex, 1: 1-47, 1991. FOX CJ and BARTON JJ. What is adapted in face adaptation? The neural representations of expression in the human visual system. Brain Res, 1127: 80-9, 2007. GANEL T, GOSHEN-GOTTSTEIN Y, and GANEL T. Effects of familiarity on the perceptual integrality of the identity and expression of faces: The parallel-route hypothesis revisited. J Exp Psychol Hum Percept Perform, 30: 583-97, 2004. GIBSON JJ and RANDER M. Adaptation, aftereffect and contrast in the perception of tilted lines. Journal of Experimental Psychology, 20: 453-467, 1937. GOFFAUX V and ROSSION B. Faces are "Spatial"--holistic face perception is supported by low spatial frequencies. J Exp Psychol Hum Percept Perform, 32: 1023-39, 2006. HUBEL DH and WIESEL TN. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J Physiol, 160: 106-54, 1962. HUBER DE and O'REILLY RC. Persistence and accommodation in short-term priming and other perceptual paradigms: Temporal segregation through synaptic depression. Cognitive Science, 27: 403-430, 2003. KURAOKA K and NAKAMURA K. Impacts of facial identity and type of emotion on responses of amygdala neurons. Neuroreport, 17: 9-12, 2006. LEOPOLD DA, O'TOOLE AJ, VETTER T, and BLANZ V. Prototype-referenced shape encoding revealed by high-level aftereffects.[see comment]. Nature Neuroscience, 4: 89-94, 2001. LUNDQVIST D and LITTON JE. The averaged karolinksa directed emotional faces - akdef. 1998. PERRETT DI, ROLLS ET, and CAAN W. Visual neurones responsive to faces in the monkey temporal cortex. Exp Brain Res, 47: 329-42, 1982. REGAN D and HAMSTRA SJ. Shape-discrimination and the judgment of perfect symmetry - dissociation of shape from size. Vision Research, 32: 1845-1864, 1992. RHODES G, BRENNAN S, and CAREY S. Identification and ratings of caricatures - implications for mental representations of faces. Cognitive Psychology, 19: 473-497, 1987. SCHWEINBERGER SR, BURTON AM, and KELLY SW. Asymmetric dependencies in perceiving identity and emotion: Experiments with morphed faces. Percept Psychophys, 61: 1102-15, 1999.   104SCHWEINBERGER SR and SOUKUP GR. Asymmetric relationships among perceptions of facial identity, emotion, and facial speech. J Exp Psychol Hum Percept Perform, 24: 1748-65, 1998. SHEPHERD J, DAVIES G, and ELLIS H. Studies of cue saliency. London: Academic Press, 1981. SINGER JM and SHEINBERG DL. Holistic processing unites face parts across time. Vision Res, 46: 1838-47, 2006. SUZUKI S. Attentional selection of overlapped shapes: A study using brief shape aftereffects. Vision Research, 43: 549-561, 2003. SUZUKI S and CAVANAGH P. A shape-contrast effect for briefly presented stimuli. Journal of Experimental Psychology-Human Perception and Performance, 24: 1315-1341, 1998. TANAKA JW, KAY JB, GRINNELL E, STANSFIELD B, and SZECHTER L. Face recognition in young children: When the whole is greater than the sum of its parts. Visual Cognition, 1998. VALENTINE T. A unified account of the effects of distinctiveness, inversion, and race in face recognition. Quarterly Journal of Experimental Psychology Section a-Human Experimental Psychology, 43: 161-204, 1991. WALLRAVEN C, SCHWANINGER A, and BULTHOFF HH. Learning from humans: Computational modeling of face recognition. Network, 16: 401-18, 2005. WEBSTER MA, KAPING D, MIZOKAMI Y, and DUHAMEL P. Adaptation to natural facial categories. Nature, 428: 557-61, 2004. WERNER JS and CHALUPA LM. The visual neurosciences. Cambridge, Mass.: MIT Press, 2004. WILSON HR and DIACONESCU A. Learning alters local face space geometry. Vision Res, 46: 4143-51, 2006. WINSTON JS, HENSON RN, FINE-GOULDEN MR, and DOLAN RJ. Fmri-adaptation reveals dissociable neural representations of identity and expression in face perception. J Neurophysiol, 92: 1830-9, 2004. YOVEL G and DUCHAINE B. Specialized face perception mechanisms extract both part and spacing information: Evidence from developmental prosopagnosia. J Cogn Neurosci, 18: 580-93, 2006. ZHAO L and CHUBB C. The size-tuning of the face-distortion after-effect. Vision Res, 41: 2979-94, 2001.                1055   THE CORRELATES OF SUBJECTIVE PERCEPTION OF IDENTITY AND      EXPRESSION IN THE FACE NETWORK: AN FMRI ADAPTATION STUDY4    5.1  Introduction Face perception involves multiple cortical regions (Gobbini and Haxby, 2007; Haxby, Hoffman et al., 2000; Ishai, Schmidt et al., 2005; Rossion, Caldara et al., 2003). Current models divide these into a core system which is predominantly involved in the processing of facial stimuli, and an extended system which contributes to, but is not solely involved in, face perception (Haxby, Hoffman et al., 2000). The core system is comprised of three cortical regions which consistently show increased activity to faces over objects in functional magnetic resonance imaging (fMRI) studies: the occipital face area (OFA), located on the inferior occipital gyrus; the fusiform face area (FFA), located on the lateral fusiform gyrus; and a face-selective region in the posterior superior temporal sulcus (pSTS) (Haxby, Hoffman et al., 2000; Kanwisher, McDermott et al., 1997; Ishai, Schmidt et al., 2005). Functionally, the OFA has been modeled as an ?entry point? region involved in the early perception of facial features or structure, with the FFA and pSTS as two subsequent independent and complementary modules, one (FFA) involved in the perception of temporally invariant aspects of faces (i.e.- facial identity), the other (pSTS) involved in the perception of dynamic aspects of the face (i.e.? facial expression) (Haxby, Hoffman et al., 2000). Regions of the extended system (i.e.- areas involved in the retrieval of semantic information, emotional connotations, etc.) are then activated by the output of this core system (Haxby, Hoffman et al., 2000).   However, a number of aspects of this model have been debated. First, the role of the OFA as an entry point for the core system of face perception has been challenged by the finding of an FFA in a prosopagnosic patient who lacks an ipsilateral OFA, suggesting that FFA activation is                                                  4 A version of this chapter has been submitted for publication. Fox C.J., Moon S.Y., Iaria G., and Barton J.J.S. The correlates of subjective perception of identity and expression in the face network: an fMRI adaptation study.   106not dependent on input from the OFA (Rossion, Caldara et al., 2003). On the other hand, a role for the OFA in the early perception of faces has received support from studies using fMRI adaptation (Rotshtein, Henson et al., 2005), which is thought to reflect reduced neural responses with repeated presentations of a stimulus (Grill-Spector, Henson et al., 2006). By varying certain aspects of a stimulus while holding others constant, one can use adaptation to determine which aspects of that stimulus a cortical region specifically encodes. In the OFA, adaptation was observed when the second face was identical to the first, but any structural change in the second face resulted in a release from adaptation (Rotshtein, Henson et al., 2005), regardless of whether this structural change crossed a categorical boundary between identities or not (Rotshtein, Henson et al., 2005). The authors conclude that the OFA is sensitive to any structural change in a face (i.e.- early perception of facial structure) but is not sensitive to facial identity (Rotshtein, Henson et al., 2005). This study did not examine the effects of expression changes, however, although current models suggest that the OFA also provides input to the pSTS. Thus the first aim of the present study was to determine whether the findings of Rotshtein et al (2005) regarding OFA adaptation can be replicated for facial identity and extended to facial expression.    Second, the proposal that the processing of facial identity and expression are independent of each other has been questioned (Calder and Young, 2005; Fox and Barton, 2007; Fox, Oruc et al., 2008; Ganel, Goshen-Gottstein et al., 2004; Ganel, Valyear et al., 2005; Gorno-Tempini, Pradelli et al., 2001; Ishai, Schmidt et al., 2005; Kaufmann and Schweinberger, 2004; Palermo and Rhodes, 2007; Vuilleumier, Armony et al., 2001; Winston, Henson et al., 2004). Calder and Young (2005) suggest a relative rather than absolute segregation of identity and expression processing, with some regions involved in the perception of both. Behavioral studies of aftereffects (Fox and Barton, 2007; Fox, Oruc et al., 2008), interference effects (Ganel, Goshen-Gottstein et al., 2004) and recognition (Kaufmann and Schweinberger, 2004) all show functional interactions between facial identity and expression. Likewise, fMRI studies provide evidence for   107functional overlap: the pSTS shows significant fMRI adaptation not only to expression but also to identity (Winston, Henson et al., 2004). Despite its purported focus on identity, the FFA shows more activity for expressive than neutral faces (Ishai, Schmidt et al., 2005; Vuilleumier, Armony et al., 2001), and attending to facial expression increases activity not only in the pSTS as expected (Narumoto, Okada et al., 2001), but also in the FFA (Ganel, Valyear et al., 2005; Gorno-Tempini, Pradelli et al., 2001). As discussed for the OFA, the study of categorical effects can be of interest, in that this can establish sensitivity to specific representations beyond just the physical properties of the stimulus. The same study that showed a lack of categorical effects for identity in the OFA reported such effects in the FFA (Rotshtein, Henson et al., 2005), but since categorical effects for expression were not studied, it is unclear if this effect in the FFA is specific for facial identity. The second aim of the present study is thus to examine categorical effects for both facial identity and expression within both the FFA and pSTS to determine if the pattern of effects suggests a complementary selectivity or functional overlap between these regions.    5.2  Methods  5.2.1 Participants Sixteen right-handed healthy participants (8 females; Mean age + SD:  24.2 + 3.4 years) with normal or corrected-to-normal vision and no history of neurological disorders participated. Informed consent was obtained and the protocol approved by the institutional review boards of the University of British Columbia and Vancouver General Hospital, in accordance with The Code of Ethics of the World Medical Association, Declaration of Helsinki (Rickham, 1964). One participant was excluded from all analyses due to excessive head motion (>2?) during the fMRI scanning session.   1085.2.2 Stimuli Stimuli used within the experimental scans were selected from the Karolinska Database of Emotional Faces (Lundqvist and Litton, 1998). Images of eight different identities (4 female), displaying four different expressions (angry, afraid, disgusted, happy) were chosen, for a total of 32 images. We used Adobe Photoshop CS2 9.0.2 (www.adobe.com) to remove background, hair and neck, while preserving facial features and external jaw contour. Distinguishing marks, such as moles, were removed using the Spot Healing Brush Tool. Images were then cropped to ensure that all faces were centrally located in the image frame, and resized to a standard width of 400 pixels.    In order to test the perception of structural changes that do or do not cross categorical boundaries of facial identity and facial expression, morphed faces were used (Rotshtein, Henson et al., 2005). A morph matrix was created by selecting 4 images (2 different expressions for one identity and the same 2 different expressions for another identity), as the corners of the matrix. We then used Abrosoft Fantamorph 3.0 (www.fantamorph.com) to generate 1/3:2/3 morphs to fill in a two-dimensional 4X4 matrix of 16 images, with the two dimensions representing identity and expression (Figure 5.1). Sixteen unique matrices were created for each gender, for a total of 32 morph matrices.   Images were then paired to create the five experimental conditions (Figure 5.1). One trial for each condition was selected from each morph matrix for a total of 32 trials per condition, and 160 unique trials in total. All five trials selected from the same morph matrix shared the same first image (2/3ID1:1/3ID2 and 2/3EX1:1/3EX2, where ID = identity and EX = expression). In the identical condition the second image was identical to the first. In the similar-identity category condition the second image was a 33% morph difference that did not cross the category boundary for identity (3/3ID1:0/3ID2 and 2/3EX1:1/3EX2). (This identity boundary should be approximately located around a morph that contains equal amounts of the two identities.)    109Figure 5.1   Example morph matrix and adaptation face pairs   An example of a 4X4 morph matrix used to create the five experimental conditions. Thirty-two unique matrices were created. Experimental conditions consisted of a pair of face images. The first image in each experimental condition, and the second image in the identical condition, was always the same. The second images in each of the other four conditions are labeled. Pairs in the different-identity condition cross a categorical boundary of identity (blue line), while pairs in the different-expression condition cross a categorical boundary of expression (red line). Subsequent figures use the same coloring to represent the five experimental conditions; gray = identical, pale blue = similar-identity, dark blue = different-identity, pale red = similar-expression, dark red = different-expression. For each morph matrix only five images were used to create the different experimental conditions (as labeled).      110In the different-identity category condition the second image was a 33% morph step across an identity boundary (1/3ID1:2/3ID2 and 2/3EXP1:1/3EX2), so that the image now contained more of identity 2 than identity 1. In the similar-expression category condition the second image was a 33% morph step within an expression boundary (2/3ID1:1/3ID2 and 3/3EX1:0/3EX2). In the different-expression category condition the second image was a 33% morph step across an expression boundary (2/3ID1:1/3ID2 and 1/3EX1:2/3EX2), so that it now contained more of expression 2 than 1. With this strategy, any change in the second stimulus occurs along either the expression or identity axis, but not both simultaneously, and second, both similar and different conditions have the same size of morph step, or structural change, so that the only difference between similar and different conditions is whether the change crossed a categorical boundary.    To establish familiarity with the chosen identities, and the development of identity categories, participants were given one uncropped neutral image for each of the 8 identities and a name for each one, and were asked to learn to recognize each person during the few days before testing. Immediately prior to the fMRI scan, participants performed a familiarity test on a 17? widescreen Compaq nx9600 notebook. Fourteen uncropped images (2 versions of angry, afraid, happy, disgusted, surprised, sad, and neutral images) of each of the 8 identities (112 images total) were presented in a random order using SuperLab Pro 2.0.4 (www.cedrus.com). Participants were given unlimited time to select the appropriate name with a key press. Immediate feedback was given in the form of the correct name presented on the screen for 500ms, after which the next image was presented. If >95% accuracy was achieved, participants began the fMRI session, otherwise the familiarity test was repeated until >95% accuracy was achieved.      1115.2.3 FMRI data acquisition  All scans were acquired in a 3.0 Tesla Philips scanner. Stimuli were presented using Presentation 9.81 software and rear-projected onto a mirror mounted on the head coil. Whole brain anatomical scans were acquired using a T1-weighted echoplanar imaging (EPI) sequence, consisting of 170 axial slices of 1mm thickness (1mm gap) with an in-plane resolution of 1mm X 1mm (FOV=256). T2-weighted functional scans (TR=2s; TE=30ms) were acquired using an interleaved ascending EPI sequence, consisting of 36 axial slices of 3mm thickness (1mm gap) with an in-plane resolution of 1.875mm X 1.875mm (FOV=240). The functional scans consisted of 224 functional volumes, with the first volume of each functional scan discarded to allow for scanner equilibration.  The first functional scan, a functional localizer, was used to identify face-selective regions-of-interest. During the localizer, participants viewed static photographs of non-living objects (e.g. - television, basketball) and faces (neutral and expressive) presented in separate blocks (Kanwisher, McDermott et al., 1997; Saxe, Brett et al., 2006). Participants performed an irrelevant ?one-back task?, pressing a button if an image was identical to the previous one. The localizer began and ended with a fixation block showing a cross in the centre of an otherwise blank screen. Additional fixation blocks were alternated with image blocks, with all blocks lasting 12 seconds. Six blocks of each image category (object, neutral face, expressive face) were presented in a counterbalanced order. Each image block consisted of 15 images (12 novel and 3 repeated), all sized to a standard width of 400 pixels and presented at screen center for 500ms, with an inter-stimulus-interval of 300ms. Following the localizer scan, participants underwent two experimental scans. Experimental scans began with 6 fixation trials, followed by 160 experimental trials (32 for each of 5 conditions) and 50 fixation trials in one of five random orders, and ended with 6 fixation trials. During each trial the first image was presented 150 pixels to the left of center for 500ms   112followed by a 300ms inter-stimulus-interval. The second image was then presented 150 pixels to the right of center for 500ms followed by a 700ms inter-trial-interval. Left/right image presentation was used to ensure participants made perceptual decisions on the correct image pairs (i.e. always responding after the right-sided image). Face images were replaced with a fixation cross during fixation trials. Jittering was achieved through the randomized presentation of fixation trials throughout the scan (Serences, 2004; Dale, 1999).  We had subjects perform two experimental runs because of the potential for the effects to be modulated by task. Most fMRI adaptation studies have required participants to perform irrelevant tasks during functional scans, while maintaining attention on the face (Rotshtein, Henson et al., 2005; Winston, Henson et al., 2004). Attention is critical since others have shown that when a face is not attended (because of a distracter face, for example) adaptation is no longer observed (Ishai, Pessoa et al., 2004). Other studies have shown task-modulation of the BOLD signal in several face areas: OFA- greater activity during expression than gender identification tasks (Gorno-Tempini, Pradelli et al., 2001); FFA ? greater activity when attending facial identity than facial contour in a delayed match to sample task (Narumoto, Okada et al., 2001), or when explicitly judging expression as compared to identity changes (Ganel, Valyear et al., 2005); pSTS -  greater activity when attending facial expression than facial identity in a delayed match to sample task (Narumoto, Okada et al., 2001). In our experiment we decided to ask subjects to engage directly in tasks relevant to the dimensions we were exploring. Thus, in one experimental run participants made same/different judgments about the facial identity of image pairs, while in the other they made same/different judgments about the facial expression of image pairs. The order of the two experimental runs was determined randomly for each subject. The same stimuli were used during both runs but in a different random order, so that task effects were not confounded by stimulus differences.    1135.2.4 Analysis  5.2.4.1 Functional localization All MRI data were analyzed using BrainVoyager QX Version 1.8 (www.brainvoyager.com). Anatomical scans were not preprocessed. Preprocessing of functional scans consisted of corrections for slice scan time acquisition, head motion (trilinear interpolation), and temporal filtering with a high pass filter in order to remove frequencies less than 3 cycles/time course. For each participant, functional scans were individually co-registered to their respective anatomical scan, using the first retained functional volume to generate the co-registration matrix. The localizer time course was analyzed using a single subject GLM, with object (O), neutral (NF) and expressive (EF) faces as predictors. Analysis of NF+EF>2*O was overlaid on the whole brain and significance was set at a False Discovery Rate of q<0.05, corrected for multiple comparisons. Within each participant, we attempted to define five regions-of-interest. Contiguous clusters of >10 voxels located on: i) the lateral surface of the inferior occipital gyrus were designated as the OFA; ii) the lateral temporal portion of the fusiform gyrus were designated as the FFA; iii) the posterior segment of the superior temporal sulcus were designated as the pSTS. In addition, we assessed responses in two other areas beyond the classic core system: iv) the middle segment of the superior temporal sulcus were designated as the mSTS; v) the hemispheric midline, anterior to the occipitotemporal fissure, were designated as the precuneus. The OFA, FFA, pSTS and mSTS were only localized in the right hemisphere due to strong evidence for right hemisphere dominance from fMRI (Kanwisher, McDermott et al., 1997) and from patients with unilateral right-hemisphere damage who exhibit perceptual impairments for facial identity (Barton, 2003) or facial expression (Adolphs, Damasio et al., 1996). As the precuneus was located at the hemispheric midline, right and left hemisphere regions were indistinguishable.    1145.2.4.2 Adaptation effects We first verified that our stimuli generated behavioral responses consistent with the proposal that the two stimuli in the different conditions were located across the appropriate category boundary while the two stimuli in the similar conditions were located on the same side of the appropriate category boundary. If so, subjects should respond significantly more frequently that the two faces differ in terms of identity in the different-identity category condition than in the different-expression category, similar-expression category, or similar-identity category conditions. Likewise, they should respond more often that the two faces differ in terms of expression in the different-expression category condition than in the other three conditions.  For each subject we calculated the proportion of different responses for each experimental condition and performed a general linear model (GLM) with condition (identical, similar-identity category, different-identity category, similar-expression category, different-expression category) and task (identity run, expression run) as fixed factors, subject as a random factor, and proportion of different responses as the dependent measure. Linear contrasts were used to examine any significant main or interaction effects. An interaction, with significantly more different responses for the different-identity condition during the identity run and for the different-expression condition during the expression run would indicate categorical perception of this stimulus set. Experimental MRI scans were analyzed using a deconvolution analysis that accounts for non-linear summation of the blood oxygen level dependent (BOLD) response in rapid event-related designs. The deconvolution analysis samples BOLD activity at trial onset (time = 0sec) and again 9 more times in 2sec intervals, resulting in an unbiased model of the hemodynamic response (HDR). HDRs were estimated independently for each experimental condition. From the estimated HDRs we can compare adaptation across different experimental conditions. In discussing adaptation of the BOLD signal, Grill-Spector et al. (2006) suggest three possible means through which reduced activity could be achieved; a general neural fatigue,   115neural sharpening of the response to include only highly selective neurons, or a facilitation of the response resulting in faster activation and deactivation (Grill-Spector, Henson et al., 2006). Translating these postulated changes in the BOLD signal to changes in the shape of the HDR one could expect adaptation to result in either a reduction in the HDR peak (fatigue) or a narrowing of the HDR (facilitation). In order to account for both options we chose to examine the area-under-the-curve (AUC) of the HDR. The hemodynamic response normally begins its ascent after 2sec post-stimulus and has returned to baseline by 12sec post-stimulus (Serences, 2004). With a 2sec TR, this leaves 4 consecutive time points (4sec, 6sec, 8sec, 10sec) representing the full positive component of the HDR, although there is some variability in this timing (Yovel and Kanwisher, 2005). We determined the AUC of each HDR by finding the 4 consecutive time points with the maximal summed percent signal change (%SC) and divided this activity by 6sec, resulting in an AUC value in units of %signal change/sec.  To analyze the effect of condition in the adaptation data, we performed two types of analysis. Since fMRI results during a face/object detection task showed that the participant?s perception drive the measured effects more than the stimulus categories (Grill-Spector, Knouf et al., 2004), it is possible that ?categorical? effects may be driven less by the morph characteristics of the image pairs and more by whether the subject perceives the image pairs as same or different. Thus, our first analysis was based on our a priori stimulus categories, reflecting whether image pairs lay on the same side or different sides of the 50:50 morph boundary. Our second analysis reclassified trials based on whether they were perceived as same or different identities during the experimental run requiring identity judgments, and again as to whether they were perceived as same or different expressions during the run requiring expression judgments. For both types of analysis, the responses during all trials in the identical condition were averaged to give the baseline estimate of full adaptation. In the first ?stimulus-based? analysis, the other conditions were labeled as similar-identity category, different-identity category, similar-  116expression category, and different-expression category as above. In the second ?perception-based? analysis, the conditions were labeled as similar-identity perception, different-identity perception, similar-expression perception, and different-expression perception. A GLM was then performed, with condition (identical, similar-identity, different-identity, similar-expression, different-expression), task (identity run, expression run) and analysis (stimulus-based, perception-based) as fixed factors, subject as a random factor and AUC as the dependent measure. Linear contrasts were used to examine any significant main or interaction effects. Significance on all statistical tests was set at ? <0.05. For graphic purposes, and to account for between-subject variability in baseline hemodynamic activity, the AUC of the identical condition (representing maximal adaptation) was subtracted from the AUC of other experimental conditions. This resulted in a ?release from adaptation? variable, again measured in units of % signal change/sec.    5.3  Results  5.3.1 Behavioral data The GLM of responses recorded during the scanning session revealed a main effect of condition [F(1,14)=36.36; p<0.001], which was modified by an interaction between condition and task [F(1,14)=30.31; p<0.001]. This was due to a significantly higher frequency of different responses for the different-identity category condition (Mean frequency of different responses + SEM = 0.37+0.05) than for any other condition (p<0.01, all tests) during the identity run, and a significantly higher frequency of different responses for the different-expression category condition (mean frequency of different responses + SEM = 0.50+0.03) than for any other condition (p<0.001, all tests) during the expression run (Figure 5.2). Thus the behavioral data are consistent with the classification of our stimuli regarding category boundaries for both identity    117Figure 5.2  Behavioral responses recorded during the fMRI session  Results of behavioral responses recorded during the fMRI session (Mean p of different response + SEM; gray = identical, pale blue = similar-identity, dark blue = different-identity, pale red = similar-expression, dark red = different-expression). Categorical effects can be seen during both the identity and expression runs (indicated with a  ? ), with significantly more different responses for stimulus pairs that cross the task-specific category boundary than those that do not. Significantly more different responses were observed in all conditions (indicated with an *) as compared to the identical condition, validating this condition as a baseline measure of maximal adaptation.           118and expression. In addition, the proportion of different responses for the identical condition was lower than any other condition during both runs (identity run ? 0.02+0.01; p<0.05, all tests. expression run ? 0.03+0.01; p<0.01, all tests), making the identical condition an appropriate baseline measure of maximal adaptation for the subsequent fMRI analyses.  5.3.2 Adaptation ? Core system We identified the OFA in 13 of 15 participants (Table 5.1; Figure 5.3A). The GLM revealed a significant main effect of condition [F(4,48)=3.32; p<0.05], due to a significantly smaller AUC in the identical condition (Mean AUC + SEM; 1.69+0.11) than in any other condition (p<0.05, all tests). The remaining four experimental conditions did not differ from each other (p>0.20, all tests). Thus, a significant release from adaptation occurs in the OFA with any structural change in the face, whether along an identity or expression axis, and regardless of whether the structural change crosses a categorical boundary for identity or expression (Figure 5.3B and C). No other main effects or interactions were significant, indicating that this effect in the OFA is independent of task demands and independent of whether the trials are classified by stimulus characteristics or by the perceptual experience of the individual subject.   We identified the FFA in all 15 participants (Table 5.1; Figure 5.4A). The GLM revealed a significant main effect of condition [F(4,56)=10.67; p<0.001] modified by a significant interaction between condition and analysis [F(4,56)=3.41; p<0.05]. Examining the stimulus-based analysis alone showed a significantly smaller AUC in the identical condition (1.32+0.10) than in any other condition (p<0.05, all tests), but the remaining four experimental conditions did not differ from each other (p>0.40, all tests; Figure 5.4B). In the perception-based analysis there was again a significantly smaller AUC in the identical condition (1.28+0.09) than in any other condition (p<0.001, all tests). However, categorical effects were also apparent in the perception-based analysis, with a significantly larger AUC in the different-expression perception   119Table 5.1  Average results from the functional localizer  Region-of-Interest  # of subjects(/15) t-value of Peak Voxel # of Voxels (FDR; q<0.05) X   Y  Z  OFA 13 6.85 + 0.74  510 + 163  38 + 2  -78 + 3 -12 + 2FFA 15 7.54 + 0.62  794 + 217  37 + 1  -47 + 2 -19 + 1pSTS 15 6.05 + 0.36  382 + 68  52 + 1  -50 + 2 8 + 1 mSTS 13 4.49 + 0.19  78 + 17  52 + 2  -8 + 2  -10 + 2precuneus 10 5.20 + 0.30  665 + 240  1 + 1  -61 + 2 27 + 3  The t-value of the peak voxel and number of voxels are reported from regions-of-interest localized in native non-standardized space, and all analyses were performed on these regions-of-interest. Mean Talairach coordinates are reported for comparison with other studies only.             120Figure 5.3  Adaptation effects in the right OFA   (A) Examples of the right occipital face area in three subjects (coronal slices). (B) Results of the stimulus-based (B) and perceptual-bases analyses (C) in the right OFA, demonstrating sensitivity to structural changes (indicated with an *), as compared to the identical condition. No categorical effects of identity or expression are observed, suggesting the OFA does not participate in the perception of these facial characteristics. Bars in this and following figures represent mean release from adaptation values + SEM.              121 Figure 5.4  Adaptation effects in the right FFA    (A) Examples of the right fusiform face area in three subjects (coronal slices). (B) The stimulus based analysis reveals a significant release from adaptation with any structural change (indicated with an *), as compared to the identical condition. (C) The perception-based analysis reveals a significant release from adaptation for perceived expression changes (indicated with an  ? ) with a trend to the same effect for perceived identity changes (indicated with an  # ), evidence for a functional overlap of identity and expression processing within the FFA.          122condition (1.75+0.11) than in the similar-expression perception condition (1.55+0.11; p<0.01) and a trend to a larger AUC in the different-identity perception condition (1.69+0.12) than the similar-identity perception condition (1.56+0.11; p=0.06). Thus, the perception-based analysis suggests that when subjects perceive a change in identity or expression there is release from adaptation in the FFA (Figure 5.4C). No other main effects or interactions were significant, indicating that this effect in the FFA is independent of task demands.   We identified the pSTS in all 15 participants (Table 5.1; Figure 5.5A). The GLM revealed a significant main effect of condition [F(4,56)=4.59; p<0.01], modified by an interaction between condition and analysis [F(4,56)=2.97; p<0.05] and a three-way interaction between condition, task and analysis [F(4,56)=2.76; p<0.05]. Examining data from the identity experimental run showed that the identical condition did not differ from any other condition (p>0.20, all tests), within either the stimulus-based analysis (Figure 5.5B) or the perception-based analysis (Figure 5.5C). No categorical effects were observed (p>0.10, all tests). Thus, when attention is focused on facial identity there is no measurable adaptation within the pSTS. However, examining the expression experimental run revealed a pattern similar to that seen in the FFA. The stimulus-based analysis showed a significantly smaller AUC in the identical condition (0.42+0.08) than in the similar-identity perception, similar-expression perception, or different-expression perception conditions (p<0.05, all tests) with a trend in the same direction for the different-identity perception condition (p=0.07). The remaining four experimental conditions did not differ from each other (p>0.15, all tests; Figure 5.5D). The perception-based analysis showed a significantly smaller AUC in the identical condition (0.44+0.08) than in any other condition (p<0.05, all tests). In addition, categorical effects were apparent with a significantly larger AUC in the different-identity perception condition (0.83+0.10) than the similar-identity perception condition (0.60+0.06; p<0.01) and a trend to a larger AUC in the different-expression perception condition (0.77+0.09) than the similar-expression perception    123Figure 5.5  Adaptation effects in the right pSTS                                      (A) Examples of the right posterior superior temporal sulcus in three subjects (coronal slices). No release from adaptation is apparent during the identity run, in either the stimulus-based (B) or perception-based (C) analyses. (D) During the stimulus based analysis of the expression run a significant release from adaptation (indicated with an *), as compared to the identical condition is observed in all but one condition (stimulus analysis, different-identity condition). (E) The perception-based analysis of the expression run reveals a significant release from adaptation for perceived identity changes (indicated with an  ? ) with a trend to the same effect for perceived expression changes (indicated with an  # ), evidence for a functional overlap of identity and expression processing within the pSTS.    124condition (0.58+0.08; p=0.052). Thus, the perception-based analysis of the pSTS data showed a release from adaptation when subjects perceived a change in either identity or expression, but only when task demands direct attention toward facial expression (Figure 5.5E).  5.3.3 Adaptation ? Beyond the core system The face that both the FFA and pSTS showed a similar release from adaptation for perceived changes in either identity or expression led us to ask if there were other cortical regions that might show more selective effects for identity alone or expression alone. Prior work has suggested that the mSTS is more selective than the pSTS for expression (Winston, Henson et al., 2004), and the precuneus has shown some form of identity discrimination with greater responses for familiar than novel faces (Gobbini and Haxby, 2006, 2007; Kosaka, Omori et al., 2003); therefore, these two regions seemed appropriate candidates for further region-of-interest analyses. We identified the mSTS in 13 of 15 participants (Table 5.1; Figure 5.6A). The GLM revealed a significant interaction between condition and analysis [F(4,48)=2.57; p=0.05]. Examining the stimulus-based analysis alone revealed no significant differences between any of the five experimental conditions (p>0.20; Figure 5.6B). Examining the perception-based analysis alone showed that the identical condition (0.59+0.11) again did not differ significantly from any of the other experimental conditions (p>0.05, all tests). However, a categorical effect of expression was observed, with a significantly larger AUC in the different-expression perception condition (0.69+0.09) than the similar-expression perception condition (0.42+0.05; p<0.01). No such effect was observed between identity conditions (p>0.15). Thus, the perception-based analysis suggests a release from adaptation in the mSTS when subjects perceive a change in expression, but not when they perceive a change in identity (Figure 5.6C). As no other main or interaction effects were significant, this effect in the mSTS is not modulated by task demands.   125Figure 5.6  Adaptation effects in the right mSTS       (A) Examples of the right middle superior temporal sulcus in three subjects (coronal slices). (B) The stimulus-based analysis does not reveal a significant release from adaptation in any experimental condition. (C) However, the perception-based analysis demonstrates a significant release from adaptation for perceived changes in expression (indicated with an  ? ), evidence for the mSTS involvement in the perception of facial expression only.          126  We localized the precuneus in 10 of 15 subjects (Table 5.1; Figure 5.7A). The GLM revealed a significant main effect of condition [F(4,36)=4.90; p<0.01], which was modified by a significant interaction between condition and analysis [F(4,36)=3.98; p<0.01]. Examining the stimulus-based analysis alone showed that the identical condition (0.52+0.05) did not differ significantly from any of the other experimental conditions (p>0.05, all tests), nor was there evidence for any categorical effects (p>0.05, all tests; Figure 5.7B). However, when examining the perception-based analysis alone a significantly smaller AUC in the identical condition (0.48+0.04) than in the different-identity perception condition (0.80+0.06) was observed (p<0.001). The AUC in the different-identity perception condition was also significantly larger than that in the similar-identity perception condition (0.48+0.04; p<0.001). No other conditions differed from the identical condition (p>0.30), nor was there evidence for the categorical perception of expression (p>0.60). Thus, the perception-based analysis suggests a release from adaptation in the precuneus when subjects perceive a change in identity, but not when they perceive a change in expression (Figure 5.7C). As no other main or interaction effects were significant, this effect in the precuneus is not modulated by task demands.    5.4  Discussion This study examined the role of a number of cortical regions in the categorical perception of facial identity and expression. Categorical perception suggests sensitivity to physical changes which cross a categorical boundary, but not to physical changes of an equal magnitude which do not cross a categorical boundary (Rotshtein, Henson et al., 2005). The use of morphed faces allows one to create stimuli with approximately equivalent physical differences between face pairs that do or do not cross categorical boundaries (Figure 5.1). An analysis of behavioral responses gathered during the fMRI scanning session suggested that our classification of stimulus pairs as crossing or not crossing category boundaries for expression or identity was    127Figure 5.7  Adaptation effects in the precuneus        (A) Examples of the precuneus in three subjects (sagittal slices). (B) The stimulus-base analysis does not reveal a significant release from adaptation in any experimental condition. (C) However, the perception-based analysis demonstrates a significant release from adaptation in the different-identity perception condition as compared to the identical condition (indicated with an *) and to the similar-identity perception condition (indicated with an  ? ), evidence for precuneus involvement in the perception of facial identity only.           128correct (Figure 5.2), validating the use of this stimulus-based classification in the fMRI analysis. However, it was also clear from Figure 5.2 that a substantial proportion of stimulus pairs crossing categorical boundaries were perceived as ?same? by subjects, while a smaller proportion of stimulus pairs that did not cross the categorical boundary were perceived as ?different?. This led us to construct a second ?perception-based? analysis with stimulus pairs classified according to the perceptual experience of each subject. Our results clearly show that adaptation effects emerged more distinctly in the perception-based analysis, consistent with prior reports which show that functional effects follow subjective experience more than stimulus properties (Grill-Spector, Knouf et al., 2004; Tong, Nakayama et al., 1998).   5.4.1 OFA involvement in the perception of facial structure Within the OFA maximal adaptation was observed for the identical condition, as expected. A release from adaptation was observed in all other conditions (similar-identity, different-identity, similar-expression, different-expression), and there was no difference in the magnitude of this release across these four conditions (Figure 5.3). Thus, the OFA appears to show sensitivity to any structural change in a face, whether in expression or identity, regardless of the categorical effects of these changes, and independent of task demands. These findings replicate the findings of Rotshtein et al (2005) for identity, show that the same pattern is observed even if the analysis is based upon the subjective perception of an identity change, and more importantly extend these findings to show that they also apply to physical changes related to facial expression. These results provide strong support for a role of the OFA in the early perception of facial structural properties related to the perception of both facial identity and expression (Haxby, Hoffman et al., 2000).     1295.4.2 Category effects for identity and expression perception in the FFA and pSTS Like the OFA, the FFA showed maximal adaptation for the identical condition and release from adaptation was observed in all other conditions (similar-identity, different-identity, similar-expression, different-expression). The stimulus-based analysis of FFA activity showed no difference between any of these four experimental conditions, suggesting that, like the OFA, the FFA may be sensitive to any structural change in the face. However, the perception-based analysis showed greater release from adaptation when subjects perceived a difference in either identity or expression. This result suggests that FFA activity is modulated not only with the subjective perception of facial identity, but also of facial expression (Figure 5.4).   Our findings for facial identity in the FFA replicate those of Rotshtein et al. (2005), and suggest that the effect is driven more by the subjective perception of differences in facial identity. The expression data extend beyond that of Rotshtein et al. (2005) to show that the effect of perceived difference is not specific for facial identity, but is also seen for facial expression. This may be consistent with other evidence that FFA activity is modulated by expression in the face (Ganel, Valyear et al., 2005; Ishai, Schmidt et al., 2005; Palermo and Rhodes, 2007; Vuilleumier, Armony et al., 2001). Some of these studies demonstrate increased activity in the FFA when viewing expressive faces as compared to neutral ones (Ishai, Schmidt et al., 2005; Vuilleumier, Armony et al., 2001), while others show increased activity in the FFA for expression changes, even while attending to facial identity (Ganel, Valyear et al., 2005).   The pattern of adaptation seen in the pSTS closely resembles that of the FFA, in that there is general non-specific release from adaptation compared to the identical condition in the stimulus-based analysis, but greater release from adaptation for perceived changes in either identity or expression in the perception-based analysis. While prior adaptation studies have shown evidence that the pSTS is sensitive to changes in facial expression in the image (Winston, Henson et al., 2004), our data shows more specifically that release from adaptation is enhanced   130not just when there is a change in the image, but when the viewer perceives a change in the image?s expression. Furthermore, our finding of a similar effect for facial identity again replicates previously reported sensitivity to changes in facial identity in the pSTS (Winston, Henson et al., 2004), and again shows that this release from adaptation is driven by the perceived change in the image?s identity. Unlike the FFA in our study, the pSTS only shows this adaptation pattern during the expression experimental run, when attention is directed specifically toward facial expression (Figure 5.5). This task-modulation may parallel other reports of increased activity in the pSTS when the subject explicitly attends to facial expression (Narumoto, Okada et al., 2001).  5.4.3 Adaptation in the mSTS and precuneus The lack of specificity for identity versus expression in our analysis of the FFA and pSTS led us to ask if there were other areas in which more selective effects might be found. Based on prior work we focused on two additional regions, the mSTS and the precuneus. Winston et al. (2004) used fMRI adaptation to show that the mSTS is sensitive to changes in facial expression but not to changes in facial identity. We find that the mSTS showed release from adaptation when the subject perceived a change in expression, but not a change in identity (Figure 5.6). Winston et al (2004) suggest that, contrary to current models, the independent processing of expression may occur in a portion of the superior temporal sulcus more anterior to the pSTS and our findings support this conclusion.   The precuneus showed the opposite pattern of adaptation to the mSTS, with release from adaptation when subjects perceived a change in facial identity but not when they perceived a change in facial expression (Figure 5.7). Previous studies of the precuneus have demonstrated differential activity with familiar versus novel identities (Gobbini and Haxby, 2006; Kosaka, Omori et al., 2003). One interpretation of these observations and our data is that the precuneus   131may be involved in processing facial identity, and not just semantic associations related to known faces. Our stimulus set used faces previously unknown to the subjects, and therefore lacking strong semantic associations. However, it is possible that the learning of names and the few days of training was sufficient for participants to create semantic associations for these faces. In that case, a perception of changed facial identity would also be accompanied by a change in semantic associations. Therefore our data cannot determine whether the release of adaptation seen in the precuneus is related to the specific processing of facial identity, or to the processing of semantic associations linked to facial identity, even though these were relatively weak given our use of novel faces.  5.4.4 Implications for models of identity and expression perception In a recent review challenging the independence of facial identity and expression processing in current models (Haxby, Hoffman et al., 2000), Calder and Young (2005) argued for a relative rather than absolute segregation of identity and expression perception. They based this argument on findings from principle component analyses showing that certain components are necessary for discriminating facial identity, others for discriminating facial expression, and yet others for discriminating either. They suggest that the network underlying face perception may follow this pattern, with both modules that are selective for identity and expression and modules that participate in both identity and expression discriminations (Calder and Young, 2005).    Our data suggest that, as currently modeled (Haxby, Hoffman et al., 2000), the OFA is involved in the early perception of facial structural properties related to either facial identity or expression; however, activity in the OFA does not vary with whether the subject perceives a change in the face or not. As long as there is a structural change in the face the OFA shows release from adaptation, whether or not this change causes a change in the percept of expression or identity. In contrast, change in percept does lead to release from adaptation in the FFA and   132pSTS; however, against complete independence of processing at this level, we find that both the FFA and the pSTS show this release from adaptation for both expression and identity.    Several interpretations of this finding are possible. It may be that both the pSTS and the FFA make complementary contributions to expression and identity recognition: this would not necessarily deny a dominant role for the FFA in identity processing and the pSTS in expression processing, but it would imply relative rather than absolute independence. If so, this hypothesis predicts (1) more severe deficits in expression or identity processing from lesions of both the pSTS and FFA than from lesions of either alone, and (2) partial defects in expression processing along with more severe deficits in identity processing from an FFA lesion, and the reverse with a pSTS lesion. Both of these predictions could be tested in a lesion model. A second interpretation is that the expression signal in the FFA and the identity signal in the pSTS are modulatory. Thus, for example, the identity signals in the pSTS might not indicate that it is involved in recognizing identity, but these identity signals may be modulating expression processing instead. There is some work showing that correct interpretations of facial expressions requires modulation by identity (Ganel, Goshen-Gottstein et al., 2004; Martinez and Neth, 2007), and therefore some interaction is required for optimal recognition performance. In this scenario, lesions of the FFA or pSTS alone would still be selective for identity or expression processing respectively. Our data also raise the possibility that selective processing of identity or expression processing may involve the precuneus for identity and the mSTS for expression. Prior fMRI studies support a role for the mSTS in selective expression processing (Winston, Henson et al., 2004), though there is as yet no lesion data on the impact of selective lesions of the mSTS on expression processing. The greater response of the precuneus to familiar than novel faces has been interpreted as due to retrieval of semantic associations, but it is also likely that familiar faces have stronger identity representations. The precuneus has been included in the lesions of at least some patients with prosopagnosia (Suzuki, Yamadori et al., 1996), but these lesions have   133been large and involved multiple areas, limiting any conclusions about the role of this region in identity recognition. Further investigations are required in order to determine the specific role of the mSTS and precuneus in the processing of faces.    134  5.5  References ADOLPHS R, DAMASIO H, TRANEL D, and DAMASIO AR. Cortical systems for the recognition of emotion in facial expressions. J Neurosci, 16: 7678-7687, 1996. BARTON JJ. Disorders of face perception and recognition. Neurol Clin, 21: 521-48, 2003. CALDER AJ and YOUNG AW. Understanding the recognition of facial identity and facial expression. Nat Rev Neurosci, 6: 641-51, 2005. DALE AM. Optimal experimental design for event-related fmri. Hum Brain Mapp, 8: 109-14, 1999. FOX CJ and BARTON JJ. What is adapted in face adaptation? The neural representations of expression in the human visual system. Brain Res, 1127: 80-9, 2007. FOX CJ, ORUC I, and BARTON JJS. It doesn't matter how you feel. The facial identity aftereffect is invariant to changes in facial expression. Journal of Vision, 8: 11 (1-13), 2008. GANEL T, GOSHEN-GOTTSTEIN Y, and GANEL T. Effects of familiarity on the perceptual integrality of the identity and expression of faces: The parallel-route hypothesis revisited. J Exp Psychol Hum Percept Perform, 30: 583-97, 2004. GANEL T, VALYEAR KF, GOSHEN-GOTTSTEIN Y, and GOODALE MA. The involvement of the "Fusiform face area" In processing facial expression. Neuropsychologia, 43: 1645-1654, 2005. GOBBINI MI and HAXBY JV. Neural response to the visual familiarity of faces. Brain Res Bull, 71: 76-82, 2006. GOBBINI MI and HAXBY JV. Neural systems for recognition of familiar faces. Neuropsychologia, 45: 32-41, 2007. GORNO-TEMPINI ML, PRADELLI S, SERAFINI M, PAGNONI G, BARALDI P, PORRO C, NICOLETTI R, UMITA C, and NICHELLI P. Explicit and incidental facial expression processing: An fmri study. Neuroimage, 14: 465-73, 2001. GRILL-SPECTOR K, HENSON R, and MARTIN A. Repetition and the brain: Neural models of stimulus-specific effects. Trends Cogn Sci, 10: 14-23, 2006. GRILL-SPECTOR K, KNOUF N, and KANWISHER N. The fusiform face area subserves face perception, not generic within-category identification. Nat Neurosci, 7: 555-562, 2004. HAXBY JV, HOFFMAN EA, and GOBBINI MI. The distributed human neural system for face perception. Trends Cogn Sci, 4: 223-233, 2000. ISHAI A, PESSOA L, BIKLE PC, and UNGERLEIDER LG. Repetition suppression of faces is modulated by emotion. Proc Natl Acad Sci U S A, 101: 9827-32, 2004. ISHAI A, SCHMIDT CF, and BOESIGER P. Face perception is mediated by a distributed cortical network. Brain Res Bull, 67: 87-93, 2005. KANWISHER N, MCDERMOTT J, and CHUN MM. The fusiform face area: A module in human extrastriate cortex specialized for face perception. J Neurosci, 17: 4302-11, 1997. KAUFMANN JM and SCHWEINBERGER SR. Expression influences the recognition of familiar faces. Perception, 33: 399-408, 2004. KOSAKA H, OMORI M, IIDAKA T, MURATA T, SHIMOYAMA T, OKADA T, SADATO N, YONEKURA Y, and WADA Y. Neural substrates participating in acquisition of facial familiarity: An fmri study. Neuroimage, 20: 1734-42, 2003. LUNDQVIST D and LITTON JE. The averaged karolinska directed emotional faces-akdef, 1998. MARTINEZ A and NETH D. Face configuration biases the perception of facial expressions [abstract]. Journal of Vision, 7: 943a, 2007. NARUMOTO J, OKADA T, SADATO N, FUKUI K, and YONEKURA Y. Attention to emotion modulates fmri activity in human right superior temporal sulcus. Brain Res Cogn Brain Res, 12: 225-31, 2001.   135PALERMO R and RHODES G. Are you always on my mind? A review of how face perception and attention interact. Neuropsychologia, 45: 75-92, 2007. RICKHAM PP. Human experimentation. Code of ethics of the world medical association. Declaration of helsinki. Br Med J, 2: 177, 1964. ROSSION B, CALDARA R, SEGHIER M, SCHULLER A-M, LAZEYRAS F, and MAYER E. A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing. Brain, 126: 2381-2395, 2003. ROTSHTEIN P, HENSON RN, TREVES A, DRIVER J, and DOLAN RJ. Morphing marilyn into maggie dissociates physical and identity face representations in the brain. Nat Neurosci, 8: 107-13, 2005. SAXE R, BRETT M, and KANWISHER N. Divide and conquer: A defense of functional localizers. Neuroimage, 30: 1088-96; discussion 1097-9, 2006. SERENCES JT. A comparison of methods for characterizing the event-related bold timerseries in rapid fmri. Neuroimage, 21: 1690-1700, 2004. SUZUKI K, YAMADORI A, NAGAMINE Y, and ITOYAMA Y. Transient prosopagnosia and lasting topographical disorientation after the total removal of a right occipital arteriovenous malformation [article in japanese]. Rinsho Shinkeigaku, 36: 1114-1117, 1996. TONG F, NAKAYAMA K, VAUGHAN JT, and KANWISHER N. Binocular rivalry and visual awareness in human extrastriate cortex. Neuron, 21: 753-9, 1998. VUILLEUMIER P, ARMONY JL, DRIVER J, and DOLAN RJ. Effects of attention and emotion on face processing in the human brain: An event-related fmri study. Neuron, 30: 829-41, 2001. WINSTON JS, HENSON RN, FINE-GOULDEN MR, and DOLAN RJ. Fmri-adaptation reveals dissociable neural representations of identity and expression in face perception. J Neurophysiol, 92: 1830-9, 2004. YOVEL G and KANWISHER N. The neural basis of the behavioral face-inversion effect. Curr Biol, 15: 2256-2262, 2005.                    1366   DEFINING THE FACE PROCESSING NETWORK: OPTIMIZATION OF THE      FUNCTIONAL LOCALIZER IN FMRI5      6.1  Introduction Functional magnetic resonance imaging (fMRI) studies have identified a network of cortical regions that are involved in face perception. The first face-selective region found was the fusiform face area (FFA), located in the right fusiform gyrus (Kanwisher, McDermott et al., 1997), consistent with the predominance of right inferotemporal lesions in prosopagnosia (Damasio, 1985; de Renzi, 1986; Landis, Cummings et al., 1986; Meadows, 1974). Subsequently, other studies have characterized face-selective regions in the inferior occipital gyrus (occipital face area, OFA) and the posterior superior temporal sulcus (pSTS) (Ishai, Schmidt et al., 2005; Rossion, Caldara et al., 2003; Winston, Henson et al., 2004). Together, these three regions are currently considered the ?core system? for face processing (Gobbini and Haxby, 2007; Haxby, Hoffman et al., 2000) (Figure 6.1).  Beyond the core system are a number of additional regions that may contribute to face perception. These include regions in the inferior frontal gyrus (Ishai, Schmidt et al., 2005), amygdala (Adolphs, Tranel et al., 1994), precuneus (Kosaka, Omori et al., 2003), anterior paracingulate gyrus (Gobbini and Haxby, 2006) and a more anterior portion of the superior temporal sulcus (Winston, Henson et al., 2004), among others. These regions comprise the ?extended system? for face perception (Gobbini and Haxby, 2007; Haxby, Hoffman et al., 2000) (Figure 6.2). The identification of face-related cortical regions has led to the question of what roles each of these regions may play within face perception (Haxby, Hoffman et al., 2000). One way to examine the function of these face-related regions is by first identifying or ?localizing? the                                                   5 A version of this chapter has been published. Fox C.J.*, Iaria G.*, and Barton J.J.S. (in press). Defining the face-processing network: optimization of the functional localizer in fMRI. Human Brain Mapping. ? 2008 Wiley-Liss, Inc.   137Figure 6.1  FMRI localized core system of face perception    Representative fMRI images for regions comprising the core system of face perception: Occipital Face Area (OFA); Fusiform Face Area (FFA); Posterior Superior Temporal Sulcus (pSTS). Overlay maps of the face>object contrast are set at the threshold of p<0.05 (1-tailed Bonferroni). Results from the static localizer are overlaid in blue and the results of the dynamic localizer in orange-yellow. Clear overlaps in all regions can be seen, with more widespread activity readily apparent in the dynamic localizer maps.             138Figure 6.2  FMRI localized extended system of face perception    Representative fMRI images for regions comprising the extended system of face perception. Top Left: bilateral Middle Superior Temporal Sulcus (mSTS). Top Right: bilateral Amygdala (AMG). Bottom Left: Inferior Frontal Gyrus (IFG). Bottom Right: Precuneus (preC; red arrow) and Anterior Paracingulate Cortex (aPC; white arrow). Due to the poor localization of these regions using the static localizer, overlaid maps are results from the statistical analysis of the dynamic localizer (faces>objects; p<0.05, 1-tailed Bonferroni).         139region-of-interest (ROI) and then studying functional changes within this ?localized? ROI on subsequent experimental tasks (Saxe et al. (2006); but see Friston et al. (2006)). Functional face localizers today are very similar to the scans used by Kanwisher and colleagues (1997) that first identified the FFA. These localizers normally contrast hemodynamic activity during blocks where subjects view static images of faces with blocks where subjects view static images of diverse objects, scrambled faces, or a single non-face object class, such as houses (Andrews and Ewbank, 2004; Eger, Schyns et al., 2004; Gauthier, Tarr et al., 2000; Golarai, Ghahremani et al., 2007; Golby, Gabrieli et al., 2001; Ishai, Schmidt et al., 2005; Ishai, Ungerleider et al., 2000; Mazard, Schiltz et al., 2006; Pyles, Garcia et al., 2007; Rotshtein, Henson et al., 2005; Schiltz and Rossion, 2006; Yovel and Kanwisher, 2005). As noted by Kanwisher and colleagues (1997), and illustrated by these other studies, a contrast between static images of faces and objects does quite well at identifying the right FFA but is not as reliable in identifying the right OFA, right STS or the left hemispheric counterparts of these three regions. Furthermore, as face-specificity decreases in regions of the extended system, the power of the standard face localizer also decreases (Ishai, Schmidt et al., 2005), making it difficult to identify regions consistently across subjects, and thereby limiting the feasibility of examining specific hypotheses within many of these regions. One approach to dealing with this problem of inconsistent functional localization is by performing group analyses on normalized brains (Kosaka, Omori et al., 2003; Winston, Henson et al., 2004; Friston, Rotshtein et al., 2006). Group analyses can identify cortical ROIs in the group data that are not seen consistently in each individual subject. However, normalizations often fail to fully account for between-subject structural anatomic variability (Nieto-Castanon, Ghosh et al., 2003; McKeown and Hanlon, 2004), and even if they did, the group analysis would still be complicated by between-subject functional anatomic variability, in that functionally   140active regions may vary in their cerebral location from one subject to another (Wohlschlager, Specht et al., 2005).  A second approach to dealing with inconsistent functional localization that avoids normalization has been to vary the statistical threshold at which a ROI is identified in single subjects. Thresholds used in prior studies have ranged from very conservative (e.g. ?p<0.05 with Bonferroni correction for multiple comparisons? (Andrews and Ewbank, 2004; Eger, Schyns et al., 2004; Schiltz and Rossion, 2006; Schiltz, Sorger et al., 2006; Sorger, Goebel et al., 2007)) to more liberal ones (e.g. ?p<0.001 uncorrected? (Golarai, Ghahremani et al., 2007; Ishai, Schmidt et al., 2005) or ?t>2.0 uncorrected? (Golby, Gabrieli et al., 2001)). More liberal thresholds may reveal more areas more consistently, but do not account for the problem of multiple comparisons across the thousands of voxels within the brain, and are often not objectively set, raising questions about their statistical validity (Genovese, Lazar et al., 2002). Furthermore, these variations in threshold can affect not only the likelihood of identifying a functional region, but the size of the functionally activated region. More liberal thresholds tend to show larger areas of activation, and more conservative ones show smaller areas. Which of these thresholds more accurately reflects the anatomic reality is open to question, especially in situations where there are no guiding estimates of the size of a cortical region from primate neurophysiology or human histology. The inconsistent detection of face-selective areas with current localizer protocols is also problematic for another field, the application of functional imaging to neuropsychological patients. Increasingly fMRI is being used to determine if lesions have affected specific functional regions in specific individuals (Rossion, Caldara et al., 2003; Avidan, Hasson et al., 2005). This is particularly the case in patients with acquired lesions, where the variability in lesion anatomy makes group analyses inadvisable (Barton, 2003). However, it is difficult to make firm conclusions if these areas cannot be consistently demonstrated in all healthy individuals.    141In the present study we addressed two issues regarding the localization of face-processing areas in the human brain. First, we asked whether it was possible to create a better face-localizer. Standard functional localizers for face processing rely on a contrast between static images of faces and objects. However, normal experience with these stimuli is dynamic, not static. Also, there is some evidence of increased activity within face-related ROIs for dynamic faces as compared to static faces (Kilts, Egan et al., 2003; Sato, Kochiyama et al., 2004). Furthermore, neurophysiological studies show that there are neurons that respond to different facial images (e.g. viewpoint differences (Perrett, Oram et al., 1991)), which may suggest that the changes in facial image inherent to dynamic stimuli may cause activation of a greater pool of neurons. For these reasons we hypothesized that the dynamic localizer would result in (1) a higher likelihood of localizing face-processing areas in individual subjects and (2) more robust activity within all face-related ROIs. To study this we compared two functional localizers, one contrasting static images of faces versus objects and the other contrasting dynamic video clips of faces versus objects. Second, we asked, how large should a face-selective ROI be?  Although face-selective responses have been identified within the inferotemporal cortex and superior temporal sulcus in monkeys (Perrett, Rolls et al., 1982), there is no neurophysiologically based estimate of size for any potential homologues of the FFA, OFA or STS (unlike the case with the V5 complex (Brewer, Press et al., 2002)). In the absence of such anatomic data, we asked whether it would be possible to derive statistical criteria that would provide more consistent estimates of the size of face-selective regions, with optimum specificity for face stimuli.       142  6.2  Defining face-selective ROIs: Localizers with static or dynamic stimuli  6.2.1 Methods  6.2.1.1 Participants. Sixteen right-handed healthy participants (8 females; Mean age + SD:  25.6 + 4.1 years) with normal or corrected-to-normal vision and no history of neurological disorders participated. Informed consent was obtained and the protocol approved by the institutional review boards of the University of British Columbia and Vancouver General Hospital, in accordance with The Code of Ethics of the World Medical Association, Declaration of Helsinki (Rickham, 1964).  6.2.1.2 Stimuli Participants underwent two functional scans. During the first functional scan participants viewed static photographs of non-living objects (e.g. television, basketball) and faces (neutral and expressive) presented in separate blocks. This static localizer is similar to those used to identify face-related cerebral regions by Kanwisher et al (1997) and others (Andrews and Ewbank, 2004; Golarai, Ghahremani et al., 2007; Golby, Gabrieli et al., 2001; Grill-Spector, Knouf et al., 2004; Mazard, Schiltz et al., 2006; Pyles, Garcia et al., 2007; Reddy and Kanwisher, 2007; Rhodes, Byatt et al., 2004; Schiltz, Sorger et al., 2006; Schwarzlose, Baker et al., 2005; Sorger, Goebel et al., 2007; Spiridon and Kanwisher, 2002; Yovel and Kanwisher, 2005). Participants performed a ?one-back task?: that is, to press a button if an image was identical to the previous one. The static localizer began and ended with a fixation block showing a cross in the centre of an otherwise blank screen. Additional fixation blocks were alternated with image blocks, with all blocks lasting 12 seconds. Six blocks of each image category (object, neutral face, expressive face) were presented in a counterbalanced order. Each image block consisted of 15 images (12 novel and 3   143repeated), all sized to a width of 400 pixels and presented at screen center for 500ms, with an inter-stimulus-interval (ISI) of 300ms. The static localizer took 444 seconds in total. During the second functional scan, participants viewed video-clips of non-living objects and faces presented in separate blocks. We referred to this scan as the dynamic localizer. Video-clips of faces all displayed dynamic changes in facial expression (i.e.- from neutral to happy). So that dynamic changes in objects were comparable to those seen in faces, all video-clips of objects displayed types of motion that did not create large translations in position (see Table 6.1). Participants again performed a one-back task. Identical fixation blocks began and ended the session and were alternated with image blocks, with all blocks lasting 12 seconds. Eight blocks of each image category (object, face) were presented in a counterbalanced order. Each image block consisted of 6 video-clips (5 novel and 1 repeated) presented centrally for 2000ms each. Video-clips of objects where gathered from the internet, and video-clips of faces were provided by Chris Benton (Department of Experimental Psychology, University of Bristol, UK). All video-clips were resized to a width of 400 pixels. The dynamic localizer took 396 seconds in total.  6.2.1.3 FMRI data acquisition and analysis All scans were acquired in a 3.0 Tesla Philips scanner. Stimuli were presented using Presentation 9.81 software and were rear-projected onto a mirror mounted on the head coil. Whole brain anatomical scans were acquired using a T1-weighted echoplanar imaging (EPI) sequence, consisting of 170 axial slices of 1mm thickness (1mm gap) with an in-plane resolution of 1mm X 1mm (FOV=256). T2-weighted functional scans (TR=2s; TE=30ms) were acquired using an interleaved ascending EPI sequence, consisting of 36 axial slices of 3mm thickness (1mm gap) with an in-plane resolution of 1.875mm X 1.875mm (FOV=240). The static localizer scan     144Table 6.1  Objects and motions included in the dynamic localizer  Fountain spraying.  Piano keys depressing. Ceiling fan spinning.  Plant blowing. Globe spinning.  Record player rotating. Merry-go-round rotating.  Roulette wheel spinning. Jiffy-pop expanding.  Scale balancing. Juice pouring.  Scale needle rotating. Kettle steaming.  Oscilliscope wave fluctuating. Tree branch blowing.  Stopwatch numbers changing. Newton?s balls bouncing.  Tennis ball spinning. Gears cranking.  Car tire rotating. Toilet water flushing.  Coffee machine pouring. Top spinning.  Film reels rotating. Traffic light changing.  Grass and heather blowing. Office fan oscillating.  Waterfall flowing. Washing machine spinning.  Fireplace burning. Water faucet dripping.  Fireworks exploding. Windmill rotating.  Flag waving. Blender mixing.  Eggs and water boiling. Candle flickering.  Flower blowing. Cigarette burning.  Sewing machine sewing.               145consisted of 223 functional volumes, while the dynamic localizer scan consisted of 199 functional volumes. The first volume of each functional scan was discarded to allow for scanner equilibration. All MRI data were analyzed using BrainVoyager QX Version 1.8 (www.brainvoyager.com). Anatomical scans were not preprocessed. Preprocessing of functional scans consisted of corrections for slice scan time acquisition, head motion (trilinear interpolation), and temporal filtering with a high pass filter in order to remove frequencies less than 3 cycles/time course. For each participant, functional scans were individually co-registered to their respective anatomical scan, using the first retained functional volume to generate the co-registration matrix. The static localizer time course was analyzed with a single subject general linear model (GLM), with object (O), neutral (NF) and expressive (EF) faces as predictors. Analysis of NF+EF>2*O was overlaid on the whole brain and significance was set at p<0.05, with correction for multiple comparisons (1-tailed Bonferroni). A similar procedure was adopted for the dynamic localizer, the time course of which was analyzed via a single subject GLM with objects (O) and faces (F) as predictors. Analysis of F>O was overlaid on the whole brain and significance was set at p<0.05, with correction for multiple comparisons (1-tailed Bonferroni).  6.2.1.4 ROI localization and analysis  Within each participant we first attempted to define each of the three face-related regions comprising the core system of face perception. Contiguous clusters of face-related voxels located on the lateral temporal portion of the fusiform gyrus were designated as the fusiform face area (FFA), while clusters located on the lateral surface of the inferior occipital gyrus were designated as the occipital face area (OFA). Face-related clusters located on the posterior segment of the superior temporal sulcus were designated as the pSTS. All regions were defined in both right and left hemispheres.   146In addition to these ?core? face-processing regions, we also examined regions comprising the extended system of face perception. Contiguous clusters of face-related voxels located on the anterior segment of the superior temporal sulcus of the right or left hemispheres were designated as the middle superior temporal sulcus (mSTS) as described by Winston et al. (2004). Face-related clusters within the amygdala (AMG) and inferior frontal gyrus (IFG) were defined bilaterally. Face-related clusters within the precuneus (preC) and the anterior paracingulate cortex (aPC) were defined unilaterally due to their location at the midline between cerebral hemispheres. For each identified ROI, the t-value of the peak voxel and cluster size (number of voxels) was determined. ROIs that were not identified at the a priori statistical threshold (p<0.05, 1-tailed Bonferroni) were assigned a cluster size of zero, but the statistical threshold was lowered to a more liberal False-Discovery-Rate threshold of q<0.05 (corrected for multiple comparisons) to determine the t-value of the peak voxel within that region. For statistical analysis, failing to localize a region at the False-Discovery-Rate threshold resulted in this region being assigned the False-Discovery-Rate threshold as a default t-value of its peak voxel. Analyses were performed separately for the core and extended systems. An initial paired t-test was performed within each system, with the number of ROIs localized within each subject as the dependent variable, in order to determine whether one localizer more consistently localized face-related ROIs than the other. All core system General Linear Models (GLM) consisted of localizer (static, dynamic), ROI (OFA, FFA, pSTS), and hemisphere (right, left) as fixed factors and subject as a random factor. The unilateral nature of the precuneus and anterior paracingulate cortex made it impossible to include hemisphere as a factor, thereby restricting all extended system GLMs to localizer (static, dynamic) and ROI (right-mSTS, left-mSTS, right-AMG, left-AMG, right-IFG, left-IFG, preC, aPC) as fixed factors and subject as a random factor. Within each system, five separate univariate GLMs were performed, each with a different dependent   147variable: i) t-value of the peak voxel; ii) cluster size; iii) X Talairach coordinate of the peak voxel; iv) Y Talairach coordinate peak voxel; and v) Z Talairach coordinate of the peak voxel.  GLMs considering t-value of the peak voxel and cluster size assessed the robustness and extent of face-related activity respectively. GLMs considering Talairach coordinates of the peak voxel determined whether both the static and dynamic localizers did in fact localize the same functional regions. Post-hoc t-tests were performed to analyze all significant effects. All statistical analyses were performed using SPSS 14.0 (www.spss.com), and significance on all tests was set at ? <0.05.    6.2.2  Results  6.2.2.1 Core System When localizing ROIs comprising the core system of face perception, the static localizer operated at a 72% success rate, whereas the dynamic localizer achieved a 98% success rate, a statistically significant difference [t(15)=-3.93; p=0.001]. The dynamic localizer was more successful than the static localizer in localizing all regions of the core system except the left-FFA, which was localized in all subjects with both localizers (Table 6.2). The only regions not identified with 100% success with the dynamic localizer were the left OFA and left pSTS, which were missed in one subject each. In contrast, the static localizer had difficulty locating even the right FFA in 3 subjects. Regarding the t-value of the peak voxel, we observed a significant main effect of localizer [F(1,15)=46.15; p<0.001] with the dynamic localizer (Mean t-value + SEM; 9.57+0.37) eliciting more robust face-related activity than the static localizer (6.74+0.26) (Figure 6.1). We also observed a main effect of ROI [F(2,30)=4.45; p<0.05] and an interaction between ROI and Hemisphere [F(2,30)=4.15; p<0.05], with the strongest face-related activity within the FFA    148Table 6.2  Core system of face perception as localized by both localizers  Region-of-Interest (Core System) Localizer  # of Subjects (n=16) X  Y  Z Right OFA  Static  10  40 + 11  -75 + 20  -9 + 8   Dynamic  16  36 + 10  -79 + 19  -14 + 8 Left OFA  Static  14  -38 + 10  -73 + 19  -15 + 8   Dynamic  15  -37 + 10  -74 + 19  -17 + 7 Right FFA  Static  13  35 + 9  -48 + 14  -19 + 7   Dynamic  16  37 + 9  -48 + 15  -20 + 7 Left FFA  Static  16  -35 + 10  -42 + 12  -20 + 6   Dynamic  16  -37 + 9  -43 + 13  -20 + 6 Right pSTS  Static  11  53 + 13  -44 + 13  4 + 5   Dynamic  16  54 + 13  -41 + 13  4 + 5 Left pSTS  Static  5  -50 + 14  -55 + 16  8 + 5   Dynamic  15  -51 + 14  -52 + 15  6 + 5  Number of participants in which the ROI was localized and average Talairach coordinates (Mean + SD) are reported for both the static and dynamic localizers. ROIs were localized using the contrast faces>objects with a statistical threshold of p<0.05 (1-tailed Bonferroni). The dynamic localizer was more consistent in identifying face-related ROIs in all areas of the core system excluding the left FFA which was identified in all subjects with both localizers. Average Talairach coordinates of the peak voxels show that static and dynamic localizers localized similarly positioned ROIs.         149(right=9.22+0.57; left=9.23+0.68) and the weakest within the left-STS (6.47+0.51). Finally, we observed a significant interaction between localizer and ROI [F(2,30)=5.41; p=0.01], with significantly more robust activity elicited by the dynamic localizer in all regions of the core system (p<0.001, all tests), but with the largest effects occurring within the pSTS (?  t-value + SEM; OFA=2.08+0.56, FFA=2.35+0.51, pSTS=4.03+0.59) (Figure 6.3A). The analysis of ROI cluster size showed a similar pattern. We found main effects of localizer [F(1,15)=21.65; p<0.001] with larger clusters elicited by the dynamic localizer (Mean # of voxels + SEM; 450+58) than the static localizer (129+23) and of hemisphere [F(1,15)=16.81; p=0.001] with larger clusters in the right hemisphere (392+58) than in the left (187+28). Significant interaction effects between localizer and ROI [F(2,30)=4.62; p<0.05] and between localizer and hemisphere [F(1,15)=13.67; p<0.005] were modified by a three-way interaction between localizer, ROI and hemisphere [F(2,30)=5.77; p<0.01]. Post-hoc t-tests revealed significantly larger clusters elicited by the dynamic localizer in the left-OFA, right-FFA, and bilateral-pSTS (p<0.05, all tests) with a trend in the same direction for the right-OFA (p=0.06). Again the largest effect of the dynamic localizer was observed in the right-pSTS (?  cluster size + SEM = 896 voxels+234; Figure 6.3B). Finally, we performed GLMs considering the Talairach coordinates of the peak voxel of all localized ROIs. While we saw significant main effects of ROI and significant interaction effects between ROI and hemisphere for all three coordinates [F(2,30)>5.00; p<0.05, all tests], as can be expected for ROIs found in different cortical areas, the only significant effect of localizer was a three-way interaction between localizer, ROI and hemisphere when considering the X (medial-lateral) coordinate [F(2,24)=4.283; p<0.05]. Post-hoc t-tests revealed a significant difference in the X coordinate of the right-FFA as localized by the static and dynamic localizers (p<0.05), with the static right-FFA slightly more medial (Mean X coordinate + SEM; 35+1) than the dynamic right-FFA (37+1). While this difference in the X coordinate of the peak voxel of the    150Figure 6.3   Comparison of static and dynamic localizers in the core system                                      Results from the statistical comparison of the static (white bars) and dynamic localizers (gray bars) (Mean + SEM). (A) When using the dynamic localizer significantly higher t-values are seen in the peak voxel of all regions in the core system (indicated with an asterisk), with the largest effects observed in the pSTS. (B)  Use of the dynamic localizer results in the localization of significantly larger clusters of face-related activity within the left-OFA, right-FFA, and bilateral-pSTS (indicated with an asterisk), with a similar but non-significant pattern observed in other regions of the core system.     151right FFA is statistically significant, the absolute difference (2mm) becomes negligible when considering the mean volume of the right-FFA (368mm3). Thus both the static and dynamic localizers appear to be localizing the same regions of face-related activity (Table 6.2; Figure 6.1).  6.2.2.2 Extended System While the static localizer only found 31% of regions in the extended system, the dynamic localizer had a 69% success rate, a significant difference [t(15)=-4.24; p=0.001]. The dynamic localizer was more successful in localizing all regions of the extended system (Table 6.3; Figure 6.2). Regarding the t-value of the peak voxel we observed a significant main effect of localizer [F(1,15)=19.18; p=0.001] with the dynamic localizer (Mean t-value + SEM; 6.12+0.18) eliciting more robust face-related activity than the static localizer (4.45+0.10). We also observed a main effect of ROI [F(7,105)=6.99; p<0.001] with the strongest face-related activity within the right-IFG (6.19+0.39) and the weakest within the amygdala (right-4.57+0.24; left-4.12+0.22). Finally, we observed a significant interaction between localizer and ROI [F(7,105)=2.55; p<0.05], with significantly more robust activity elicited by the dynamic localizer in all regions of the extended system (p<0.05, all tests) except the left-IFG, which showed a trend in the same direction (p=0.053) (Figure 6.4A). Analysis of ROI cluster size showed a similar pattern. We observed a significant main effect of localizer [F(1,15)=12.07; p<0.005] with larger clusters elicited by the dynamic localizer (Mean # of voxels + SEM; 189+36) than by the static localizer (13+4). We also observed a main effect of ROI [F(7,105)=3.28; p<0.005] with the largest clusters observed within the right-IFG (218+76) and the smallest within the amygdala (right-13+6; left-11+6). Finally, we observed a significant interaction between localizer and ROI [F(7,105)=3.14; p<0.01], with significantly    152Table 6.3  Extended system of face perception as localized by both localizers  Region-of-Interest  Localizer  # of Subjects (n=16) X  Y  Z Right mSTS  Static  5  51 + 14  -5 + 8  -9 + 6   Dynamic  11  49 + 12  -5 + 10  -11 + 7 Left mSTS  Static  3  -53 + 14  -16 + 8  -5 + 6   Dynamic  12  -55 + 15  -16 + 8  -5 + 6 Right AMG  Static  3  18 + 5  -2 + 2  -13 + 5   Dynamic  7  18 + 6  -4 + 2  -11 + 4 Left AMG  Static  0  -18 + 8  -3 + 4  -11 + 5   Dynamic  5  -19 + 7  -6 + 4  -14 + 5 Right IFG  Static  7  42 + 11  19 + 10  26 + 12   Dynamic  15  46 + 12  21 + 11  22 + 12 Left IFG  Static  6  -40 + 12  20 + 11  20 + 11   Dynamic  12  -47 + 12  17 + 7  21 + 9 preC  Static  7  0 + 4  -59 + 18  30 + 11   Dynamic  14  1 + 4  -62 + 16  30 + 10 aPC  Static  9  2 + 5  59 + 15  7 + 9   Dynamic  12  6 + 4  54 + 14  20 + 12  Number of participants in which the ROI was localized and average Talairach coordinates (Mean + SD) are reported for both static and dynamic localizers. ROIs were localized using the contrast faces>objects with a statistical threshold of p<0.05 (1-tailed Bonferroni). In all cases the dynamic localizer was more consistent in identifying face-related ROIs. Average Talairach coordinates of the peak voxels show that static and dynamic localizers localized similarly positioned ROIs.      153Figure 6.4  Comparison of static and dynamic localizers in the extended system                                       Results from the statistical comparison of the static (white bars) and dynamic localizers (gray bars) (Mean + SEM). (A) When using the dynamic localizer significantly higher t-values are seen in the peak voxel of all regions in the extended system (indicated with an asterisk), excluding the left-IFG, which shows a trend in the same direction. (B) Use of the dynamic localizer results in the localization of significantly larger clusters of face-related activity within the bilateral mSTS, right-IFG, preC, and aPC (indicated with an asterisk).    154larger clusters elicited by the dynamic localizer in the bilateral mSTS, right-IFG, precuneus and anterior paracingulate (p<0.05, all tests) (Figure 6.4B). Finally, a comparison of Talairach coordinates of the peak voxel for all localized ROIs revealed main effects of ROI for all three coordinates [F(7,105)>100; p<0.001, all tests], as can be expected for ROIs located in different cortical areas, and significant interaction effects between localizer and ROI for the X coordinate [F(7,79)=3.89; p=0.001] and the Z (superior-inferior) coordinate [F(7,79)=3.035; p<0.01], but not the Y (anterior-posterior) coordinate [F(7,79)=0.30; p>0.50]. Post-hoc t-tests revealed a significant difference in the X coordinate for the left-IFG (p<0.01), with the static left-IFG slightly more medial (Mean coordinate + SEM; -40+1) than the dynamic left-IFG (-47+1). Differences were also observed in the X coordinate (p<0.05) and the Z coordinate (p<0.005) of the anterior paracingulate cortex, with the static aPC more medial (2+1) and inferior (7+2) to the dynamic aPC (X= 6+1; Z= 20+3). While the absolute differences seen here are larger than those seen within the right-FFA (leftIFG-X = 7mm; aPC-X = 4mm; aPC-Z = 13mm) the differences in peak voxel location are still negligible when compared to the mean volume of these regions (left-IFG = 40mm3; aPC = 184mm3). Thus, with the possible exception of the left-IFG and the aPC, both the static and dynamic localizers appear to be localizing the same regions of face-related activity (Table 6.3; Figure 6.2).    6.3  Defining face-selective ROIs: the statistically optimal cluster size The dynamic localizer proved to be a much more consistent localizer of face-related activity than the static localizer, for both the core and extended systems of face perception. As well, we found increases in the cluster size of many of face-related regions-of-interest. This increase in cluster size highlights an additional question regarding functional localization: what is the ?right size? of the cortical region being identified?   155This problem of variable cluster size is not unique to our contrast between static and dynamic localizers. Variations in the statistical threshold used to define face-selective activation in the literature have resulted in a wide range of reported cluster sizes for all regions of the core system: right-OFA, 138mm3 (Sorger, Goebel et al., 2007) to 4289mm3 (Ishai, Pessoa et al., 2004); left-OFA, 312mm3 (Schiltz and Rossion, 2006) to 4430mm3 (Ishai, Pessoa et al., 2004); right-FFA, 498mm3 (Schiltz and Rossion, 2006) to 4711mm3 (Ishai, Pessoa et al., 2004); left-FFA, 379mm3 (Schiltz and Rossion, 2006) to 4500mm3 (Ishai, Pessoa et al., 2004); right-pSTS, 193mm3 (Sorger, Goebel et al., 2007) to 5695mm3 (Ishai, Pessoa et al., 2004); and left-pSTS, 156mm3 (Sorger, Goebel et al., 2007) to 3656mm3 (Ishai, Pessoa et al., 2004). Indeed, even within the regions we localized in the present study we see individual clusters identified as the right-FFA up to a volume of 766mm3 when using the static localizer and up to a volume of 1657mm3 when using the dynamic localizer.  The 10-fold or greater variation in ROI size reported in the face-processing literature underscores the problem of threshold-dependent methods of defining ROI size. While there may be some inter-subject variability in ROI size, these estimates can also vary within individuals when signal strength is modulated by additional factors such as attention and fatigue (Wojciulik, Kanwisher et al., 1998). However, adjusting findings to approximate the ?right size? of these functional areas is impossible without additional data from other sources on the anatomic size of these regions. In the absence of such information, asking what should be the ?right size? of an ROI becomes a question of what is the statistically optimal estimate of ROI size. In this direction, some efforts have been made to determine the ?right size? of localized ROIs using complex statistical techniques (Ng, Abugharbieh et al., 2007), and others have noted that face-selectivity decreases with increased ROI size (Golarai, Ghahremani et al., 2007). Within any given ROI is a peak voxel that shows the largest difference between face-related activity and object-related activity (face>object). With the statistical threshold set to this   156maximal difference we observe a cluster of only 1 voxel in size. As the threshold is reduced to include more voxels in the cluster the average face>object difference will decrease (all other voxels have a smaller face>object difference than the peak voxel), but the standard deviation of this average difference will also decrease (due to the averaging of more and more voxels). Initially the decrease in standard deviation may result in a larger statistical face>object difference for the cluster, but at some point the decrease in absolute face>object difference of the less selective voxels being added to the cluster will outweigh any further reduction in the standard deviation. Using this statistical rationale we attempted to determine at what ROI size the maximal face>object difference is achieved and thereby estimate the ?statistically optimal size? of these face-related ROIs. Due to the wide variety of inputs that can activate ROIs in the extended system (i.e.- non-face or non-visual stimuli), we restricted our analyses to ROIs in the core system which preferentially respond to viewed faces (Haxby, Hoffman et al., 2000).   6.3.1 Methods  6.3.1.1 Regions-of-interest and analysis Participants, stimuli, fMRI data acquisition, fMRI data processing and analyses, and ROI localization were provided in the previous section. Our findings indicated that static and dynamic localizers did in fact localize the same regions-of-interest within the core system, and that the dynamic localizer was more successful in localizing face-related ROIs. For these reasons only dynamically-localized ROIs were included in the following analyses. The ROIs which were not localized (2 of 96) at the statistical threshold previously used (p<0.05, 1-tailed Bonferroni) were localized at a more conservative threshold of q<0.05 (False Discovery Rate, corrected for   157multiple comparisons). This resulted in a full localization of all 6 core system ROIs in each of the 16 participants. A single subject GLM (as described in the previous section) was used to estimate the ?-weights for faces and objects within the peak voxel of each localized ROI. A t-value for this peak voxel was then calculated by dividing the difference between the face and object ?-weights by the combined standard error of these two ?-weights (this calculated t-value will subsequently be referred to as face-selectivity). Face-selectivity was then determined for larger and larger clusters, in 25 voxel increments, which remained centered about their respective peak voxel. For each ROI, when the cluster reached a maximum size of 500 voxels, or when it merged with another cluster of >25 voxels the process of determining face-selectivity at increasing cluster sizes was stopped.  A group analysis of face-selectivity as a function of cluster size was performed. Change in face-selectivity with respect to the peak voxel was used as the dependent variable rather than absolute face-selectivity (i.e. t-value(cluster)-t-value(peak voxel)), to account for between-subject variability in this measure. A GLM was performed on these values with ROI (OFA, FFA, pSTS), hemisphere (right, left) and cluster size (1, 25, 50 ,?, 450, 475, 500) as fixed factors. Post-hoc t-tests were performed on all significant main and interaction effects. Of particular interest for this analysis is a main effect of cluster size, which would indicate differing levels of face-selectivity in clusters of different sizes. Post-hoc t-tests would then indicate at what cluster sizes we observed an increase in face-selectivity over that of the peak voxel alone, allowing for an estimate of the statistically optimal size of the ROI. Next, an individual-based analysis was performed by determining the cluster size at which maximal face-selectivity was observed for each ROI in each individual. A priori 1-tailed t-tests were performed separately on each of the 6 core system ROIs to determine whether the average cluster size at which maximal face-selectivity was observed was significantly larger than 1 (i.e.-   158the peak voxel alone). Cluster size was then entered as the dependent variable in a GLM with ROI (OFA, FFA, pSTS) and hemisphere (right, left) as fixed factors and subject as a random factor to determine whether the average cluster size for the maximal face-selectivity differed across ROIs or across hemispheres. Post-hoc t-tests were performed on all significant main and interaction effects.  Finally as a practical comparison between all three methods of localization we performed a third GLM, with ROI (OFA, FFA, pSTS), hemisphere (right, left) and method (fixed statistical threshold as in the first section of this report, fixed cluster size using the sizes determined in the group portion of the analysis above, and individually-determined cluster size using the technique in the last paragraph) as fixed factors, subject as a random factor, and face-selectivity (t-value) as the dependent variable. Post-hoc t-tests were performed on all significant main and interaction effects. Significance levels were set at ? <0.05 on all statistical tests.   6.3.2 Results The initial group analysis revealed a main effect of cluster size [F(20,1322)=4.42; p<0.001]. Post-hoc t-tests comparing the face-selectivity of each cluster size with that of the peak voxel revealed significantly increased face-selectivity in clusters of 25 (p<0.001), 50 (p<0.005) and 75 voxels (p<0.05), with respect to the peak voxel. Significant decreases in face-selectivity were observed for clusters larger than 325 voxels (p<0.05, all tests) with a trend in the same direction for clusters of 300 voxels (p=0.06) (Figure 6.5A). Cluster size did not interact significantly with any other factor (p>0.50), indicating a similar pattern of face-selectivity effects within all ROIs (Figure 6.5B). However, we did observe a significant interaction between ROI and hemisphere [F(2,1322)=5.81; p<0.005]. Post-hoc t-tests revealed higher face-selectivity on the whole within the right-FFA (Mean + SEM; 0.09 + 0.08) than in any other region (p<0.05, all tests), a result    159Figure 6.5  Face selectivity as a function of cluster size in the core system                                      Face-selectivity as a function of cluster size (Mean + SEM; significant values marked with an asterisk and trends with a pound sign). (A) Averaged results from all 6 ROIs (bilateral OFA, FFA, and pSTS). Clusters between 25 and 75 voxels show significantly increased face-selectivity with respect to their respective peak voxel. Clusters of >325 voxels show significantly decreased face-selectivity. (B) To illustrate the common effect within all 6 ROIs, individual curves are plotted. A slight broadening of the right-FFA peak of face-selectivity can be observed (solid red line).     160most likely due to the broader peak of increased face-selectivity seen within this ROI (Figure 6.5B). The analysis of the individually-determined cluster sizes at which maximal face-selectivity occurs revealed no main effect of ROI or hemisphere or an interaction between the two factors (p>0.50, all tests). This result is in agreement with the GLM on the group-determined cluster size of maximal face-selectivity, which showed similar patterns in all 6 core system ROIs. An average cluster size of roughly 50 voxels (50 mm3) was the size at which maximal face-selectivity was achieved in all ROIs, and a priori t-tests showed that this average cluster size was in fact significantly larger than 1 (i.e.- the peak voxel alone), again for all 6 ROIs (p<0.05) (Figure 6.6A). A frequency plot of the individually determined cluster size with maximal face-selectivity reveals a skewed distribution with most ROIs reaching maximal face selectivity by a cluster size of 75 voxels (83.33%) and only 2 of 96 ROIs (2.08%) reaching maximal face-selectivity in clusters larger than 200 voxels (Figure 6.6B). Finally in the direct comparison of methods of localization we observed a main effect of ROI [F(2,30)=4.97; p<0.05] and an interaction between ROI and Hemisphere [F(2,30)=4.56; p<0.05], indicating differing levels of face-selectivity (t-value) across the core system ROIs. Of  particular interest was a main effect of localization method [F(2,30)=12.51; p<0.01]. All other main and interaction effects were not significant. Post-hoc analysis of the main effect of localization method revealed a significant increase in face-selectivity when using a fixed cluster size threshold over using a fixed statistical threshold of p<0.05, 1-tailed Bonferroni (p<0.001), and the method of individually-determined optimal cluster size showing greater face-selectivity than the other two methods (p<0.001, both tests) (Figure 6.7).  Furthermore, both the ROIs of individually-determined cluster size (Mean ? t-value + SEM; 0.46 + 0.06) and the ROIs localized with a fixed cluster size (0.22 + 0.07) showed higher face-selectivity than the peak voxel alone,     161Figure 6.6  Average cluster size with maximal face-selectivity in the core system                                        (A) Average cluster size at which maximal face-selectivity is observed (~50 voxels). A GLM indicates no difference between the average cluster size of all 6 ROIs (bilateral OFA, FFA, and pSTS). All cluster sizes are significantly larger than the peak voxel alone (indicated with an asterisk).  (B) A frequency plot of the cluster size at which maximal face-selectivity is observed reveals the majority of ROIs reaching maximal face selectivity by a cluster size of 75 voxels, with very few requiring clusters of greater than 200 voxels to achieve maximal face-selectivity.     162Figure 6.7  Comparison of different thresholding methods for ROI localization  Face-selectivity comparison of ROIs localized with three different methods (Mean t-value + SEM). Using a fixed statistical threshold is the least effective method of localizing face-selective ROIs. The use of a fixed cluster size localizes ROI which are more face-selective, but the use of individually-based statistics to determine optimal cluster size is the most effective way of ensuring the localization of face-selective ROIs. All differences are significant (p<0.001). The solid bar indicates average face-selectivity of the peak voxel alone. Individually determined cluster size and fixed cluster size methods result in ROIs with face-selectivity greater than the peak voxel alone, while a fixed statistical threshold results in ROIs with reduced face-selectivity with respect to the peak voxel.              163 whereas ROIs localized using a fixed statistical threshold showed, on average, lower face-selectivity than the peak voxel alone (-0.46 + 0.13).    6.4  Discussion We investigated a new functional localizer which contrasts dynamic videos of faces and objects rather than static images of faces and objects, as used in standard localizers of face-related activity (Yovel and Kanwisher, 2005). We showed that this dynamic localizer was able to more consistently identify regions comprising the core system of face perception (i.e.- OFA, FFA, pSTS, see (Haxby, Hoffman et al., 2000)) than a static localizer. In fact, localization of these regions approached 100% efficiency across 16 subjects with the dynamic localizer. ROIs localized by the dynamic localizer were more robust (higher t-value of the peak voxel) and larger (bigger clusters) than those localized by the static localizer.  Within the core system we noted the greatest effects of dynamic localizer within the pSTS, a region that has been more difficult to localize than the FFA in prior studies (Andrews and Ewbank, 2004; Kanwisher, McDermott et al., 1997; Yovel and Kanwisher, 2005). A number of studies have shown activation by biological motion or dynamic stimuli in the pSTS (Pelphrey, Morris et al., 2005; Puce, Allison et al., 1998; Puce and Perrett, 2003; Wheaton, Thompson et al., 2004; Thompson, Hardee et al., 2007). The proximity of this region to the V5 complex (Puce, Allison et al., 1998) may raise concerns that increased activation was related to motion-selective responses. However, our localizer used moving stimuli in both the face and object displays, whereas motion-selective responsivity is usually defined by a contrast between moving and static stimuli (Puce, Allison et al., 1998). Rather, the increased activation of the pSTS by our dynamic localizer may be related to the proposal that the pSTS is particularly sensitive to the dynamic aspects of a face, such as expression (Haxby, Hoffman et al., 2000).   164In addition to the improved localization of regions in the core-system, the dynamic localizer also improved localization within the extended system of face perception (Haxby, Hoffman et al., 2000), such as the middle superior temporal sulcus (Winston, Henson et al., 2004), amygdala (Adolphs, Damasio et al., 1996), inferior frontal gyrus (Ishai, Schmidt et al., 2005), precuneus (Kosaka, Omori et al., 2003), and anterior paracingulate cortex (Gobbini and Haxby, 2007). Due to the inconsistent localization of these regions using standard static localizers (Ishai, Schmidt et al., 2005), studies of these areas often rely on group-based analyses within normalized brains (Winston, Henson et al., 2004). The more consistent localization using dynamic stimuli may allow these regions to be studied in the individual brains. Why is the dynamic localizer so much more consistent in the localization of face-related activity? The simple answer may be that dynamic stimuli are more ecologically valid than static images of faces and objects: empirically, there is evidence that dynamic stimuli activate regions throughout the brain much more strongly than static versions of the same stimuli (Kilts, Egan et al., 2003; Sato, Kochiyama et al., 2004). Also, since face-selective fMRI responses show adaptation to repeated presentation of the same face, one might ask whether these responses might show rapid adaptation to static faces, which are a single unchanging image, than dynamic faces, which continuously change; however, it is not clear whether adaptation of fMRI signal can occur over the short duration of our images (500ms), and the long temporal profile of the hemodynamic response makes this somewhat unlikely. Last, even though our static images contained both neutral and expressive faces, the dynamic stimuli by their nature contained a greater range of facial images. Given the selectivity of some face-responsive neurons to specific views and types of faces (Perrett, Oram et al., 1991), this greater range might activate a larger pool of neurons than the static stimuli.  The contrast between the static and the dynamic localizer also illustrated another issue: variability in ROI cluster size. Many other studies that localize face-selective ROIs employ a   165fixed statistical threshold (Andrews and Ewbank, 2004; Eger, Schyns et al., 2004; Gauthier, Tarr et al., 2000; Golarai, Ghahremani et al., 2007; Golby, Gabrieli et al., 2001; Ishai, Schmidt et al., 2005; Ishai, Ungerleider et al., 2000; Mazard, Schiltz et al., 2006; Pyles, Garcia et al., 2007; Rotshtein, Henson et al., 2005; Schiltz and Rossion, 2006; Yovel and Kanwisher, 2005; Kanwisher, McDermott et al., 1997), a process which has led to wide variability in localized ROI size. In other studies, when the localizer failed to reveal regions consistently the threshold has been manipulated to allow the localization of ROIs in the maximum number of subjects, with the result being widely variable thresholds in the literature (e.g. conservative- (Schiltz and Rossion, 2006); liberal- (Ishai, Schmidt et al., 2005)), a practice that others suggest brings into question the objectivity and validity of the process (Genovese, Lazar et al., 2002). With this variation in statistical threshold comes a variability in ROI cluster size, with some groups reporting very large clusters of face-related activity (Ishai, Schmidt et al., 2005; Sorger, Goebel et al., 2007). Our second goal was to determine if we could define a statistically optimal size of face-related ROIs that was not dependent upon the empiric process of setting thresholds. Large ROIs may be particularly problematic because others have noted that face-selectivity decreases with increased cluster size (Golarai, Ghahremani et al., 2007). We used a statistical method that determined the cluster size with the highest face-selectivity, which we consider a ?statistically optimal? ROI. The results show that an average cluster size of approximately 50mm3 provides maximal face selectivity, and interestingly this is true for all regions of the core system for face perception.  This cluster size of 50mm3 is much smaller than the average size of face-related ROIs reported in the literature, with the normally reported range extending from 150mm3 (Sorger, Goebel et al., 2007) to 5000mm3 (Ishai, Pessoa et al., 2004). In addition, when looking at the maximal face-selectivity within individual ROIs, very few reach their maximal face-selectivity in clusters larger than 200mm3 (Figure 6.6B). At least one group has reported the use of a fixed   166cluster size for localizing face-related regions-of-interest (9 voxels; (Yi, Kelley et al., 2006)) and we argue that such an approach will result in ROIs that are more face-selective than those localized with a threshold based on a fixed t-value (Figure 6.7). Using individually-determined cluster sizes may provide even better face selectivity, but is more time-consuming.  We believe that these results show that localization and definition of face-selective areas can be accomplished both more sensitively and more selectively than with current methods. The dynamic localizer is more sensitive in the detection of face-responsive ROIs in both the core and extended systems, and provides more consistent localization of these regions across individual subjects. While this also increases the number of voxels activated according to traditional statistical threshold methods, it is possible to determine statistically optimal cluster sizes that are not vulnerable to manipulations in threshold, and also more face-selective than ROIs defined by traditional threshold criteria. Greater face-selectivity should enhance investigations of how neural responses in these ROIs vary with adaptation and other stimulus modifications, and better detection of these ROIs in individuals will enhance confidence in the results of fMRI investigations in single case-studies, such as those concerning prosopagnosia.   167 6.5  References ADOLPHS R, DAMASIO H, TRANEL D, and DAMASIO AR. Cortical systems for the recognition of emotion in facial expressions. J Neurosci, 16: 7678-7687, 1996. ADOLPHS R, TRANEL D, DAMASIO H, and DAMASIO A. Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala. Nature, 372: 669-72, 1994. ANDREWS TJ and EWBANK MP. Distinct representations for facial identity and changeable aspects of faces in the human temporal lobe. Neuroimage, 23: 905-13, 2004. AVIDAN G, HASSON U, MALACH R, and BEHRMANN M. Detailed exploration of face-related processing in congenital prosopagnosia: 2. Functional neuroimaging findings. J Cogn Neurosci, 17: 1150-67, 2005. BARTON JJ. Disorders of face perception and recognition. Neurol Clin, 21: 521-48, 2003. BREWER AA, PRESS WA, LOGOTHETIS NK, and WANDELL BA. Visual areas in macaque cortex measured using functional magnetic resonance imaging. J Neurosci, 22: 10416-26, 2002. DAMASIO A. Prosopagnosia. Trends in Neurosci: 132-5, 1985. DE RENZI E. Prosopagnosia in two patients with ct scan evidence of damage confined to the right hemisphere. Neuropsychologia, 24: 385-9, 1986. EGER E, SCHYNS PG, and KLEINSCHMIDT A. Scale invariant adaptation in fusiform face-responsive regions. Neuroimage, 22: 232-42, 2004. FRISTON KJ, ROTSHTEIN P, GENG JJ, STERZER P, and HENSON RN. A critique of functional localisers. Neuroimage, 30: 1077-87, 2006. GAUTHIER I, TARR MJ, MOYLAN J, SKUDLARSKI P, GORE JC, and ANDERSON AW. The fusiform "Face area" Is part of a network that processes faces at the individual level. J Cogn Neurosci, 12: 495-504, 2000. GENOVESE CR, LAZAR NA, and NICHOLS T. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage, 15: 870-8, 2002. GOBBINI MI and HAXBY JV. Neural response to the visual familiarity of faces. Brain Res Bull, 71: 76-82, 2006. GOBBINI MI and HAXBY JV. Neural systems for recognition of familiar faces. Neuropsychologia, 45: 32-41, 2007. GOLARAI G, GHAHREMANI DG, WHITFIELD-GABRIELI S, REISS A, EBERHARDT JL, GABRIELI JD, and GRILL-SPECTOR K. Differential development of high-level visual cortex correlates with category-specific recognition memory. Nat Neurosci, 10: 512-22, 2007. GOLBY AJ, GABRIELI JD, CHIAO JY, and EBERHARDT JL. Differential responses in the fusiform region to same-race and other-race faces. Nat Neurosci, 4: 845-50, 2001. GRILL-SPECTOR K, KNOUF N, and KANWISHER N. The fusiform face area subserves face perception, not generic within-category identification. Nat Neurosci, 7: 555-562, 2004. HAXBY JV, HOFFMAN EA, and GOBBINI MI. The distributed human neural system for face perception. Trends Cogn Sci, 4: 223-233, 2000. ISHAI A, PESSOA L, BIKLE PC, and UNGERLEIDER LG. Repetition suppression of faces is modulated by emotion. Proc Natl Acad Sci U S A, 101: 9827-32, 2004. ISHAI A, SCHMIDT CF, and BOESIGER P. Face perception is mediated by a distributed cortical network. Brain Res Bull, 67: 87-93, 2005. ISHAI A, UNGERLEIDER LG, and HAXBY JV. Distributed neural systems for the generation of visual images. Neuron, 28: 979-90, 2000. KANWISHER N, MCDERMOTT J, and CHUN MM. The fusiform face area: A module in human extrastriate cortex specialized for face perception. J Neurosci, 17: 4302-11, 1997.   168KILTS CD, EGAN G, GIDEON DA, ELY TD, and HOFFMAN JM. Dissociable neural pathways are involved in the recognition of emotion in static and dynamic facial expressions. Neuroimage, 18: 156-68, 2003. KOSAKA H, OMORI M, IIDAKA T, MURATA T, SHIMOYAMA T, OKADA T, SADATO N, YONEKURA Y, and WADA Y. Neural substrates participating in acquisition of facial familiarity: An fmri study. Neuroimage, 20: 1734-42, 2003. LANDIS T, CUMMINGS J, CHRISTEN L, BOGEN J, and IMBOF H-G. Are unilateral right posterior cerebral lesions sufficient to cause prosopagnosia ? Clinical and radiological findings in six additional patients. Cortex, 22: 243-52, 1986. MAZARD A, SCHILTZ C, and ROSSION B. Recovery from adaptation to facial identity is larger for upright than inverted faces in the human occipito-temporal cortex. Neuropsychologia, 44: 912-22, 2006. MCKEOWN MJ and HANLON CA. A post-processing/region of interest (roi) method for discriminating patterns of activity in statistical maps of fmri data. J Neurosci Methods, 135: 137-47, 2004. MEADOWS JC. The anatomical basis of prosopagnosia. J Neurol Neurosurg Psychiatry, 37: 489-501, 1974. NG B, ABUGHARBIEH R, PALMER SJ, and MCKEOWN MJ. Joint spatial denoising and active region of interest delineation in functional magnetic resonance imaging. Conf Proc IEEE Eng Med Biol Soc, 1: 3404-7, 2007. NIETO-CASTANON A, GHOSH SS, TOURVILLE JA, and GUENTHER FH. Region of interest based analysis of functional imaging data. Neuroimage, 19: 1303-16, 2003. PELPHREY KA, MORRIS JP, MICHELICH CR, ALLISON T, and MCCARTHY G. Functional anatomy of biological motion perception in posterior temporal cortex: An fmri study of eye, mouth and hand movements. Cereb Cortex, 15: 1866-76, 2005. PERRETT DI, ORAM MW, HARRIES MH, BEVAN R, HIETANEN JK, BENSON PJ, and THOMAS S. Viewer-centred and object-centred coding of heads in the macaque temporal cortex. Exp Brain Res, 86: 159-73, 1991. PERRETT DI, ROLLS ET, and CAAN W. Visual neurones responsive to faces in the monkey temporal cortex. Experimental Brain Research, 47: 329-342, 1982. PUCE A, ALLISON T, BENTIN S, GORE JC, and MCCARTHY G. Temporal cortex activation in humans viewing eye and mouth movements. J Neurosci, 18: 2188-99, 1998. PUCE A and PERRETT D. Electrophysiology and brain imaging of biological motion. Philos Trans R Soc Lond B Biol Sci, 358: 435-45, 2003. PYLES JA, GARCIA JO, HOFFMAN DD, and GROSSMAN ED. Visual perception and neural correlates of novel 'biological motion'. Vision Res, 47: 2786-97, 2007. REDDY L and KANWISHER N. Category selectivity in the ventral visual pathway confers robustness to clutter and diverted attention. Curr Biol, 17: 2067-72, 2007. RHODES G, BYATT G, MICHIE PT, and PUCE A. Is the fusiform face area specialized for faces, individuation, or expert individuation? J Cogn Neurosci, 16: 189-203, 2004. RICKHAM PP. Human experimentation. Code of ethics of the world medical association. Declaration of helsinki. Br Med J, 2: 177, 1964. ROSSION B, CALDARA R, SEGHIER M, SCHULLER A-M, LAZEYRAS F, and MAYER E. A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing. Brain, 126: 2381-2395, 2003. ROTSHTEIN P, HENSON RN, TREVES A, DRIVER J, and DOLAN RJ. Morphing marilyn into maggie dissociates physical and identity face representations in the brain. Nat Neurosci, 8: 107-13, 2005.   169SATO W, KOCHIYAMA T, YOSHIKAWA S, NAITO E, and MATSUMURA M. Enhanced neural activity in response to dynamic facial expressions of emotion: An fmri study. Brain Res Cogn Brain Res, 20: 81-91, 2004. SAXE R, BRETT M, and KANWISHER N. Divide and conquer: A defense of functional localizers. Neuroimage, 30: 1088-96; discussion 1097-9, 2006. SCHILTZ C and ROSSION B. Faces are represented holistically in the human occipito-temporal cortex. Neuroimage, 32: 1385-94, 2006. SCHILTZ C, SORGER B, CALDARA R, AHMED F, MAYER E, GOEBEL R, and ROSSION B. Impaired face discrimination in acquired prosopagnosia is associated with abnormal response to individual faces in the right middle fusiform gyrus. Cereb Cortex, 16: 574-86, 2006. SCHWARZLOSE RF, BAKER CI, and KANWISHER N. Separate face and body selectivity on the fusiform gyrus. J Neurosci, 25: 11055-9, 2005. SORGER B, GOEBEL R, SCHILTZ C, and ROSSION B. Understanding the functional neuroanatomy of acquired prosopagnosia. Neuroimage, 35: 836-52, 2007. SPIRIDON M and KANWISHER N. How distributed is visual category information in human occipito-temporal cortex? An fmri study. Neuron, 35: 1157-65, 2002. THOMPSON JC, HARDEE JE, PANAYIOTOU A, CREWTHER D, and PUCE A. Common and distinct brain activation to viewing dynamic sequences of face and hand movements. Neuroimage, 37: 966-73, 2007. WHEATON KJ, THOMPSON JC, SYNGENIOTIS A, ABBOTT DF, and PUCE A. Viewing the motion of human body parts activates different regions of premotor, temporal, and parietal cortex. Neuroimage, 22: 277-88, 2004. WINSTON JS, HENSON RN, FINE-GOULDEN MR, and DOLAN RJ. Fmri-adaptation reveals dissociable neural representations of identity and expression in face perception. J Neurophysiol, 92: 1830-9, 2004. WOHLSCHLAGER AM, SPECHT K, LIE C, MOHLBERG H, WOHLSCHLAGER A, BENTE K, PIETRZYK U, STOCKER T, ZILLES K, AMUNTS K, and FINK GR. Linking retinotopic fmri mapping and anatomical probability maps of human occipital areas v1 and v2. Neuroimage, 26: 73-82, 2005. WOJCIULIK E, KANWISHER N, and DRIVER J. Covert visual attention modulates face-specific activity in the human fusiform gyrus: Fmri study. J Neurophysiol, 79: 1574-8, 1998. YI DJ, KELLEY TA, MAROIS R, and CHUN MM. Attentional modulation of repetition attenuation is anatomically dissociable for scenes and faces. Brain Res, 1080: 53-62, 2006. YOVEL G and KANWISHER N. The neural basis of the behavioral face-inversion effect. Curr Biol, 15: 2256-2262, 2005.    1707   PERCEPTUAL DEFICITS IN FACIAL IDENTITY AND EXPRESSION      PROCESSING6    7.1  Introduction As a visual stimulus, the face is a source of multiple types of information, including identity of the person, expression, gaze direction, age, and gender, among others (Barton, 2003; Palermo and Rhodes, 2007; Posamentier and Abdi, 2003). Deriving these different forms of information from a face may involve different types of analyses of the face (Gosselin and Schyns, 2001; Joyce, Schyns et al., 2006), and these in turn may rely on different anatomic substrates. In particular, the perception of facial identity and facial expression are considered strong candidates for independent processing. Identity recognition may require analysis of temporally invariant properties of the face, so that it can be recognized regardless of short term variations in expression and long term variations from aging, whereas expression analysis may require analysis of the dynamic properties of the face, ignoring static structural properties so that expression judgments can be generalized across different individuals (Haxby, Hoffman et al., 2000). Independence of expression and identity processing is a prominent aspect of both current cognitive (Bruce and Young, 1986) and anatomic models (Haxby, Hoffman et al., 2000).  Evidence from both neuroimaging and neuropsychological studies has been used to both support and question this proposed independence of identity and expression processing. Regarding identity, the fusiform face area (FFA), the first major region identified with greater activation by faces than objects (Kanwisher, McDermott et al., 1997) has shown release from adaptation in fMRI studies when the identity of the face changes (Rotshtein, Henson et al., 2005; Winston, Henson et al., 2004; Andrews and Ewbank, 2004), suggesting that the FFA is encoding information related to identity. Whether this is specific for identity and not expression is less                                                  6 A version of this chapter has been submitted for publication. Fox C.J., Hanif H.M., Iaria G., Duchaine B.C., and Barton J.J.S Perceptual deficits in facial identity and expression processing.   171clear, however: while some studies fail to show release from adaptation with changes in expression in the FFA (Winston, Henson et al., 2004), others suggest that the presence of (Vuilleumier and Pourtois, 2007) or attention to (Ganel, Valyear et al., 2005) facial expression modulates activation in the FFA. In the neuropsychological literature, prosopagnosia, a family of disorders that may affect different stages necessary for face identification, has been associated with expression deficits in some patients (Humphreys, Avidan et al., 2007; Stephan, Breen et al., 2006; Sergent and Signoret, 1992), but not others (Duchaine, Parker et al., 2003; McNeil and Warrington, 1991; Takahashi, Kawamura et al., 1995; Tranel, Damasio et al., 1988; Young, Newcombe et al., 1993). While at least some of these studies have linked difficulty with encoding facial structural properties for identity recognition to fusiform gyral damage (Barton 2002), the anatomic correlates of impaired or spared expression processing in prosopagnosia are not known, particularly since many of the older reports predate the use of functional or even structural MRI analysis.   Even less information is available regarding the substrate of independent expression processing. Current models attribute expression perception to the superior temporal sulcus (STS) (Haxby, Hoffman et al., 2000). Adaptation studies have shown that the middle STS (mSTS) exhibits a release from adaptation for changes in expression but not identity, while the posterior STS (pSTS) exhibits a release from adaptation for both identity and expression changes (Winston, Henson et al., 2004). Little is known about the effects of STS damage in humans. One case study demonstrated deficits in the perception of gaze direction after damage to the superior temporal sulcus (Akiyama, Kato et al., 2006): gaze direction, like expression, is a dynamic property of faces and both have been modeled as important functions of the pSTS (Haxby, Hoffman et al., 2000). In other patient populations, deficits in expression perception have been linked to a myriad of lesions, including diffuse cortical atrophy (Kurucz, Soni et al., 1980), right posterior hemispheric lesions (Adolphs, Damasio et al., 1996), left posterior hemispheric lesions   172(Young, Newcombe et al., 1993), and bilateral (Adolphs, Tranel et al., 1994) or unilateral (Brierley, Medford et al., 2004) lesions of the amygdala. Whether these deficits have spared identity processing is not always clear: Young et al (1993) suggested that right-sided lesions could impair both identity and expression processing, and selective deficits for expression processing occurred only with left hemispheric lesions. However, more detailed anatomic analysis was lacking in this report.   One of the criticisms leveled at previous attempts to contrast identity and expression processing is that the different tests used varied in the level of difficulty and in the resources demanded for performance. For example, tests that require naming of identity or expression (Barton, 2003; Kurucz, Feldmar et al., 1979) are highly asymmetric in their requirements, given the potentially infinite number of unique identities versus the limited palette of expressions usually tested (indeed, many suggest that there are only six universal emotions, from which all others are derived (Ekman and Friesen, 1971; Ekman, Sorenson et al., 1969)). Even tests that require matching rather than naming of identities (Benton and van Allen, 1972) or facial expressions (Bowers, Blonder et al., 1991) are subject to the same criticism, so that some have tried to address this concern with tests that have more subtle gradations of expression (Baron-Cohen, Wheelwright et al., 2001). Because of these concerns, the first aim of the present study was to design a perceptual test of equivalent difficulty for both identity and expression, with similar task demands and design for both.  The second aim of the study was to use this experimental test to first determine if dissociations in identity and expression perception could be found in patients, and if so, how these correlated with the locus of their damage, using not only structural MRI but also functional MRI to localize the core components of the face processing network - FFA, pSTS, and occipital face area (OFA) - particularly in the right hemisphere. Our goal was to test the hypothesis that selective damage to the FFA would be associated with a selective deficit of identity processing,   173whereas selective damage to the STS would be associated with a selective deficit of expression processing. In addition, we examined two patients with more anterior temporal damage and prosopagnosia, whose lesions spared all components of the core network, to determine if these subjects showed selective deficits in the perceptual processing of identity. Anterior temporal lesions are usually linked to more associative forms of prosopagnosia, in which the key deficit is associating current perceptual experience with prior facial memories, rather than the impairments in perceiving facial structure seen with apperceptive prosopagnosia, However, the associative/apperceptive distinction is a relative one, and subtler deficits in face perception have been described in such patients ((Barton, Zhao et al., 2003; Delvenne, Seron et al., 2004), but see (Anaki, Kaufman et al., 2007)). If so, it may be that these subjects too might show impairments in our test of identity processing, in which case determining the selectivity of their deficits for identity and not expression is also of interest.    7.2  Experiment 1: Designing the morphed-face discrimination test  7.2.1 Methods   7.2.1.1 Participants Sixteen right-handed healthy participants (8 females; Mean age + SD:  25.8 + 5.8 years) with normal or corrected-to-normal vision and no history of neurological disorders participated. Informed consent was obtained and the protocol approved by the institutional review boards of the University of British Columbia and Vancouver General Hospital, in accordance with The Code of Ethics of the World Medical Association, Declaration of Helsinki (Rickham, 1964).     1747.2.1.2 Stimuli Angry and afraid images for two male identities (BM01, BM28) were selected from the Karolinska Database of Emotional Faces (Lundqvist and Litton, 1998). Background, hair, ears, and neck were removed, while external jaw contour was preserved, using Adobe Photoshop CS2 9.0.2 (www.adobe.com). Distinguishing marks, such as moles, were removed using the Spot Healing Brush Tool. Images were cropped to ensure that faces were centrally located within the image frame, and resized to a standard width of 400 pixels. A morph series of 21 images in 5% morph steps was created between the two angry faces using Abrosoft Fantamorph 3.0 (www.fantamorph.com). This process was repeated for the two afraid faces. Twenty-one morph series were then created between corresponding images from these angry and afraid morph series (i.e. ? Angry1-Afraid1, ?, Angry21-Afraid21), to create a 21X21 morph matrix (441 images total) with axes representing identity and expression. Images included in all versions of the morphed-face discrimination task were selected from this morph matrix.  7.2.1.3 Experimental design The morphed-face discrimination test required participants to choose the different face from a set of three faces. Four versions of this test were created and presented in four separate blocks. The Expression-fixed Identity Task required identity discriminations in the absence of changes in expression, and the Identity-fixed Expression Task required expression discriminations in the absence of changes in identity. The last two versions of this test examined the ability of subjects to process each facial attribute regardless of changes in the other attribute (i.e.- in an invariant fashion). The Expression-variable Identity Task required identity discriminations while ignoring irrelevant changes in expressions, and the Identity-variable Expression Task required expression discriminations while ignoring irrelevant changes in identity (Figure 7.1). The amount by which the different face actually differed from the other two faces varied from 10% to 100% morph    175Figure 7.1  Sample trials from the morphed-face discrimination test                       Sample trials from each of the four versions of the morphed-face discrimination test. (A) Expression-fixed Identity Task; 70% Morph Distance. (B) Identity-fixed Expression Task; 50% Morph Distance. (C) Expression-variable Identity Task; 90% Morph Distance. (D) Identity-variable Expression Task; 70% Morph Distance. The middle face is the correct choice in all examples.    176difference, centered around the 50:50 midpoint of the matrix: thus, the 10% morph difference required discriminations between the 55/45% and 45/55% morph faces, while the 100% morph difference required discriminations between the 100/0% and 0/100% morph faces. Each test version had 12 trials at each of the 10 morph-difference levels, for a total of 120 trials per version, and 480 trials in total.    The irrelevant facial dimension (i.e.- expression during Identity versions, and identity during Expression versions) was held constant within each trial of the ?-fixed? tasks. However, to ensure that participants did not become familiar with any particular face, the level of this irrelevant dimension was randomly varied between trials. During the ?-variable? tasks, the  irrelevant dimension randomly varied, both between the three stimuli on any given trial, and also between trials.    The three faces in each trial were presented in a vertical arrangement at the midline of a black screen (Figure 7.1), to minimize the impact of any hemifield visual defects or horizontal hemineglect in patients. Size was varied across the three faces (3.8?X4.9?, 3.3?X4.2?, 2.8?X3.5?; 56cm viewing distance) to ensure that a direct physical comparison could not be used for a correct response. The arrangement of facial sizes was randomized across trials. The screen location of the target face was balanced, resulting in an equal number of different faces in each of the three possible locations, for each level of morph difference.  7.2.1.4 Procedure Each participant performed all four test versions. In the Identity versions participants were instructed to find the face with the different identity in each set of three faces; in the Expression versions they were asked to find the face with the different expression. They were told to ignore any changes in the size or the irrelevant dimension of the face, and to indicate their selection with a key press as quickly as possible. The four task blocks were presented in random order.   177Short rest breaks were provided between task blocks and appropriate instructions were given prior to each version. Experimental trials were displayed until the participant made their response, and were followed by a black screen for 500ms, which served as the inter-trial-interval. Trials were presented on a 17? widescreen Compaq nx9600 notebook using SuperLab Pro 2.0.4 (www.cedrus.com).  7.2.1.5 Analysis A general linear model (GLM) with test version (Expression-fixed Identity Task, Identity-fixed Expression Task, Expression-variable Identity Task, Identity-variable Expression Task) and morph-difference (10%, 20%, ?, 100%) as fixed factors, subject as a random factor and proportion correct as the dependent measure was performed. A significant effect of test version or an interaction between test version and morph difference would indicate a differing level of difficulty across the different test versions. Linear contrasts were used to examine all significant interactions. Significance was set at ? <0.05 on all statistical tests.  7.2.2 Results The GLM based on the full set of behavioral results revealed a significant main effect of test version [F(3,45)=11.77; p<0.001]. All four test versions differed significantly from each other (p<0.05, all comparisons). Overall, Identity-fixed Expression Task was the easiest version (Mean p correct + SEM; 0.87+0.02), followed by Expression-fixed Identity Task (0.83+0.02), then Identity-variable Expression Task (0.80+0.02), with Expression-variable Identity Task (0.77+0.02) the most difficult. A significant main effect of morph difference was also observed [F(9,135)=224.21, p<0.001], with difficulty decreasing as morph difference was increased. Finally, there was a significant interaction between test version and morph difference [F(27,405)=2.78, p<0.001]. Significant differences between two or more test versions were seen   178at each morph difference except the 20% (p>0.20, all tests) and 70% (p>0.10, all tests) morph differences (Figure 7.2). Thus, this analysis clearly shows significant differences in level of difficulty across the four test versions and, in particular, more difficulty when the irrelevant dimension varied.   To balance the level of difficulty across test versions, the three morph differences located just before the curves reached asymptotic ?ceiling? performance were selected for each version separately (Figure 7.2). Thus, for the Expression-fixed Identity Task we chose the data for the 60-80% morph-difference levels; for the Identity-fixed Expression Task the 40-60% morph-difference levels; for the Expression-variable Identity Task the 80-100% morph-difference levels; and for the Identity-variable Expression Task the 60-80% morph-difference levels). The scores on these three points for each of the test versions in each subject were then compared in a second GLM, with test version (Expression-fixed Identity Task, Identity-fixed Expression Task, Expression-variable Identity Task, Identity-variable Expression Task) and morph difference (1, 2, 3) as fixed factors, subject as a random factor and proportion correct as the dependent measure. A significant main effect of morph difference was observed [F(2,30)=12.00; p<0.001], with the upper morph difference (0.96+0.01) slightly easier than the middle (0.94+0.01) or lower (0.93+0.01) morph differences. Importantly, there was no significant main effect of test version [F(3,45)=1.73, p>0.15] or an interaction between test version and morph difference [F(6.90)=0.92; p>0.40] indicating equivalent level of difficulty across all versions of the test (Expression-fixed Identity Task - 0.95+0.01; Identity-fixed Expression Task ? 0.95+0.01; Expression-variable Identity Task ? 0.93+0.01; Identity-variable Expression Task ? 0.95+0.01). For all test versions, these mean scores of .93-.95 are also sufficiently high (but not at ceiling) and with low variance, optimizing the ability to detect deficits in patients.       179Figure 7.2  Control data from the morphed-face discrimination test   Control data from the full presentation of the four test versions; Mean correct + SD. (A) Expression-fixed Identity Task. (B) Identity-fixed Expression Task. (C) Expression-variable Identity Task. (D) Identity-variable Expression Task. A portion of each task (3 morph differences; within dotted lines) was chosen for patient testing in order to equate difficulty across all four versions of the test. (E) Three morph distances (within dotted lines) were chosen to create balanced versions of the morphed-face discrimination test. Collapsed accuracy + standard deviation are presented for the four balanced versions of the test.       180  7.3  Experiment 2: Identifying deficits of identity and expression perception in patients  7.3.1 Methods  7.3.1.1 Subjects Five patients were included in the current study. Informed consent was obtained and the protocol approved by the institutional review boards of the University of British Columbia and Vancouver General Hospital, in accordance with The Code of Ethics of the World Medical Association, Declaration of Helsinki (Rickham, 1964).  All patients had a neurological and neuro-ophthalmological exam, and participated in a neuropsychological assessment of major cognitive functions (Table 7.1), including visuo-perceptual functioning - Hooper?s Visual Organization Test (Hooper, 1957), mental imagery - mental rotation test (Grossi, 1991), and visuospatial attention - star cancellation task (Wilson, Cockburn et al., 1987) and a visual search task (Spinnler and Tognoni, 1987). Memory was assessed with the Digit Span forward, Spatial Span forward, and Word List immediate recall taken from the Wechsler test (Wechsler, 1999), and with the Words portion of the Warrington Recognition Memory Test (Warrington, 1984). Intelligence was assessed with the Full Scale IQ from the Wechsler Abbreviated Scale of Intelligence (Wechsler, 1999) and with the Trail Making Test, A and B (Reitan, 1958). The role of these tests was to exclude more general cognitive impairments that might impact upon our results.   A series of standard neuropsychological tests were then administered to assess the status of face perception in these five patients. First, the perception of facial identity was assessed with the Benton Facial Recognition Test (Benton and van Allen, 1972), and with the Identity Discrimination portion of the Florida Affect Battery (Bowers, Blonder et al., 1991). Next, the perception of facial expression was assessed with the Affect Discrimination, Affect Naming,    181Table 7.1  Results from the neuropsychological screening  Modality  Test  Max R-IOT1 R-IOT2 R-ST1 B-AT1 R-AT2 Visuo-perceptual Hooper Visual Organization  30  27  19  25.5  20  25 Imagery Mental Rotation 10 10 9 10 10 10 Attention  Star Cancellation  54  54  52  54  54  54   Visual Search  60  54  54  53  59  59 Memory  Digit Span ? Forward   16  12 12 9 12 10   Spatial Span ? Forward 16  9 9 8 10 8  Word List  48 28 30 32 27*  37   Words, WRMT   50  41 49 47 45 41 Intelligence  Full Scale IQ, WASI  160  132  100  114  101  104   Trails A (sec)  -  39  45  37  18  33   Trails B (sec)  -  61  107  99  25  59* Faces - Identity  Benton Facial Recognition 54  45  38*  50 45 41  Identity Discrimination, FAB 20  19 20 18 20 17* Faces ? Expression Affect Discrimination, FAB 20  19  19  17  17  20   Affect Naming, FAB  20  17  17  15  18  20   Affect Selection, FAB   20  19  18  18  20  19   Affect Matching, FAB  20  18  15  15  17  20   Reading the Mind in the Eyes 36  26  28  21  24  19* Faces - Memory Faces, WRMT  50  33* 31*  33* 27* 17*  Famous Face Recognition (d?) 3.92  1.96 2.03 1.96 1.52*? 1.22*   Face Imagery (%)  100  82 86 88 N/A 71* ? - Due to poor knowledge of celebrities, a version of this test using personally familiar faces was given to AT1.  Impairments are indicated in red. (WRMT = Warrington Recognition Memory Test; WASI = Wechsler Abbreviated Scale of Intelligence; FAB = Florida Affect Battery)        182Affect Selection, and Affect Matching portions of the Florida Affect Battery (Bowers, Blonder et al., 1991), and with the revised version of the Reading the Mind in the Eyes Test (Baron-Cohen, Wheelwright et al., 2001). Finally, facial memory was assessed using the Faces portion of the Warrington Recognition Memory Test (Warrington, 1984), a famous face familiarity test (Barton, Cherkasova et al., 2001), and a facial imagery test (Barton and Cherkasova, 2003).   7.3.1.2 Patient descriptions R-IOT1 (R = right; IOT = inferior occipitotemporal) is a 49 year-old left-handed male who, twelve years prior to testing, had an occipital cerebral hemorrhage from rupture of an arterovenous malformation (Figure 7.3). Immediately following this incident, he complained of trouble recognizing hospital workers, and needed to rely on hairstyle, facial hair, or voice for person recognition, a problem which has not resolved. He also has a left superior quadrantanopia (with 20/20 vision in the remaining visual field), and mild topographagnosia (difficulty navigating in new locations). R-IOT1?s self report also indicated the need for letter-by-letter reading immediately following the cerebral hemorrhage, although this had resolved by time of testing. R-IOT1 showed normal performance on all neuropsychological tests, including the Benton Facial Recognition Test; famous face recognition and facial imagery, with the exception of the Faces portion of the Warrington Recognition Memory Test (33/50; Table 7.1). R-IOT2 is a 65 year-old right-handed male who, two and a half years prior to testing, had a right posterior cerebral arterial infarction. This stroke left him with a left superior quadrantanopia not affecting the central 10 degrees, but 20/20 acuity. He does not complain of problems recognizing faces or with topographical orientation, and neurological examination did not show any difficulties in language or color perception. R-IOT2 exhibited normal performance on most neuropsychological tests, but showed impaired performance on the Benton Facial Recognition     183Figure 7.3  Structural MRI of the five patients  Coronal slices of the five patients included in this study (standardized to Talairach space). Slices were taken in 12mm increments from y = +48mm to y = -84mm. In R-IOT1, a single right hemispheric infarct stretches from the occipital pole (-84mm) to the posterior temporal lobe (-48mm). In R-IOT2, a single right hemispheric infarct stretches from the occipital pole (-84mm) to the medial aspect of the temporal lobe (-12mm). In R-ST1, a single right hemispheric infarct stretches from the temporal pole (+24mm) along the superior temporal sulcus to the posterior temporal lobe (-36mm). In B-AT1, bilateral temporal lesions can be seen stretching from the temporal poles (+12mm) to the posterior temporal lobe (-36mm). In R-AT1, a small surgical lesion affecting the right anterior temporal lobe, hippocampus and amygdala can be seen (0mm, -12mm).                     184      185Test (38/54), a test of facial identity perception, and the Faces portion of the Warrington Recognition Memory Test (31/50; Table 7.1).   R-ST1 (ST = superior temporal) is a 57 year-old right-handed male. Four years prior to testing he had a right middle cerebral artery infarction, causing immediate left-sided loss of sensation and paralysis that persisted for only a few days. He still notes clumsiness of the left hand and tripping over the toes of the left foot. He had left hemineglect for 4 months after the stroke.  At present he does not have any problems with language, color perception, topographical orientation, or face recognition, although he does note trouble recognizing voices over the phone. Visual fields are unaffected, and acuity is 20/20 in both eyes. He was normal on all neuropsychological tests except for the Faces portion of the Warrington Recognition Memory Test (33/50). B-AT1 (B = bilateral; AT = anterior temporal) is a 24 year-old right-handed male. Three years prior to testing, he had herpes simplex encephalitis and was comatose. Since recovery, he has noted extreme difficulty in recognizing faces. He has more profound difficulties learning new faces, and can recognize some family members. General memory and mental functioning is unaffected, allowing him to attend college and hold full-time employment. He has mild topographagnosia, and difficulty recalling the names of low-frequency items (although semantic knowledge of these items is evident). He has normal visual fields with 20/20 visual acuity. He performed normally on most neuropsychological tests (Table 7.1). He was severely impaired on the Faces portion of the Warrington Recognition Memory Test (27/50) and did poorly on a modified familiar face recognition test (d?=1.52) that used pictures of his relatives rather than celebrities, due to limited knowledge of the latter (which also invalidated the test of facial imagery). Impaired performance on the Word List immediate recall was also observed (27/48), while performance on all other memory tests, including the Word portion of the Warrington Recognition Memory Test was normal.   186  R-AT1 is a 24 year-old right-handed female. One year prior to testing she had a selective right amygdalohippocampectomy for epilepsy. The surgery was successful, with only one reported seizure in the following year, but she has since noted difficulty recognizing faces, needing to rely on voice or other means to recognize individuals. General mental functioning is intact: she is currently attending university, although she reports problems with visual memory, relying on verbal strategies to study. She has normal visual fields with 20/20 visual acuity. On tests of intelligence, performance was mildly impaired on Trails ? Test B (59 sec), but was normal on Trails ? Test A and the more comprehensive Full Scale IQ test. She was impaired on the Identity Discrimination portion of the Florida Affect Battery (17/20) but showed normal performance on the more difficult discriminations involving changes in lighting and viewpoint on the Benton Facial Recognition Test. For expression, she showed normal performance on all Affect portions of the Florida Affect Battery, but was impaired on the Reading the Mind in the Eyes Test (19/36). She was impaired on the Faces portion of the Warrington Recognition Memory Test (17/50), the famous face recognition test (d?=1.22) and the facial imagery test (71% accuracy).  7.3.1.3 FMRI:  Localizing the face network Structural and functional MRIs were performed on all patients. All scans were acquired in a 3.0 Tesla Philips scanner. Stimuli were presented using Presentation 9.81 software and were rear-projected onto a mirror mounted on the head coil. Whole brain anatomical scans were acquired using a T1-weighted echoplanar imaging (EPI) sequence, consisting of 170 axial slices of 1mm thickness (1mm gap) with an in-plane resolution of 1mm X 1mm (FOV=256). T2-weighted functional scans (TR=2s; TE=30ms) were acquired using an interleaved ascending EPI sequence, consisting of 36 axial slices of 3mm thickness (1mm gap) with an in-plane resolution of 1.875mm X 1.875mm (FOV=240).    187Two functional localizers were used to identify regions comprising the core system of face processing (i.e.- right and left OFA, FFA, and pSTS) (Haxby, Hoffman et al., 2000). The first, a static localizer, presented static photographs of objects (e.g. - television, basketball) and faces (neutral and expressive) in separate blocks (Kanwisher, McDermott et al., 1997; Saxe, Brett et al., 2006). Patients performed an irrelevant ?one-back task?: that is, to press a button if an image was identical to the previous one. The localizer began and ended with a fixation block showing a cross in the centre of an otherwise blank screen. Additional fixation blocks were alternated with image blocks, with all blocks lasting 12 seconds. Six blocks of each image category (object, neutral face, expressive face) were presented in a counterbalanced order. Each image block consisted of 15 images (12 novel and 3 repeated), all sized to a standard width of 400 pixels and presented at screen center for 500ms, with an inter-stimulus-interval of 300ms. The second, a dynamic localizer, presented dynamic videos of objects and faces (Iaria, Fox et al., 2008). Video-clips of faces all displayed dynamic changes in facial expression (i.e.- from neutral to happy). So that dynamic changes in objects were comparable to those seen in faces, all video-clips of objects displayed types of motion that did not create large translations in position. Patients again performed a one-back task. Fixation blocks began and ended the session and were alternated with image blocks, with all blocks lasting 12 seconds. Eight blocks of each image category (object, face) were presented in a counterbalanced order. Each image block consisted of 6 video-clips (5 novel and 1 repeated) presented centrally for 2000ms each. Video-clips of objects were gathered from the internet, and video-clips of faces were provided by Chris Benton (Department of Experimental Psychology, University of Bristol, UK). All video-clips were resized to a width of 400 pixels.    The first volume of each functional scan was discarded to allow for scanner equilibration. All MRI data were analyzed using BrainVoyager QX Version 1.8 (www.brainvoyager.com). Anatomical scans were not preprocessed, but were standardized to Talairach space (Talairach   188and Tournoux, 1988). Preprocessing of functional scans consisted of corrections for slice scan time acquisition, head motion (trilinear interpolation), and temporal filtering with a high pass filter in order to remove frequencies less than 3 cycles/time course. Functional scans were individually co-registered to their respective anatomical scan, using the first retained functional volume to generate the co-registration matrix. The static localizer time course was analyzed with a single subject GLM, with object (O), neutral (NF) and expressive (EF) faces as predictors, and a NF+EF>2*O contrast was overlaid on the whole brain. A similar procedure was adopted for the dynamic localizer, the time course of which was analyzed via a single subject GLM with objects (O) and faces (F) as predictors, and a F>O contrast was overlaid on the whole brain. Within each patient we attempted to define, bilaterally, each of the three face-related regions comprising the core system of face perception (Haxby, Hoffman et al., 2000). Contiguous clusters of face-related voxels located on the lateral temporal portion of the fusiform gyrus were designated as the fusiform face area (FFA), while clusters located on the lateral surface of the inferior occipital gyrus were designated as the occipital face area (OFA). Face-related clusters located on the posterior segment of the superior temporal sulcus were designated as the pSTS. In order to avoid false negatives in the localization of regions-of-interest we employed multiple statistical thresholds within each patient?s analysis (all with a minimum cluster size of 50 voxels). First, a threshold of p<0.05 (1-tailed Bonferroni, corrected for multiple comparisons) was applied to the static localizer. Failure to localize all possible regions-of-interest (excluding regions located in areas of lesion) resulted in lowering this threshold to a more liberal False-Discovery-Rate of q<0.05 (corrected for multiple comparisons). If localization was still unsuccessful this process was repeated, using data from the more robust dynamic localizer (Iaria, Fox et al., 2008). While the static localizer, with a 72% success rate (Bonferroni threshold), has a significant potential for false negatives the dynamic localizer operates at a 98% success rate (Bonferroni threshold) dramatically reducing the   189likelihood of a false negative (Iaria, Fox et al., 2008). The most conservative threshold that identified all possible regions-of-interest was used to report cluster values in that particular patient.  7.3.1.4 Morphed-face discrimination test Finally, all patients participated in the four truncated versions of the morphed-face discrimination test (see Methods above). Test versions were presented in blocks, separated by a short rest break between each block. The order of versions was determined randomly. Results from each test version were analyzed separately, although due to the lack of interaction between test version and morph difference (see Results above), accuracy was collapsed across morph difference within each test version. This resulted in a single accuracy score for each of the four versions of the test, in each patient. The 95% prediction interval for each test version was calculated from control data using the following formula:  PI95 = X ? t.05(SD/? ((n+1)/n) where X represents the mean performance, t.05 the one-tailed t value with a significance of p<0.05, SD the standard deviation, and n the number of participants. Patient scores which fell below the 95% prediction interval (PI) were held to indicate impaired processing. To provide some measure of the magnitude of impairment, the 99%, 99.9%, and 99.99% prediction intervals were also calculated.   7.3.2 Results R-IOT1 has a unilateral right lesion primarily involving the occipital lobe, but also affecting the posterior portion of the right inferior temporal lobe (Figure 7.3). This lesion affects the general region of the right OFA and right FFA, and the functional localizer confirmed this, failing to show activation of the right OFA or FFA at any statistical threshold. However, the OFA and FFA   190were identified in the left hemisphere, and the pSTS was identified bilaterally (Table 7.2, Figure 7.4). On our morphed-face discrimination test, R-IOT1 was impaired on both Identity versions of the test, with performance on the Expression-fixed Identity Task version (Accuracy = 0.83) falling below the 95% PI, and performance on the Expression-variable Identity Task version (0.72) falling below the 99% PI (Figure 7.5). In contrast R-IOT1 was normal on both the Identity-fixed Expression Task (0.92) and Identity-variable Expression Task (0.89).   R-IOT2 has a medial occipitotemporal lesion, stretching from the occipital pole along the medial surface of the occipital lobe and ending in the middle portion of the inferotemporal cortex (Figure 7.3). Using fMRI we were able to localize all six regions comprising the core system of face processing, with the right OFA and right FFA being located just lateral to the region of infarct, in spared cortex. (Table 7.2, Figure 7.4). R-IOT2 was impaired on both Identity versions of test, with performance on the Expression-fixed Identity Task (0.78) and the Expression-variable Identity Task (0.72) falling below the 99% PI (Figure 7.5). In contrast he performed normally on both the Identity-fixed Expression Task (0.94) and the Identity-variable Expression Task (0.92).    R-ST1 has a large right hemispheric lesion, which extends from the right anterior temporal pole, along the superior temporal sulcus, to the posterior temporal lobe. More extensive damage is visible towards the anterior temporal lobe, although the right amygdala is spared   (Figure 7.3 - 0mm). FMRI showed activation in the right FFA and right OFA, but not the right pSTS (Figure 7.4), consistent with the fact that his lesion involved the posterior part of the superior temporal sulcus. All core regions were identified in the left hemisphere. On the morphed-face discrimination test, R-ST1 was impaired on both Expression versions of the test, with performance on the Identity-fixed Expression Task (0.78) falling below the 99% PI and performance on the Identity-variable Expression Task (0.83) falling below the 95% PI. In contrast, he performed normally on the Expression-fixed Identity Task (0.89) (Figure 7.5).    191Table 7.2  Core face network localized in the five patients  Subject  Localizer  Threshold  Region  Peak t-value  Cluster size (voxels) X  Y  Z R-IOT1 Dynamic  FDR   ROFA LESION            RFFA LESION            RpSTS 5.52  123 57 -40 13       LOFA  4.98  51  -36 -79 -14       LFFA  6.71  281  -33 -67 -23    LpSTS 6.32 785 -57 -28 -2 R-IOT2  Dynamic  FDR   ROFA  4.80  182  30  -85  -17       RFFA  6.33  606  33  -40  -23       RpSTS  9.37  1074  45  -25  -5       LOFA  3.92  204  -33  -64  -20       LFFA  5.95  168  -42  -49  -32       LpSTS  5.64  517  -51  -25  -5 R-ST1 Dynamic  FDR   ROFA 7.49  1001 27 -82 -11      RFFA 9.36  738 33 -49 -17      RpSTS LESION          LOFA 6.50 828 -39 -85 -2       LFFA  4.69  144  -42 -64 -14    LpSTS 8.10 1497 -48 -46 1 B-AT1  Dynamic  BF   ROFA  12.37  3956  30  -88  -5       RFFA  13.09  1064  39  -52  -20       RpSTS  9.67  329  46  -49  -2       LOFA  9.43  1543  -30  -85  -8       LFFA  5.96  57  -39  -55  -26       LpSTS  5.90  50  -60  -46  4 R-AT1 Static  BF  ROFA 10.32  470 30 -67 -17      RFFA 10.42  227 36 -58 -14      RpSTS 8.14  240 42 -40 4    LOFA 12.48 648 -39 -70 -8       LFFA  12.35  574  -36 -49 -14    LpSTS 6.27 149 -60 -55 1  Results of the functional localizers, in brains standardized to Talairach space. (FDR ?False Discovery Rate, q<0.05; BF ? 1-tailed Bonferroni, p<0.05)        192Figure 7.4  FMRI images of the core face network as localized in the five patients    Core system regions-of-interest identified with the functional localizers (all brains standardized to Talairach space). Due to the location of the lesion, R-IOT1 does not display a right OFA or right FFA. However a right pSTS was identified along with all three core regions in the left hemisphere. All six core regions were identified in R-IOT2, with the right OFA and FFA located just lateral to the lesion. R-ST1 showed all regions of the core system except the right pSTS which would have been located within the region of damage. All six regions of the core system were identified in B-AT1 and R-AT1.      193Figure 7.5  Results of the morphed-face discrimination test for the five patients  In all figures the solid circle represents the control data (Mean + SD), the solid triangle represents R-IOT1, the open triangle R-IOT2, the solid diamond R-ST1, the solid square B-AT1, and the open square R-AT1. Solid lines represent 95% prediction intervals for the respective versions of the test. (A) Comparison of performance on the Expression-fixed Identity Task (x-axis) and Identity-fixed Expression Task (y-axis) versions of the test. R-IOT1, R-IOT2, B-AT1, and R-AT1 are all impaired on Expression-fixed Identity Task, but show normal performance on Identity-fixed Expression Task. R-ST1 shows the opposite performance pattern. (B) Comparison of performance on the Expression-fixed Identity Task (x-axis) and Expression-variable Identity Task (y-axis) versions of the test. R-IOT1, R-IOT2, B-AT1, and R-AT1 are all impaired on both Identity versions of the test. R-ST1 is impaired on the Expression-variable Identity Task version only. (C) Comparison of performance on the Identity-fixed Expression Task (x-axis) and Identity-variable Expression Task (y-axis) versions of the test. R-IOT1, R-IOT2, B-AT1, and R-AT1 all show normal performance on both Expression versions of the test. R-ST1, in contrast, is impaired on both Expression versions of the test.                 194                           195However, performance on the Expression-variable Identity Task (0.67) was also impaired in R-ST1 (0.67, <99.9% PI).   B-AT1 has extensive bilateral damage to the temporal lobes, with damage extending from the anterior poles to the middle and inferior temporal lobes (Figure 7.3). Functional MRI revealed intact activation of the FFA, OFA and pSTS in both hemispheres (Table 7.2, Figure 7.4). Performance on our morphed-face discrimination task revealed a pattern similar to that seen in R-IOT1 and R-IOT2 (Figure 7.5). B-AT1 was impaired on both Identity versions of the test, with performance on both the Expression-fixed Identity Task (0.72) and the Expression-variable Identity Task (0.67) versions falling below the 99.9% PI. In contrast performance on both the Identity-fixed Expression Task (1.00) and the Identity-variable Expression Task (0.97) was normal.   R-AT1 has a small lesion in the anterior right temporal lobe, affecting the temporal cortex, hippocampus and amygdala (Figure 7.3). Functional MRI localized the FFA, OFA and pSTS in both hemispheres (Table 7.2, Figure 7.4). Our morphed-face discrimination test revealed severe impairments on both Identity versions of the test, with performance on both the Expression-fixed Identity Task (0.56) and the Expression-variable Identity Task (0.50) versions of the test falling below the 99.99% PI. In contrast, performance on both the Identity-fixed Expression Task (0.92) and the Identity-variable Expression Task (0.89) versions of the morphed-face discrimination task was unimpaired.     7.4  Discussion Our first goal was to design a perceptual test that did not require a verbal description of identity or expression for the response, with low variance and non-ceiling performance to increase our chances of finding subtle deficits, and importantly, with equivalent difficulty for both identity and expression discriminations. The use of verbal labels creates an imbalance between   196expression and identity, since recognizing identity requires distinction between numerous unique identities, whereas naming expression may only require knowledge of a small set of possibilities (Bowers, Blonder et al., 1991), as some suggest that all facial expressions are only derivatives of six universal facial expressions (Ekman and Friesen, 1971; Ekman, Sorenson et al., 1969). Removing a requirement for verbal labeling also reduces the impact on the data of any disturbance of semantic associations for expression or identity: our subjects merely had to indicate which face was different, stressing the perceptual aspect of the test. Last, as our initial results for the morphed-face discrimination test showed that Expression versions of the test were easier to perform than Identity versions, we equilibrated the test versions for perceptual equivalency, so that any dissociation in performance could not be attributed to variations in task difficulty.    The main goal of this paper was to use the morphed-face discrimination test to determine if dissociations between the perception of identity and expression existed in patients and how such patterns correlated with the functional anatomy of their lesions. We first examined two patients with damage restricted to the right inferior occipitotemporal cortex (R-IOT1 and R-IOT2), both of whom had some difficulties on standard tests of identity processing, but not on tests of expression perception. Our fMRI study showed a large lesion of the lateral inferior occipitotemporal cortex in R-IOT1 which affected both the right OFA and FFA, while sparing the right pSTS (Figure 7.4). In contrast, fMRI of R-IOT2 demonstrated a large lesion of the medial inferior occipitotemporal cortex which spared all three core regions in the right hemisphere, with the right OFA and FFA located just lateral to the lesion (Figure 7.4). Interestingly, on the morphed-face discrimination test, both patients showed impaired identity perception and intact expression perception (Figure 7.5). This dissociation in perceptual deficits is consistent with prior reports of acquired (Duchaine, Parker et al., 2003; McNeil and Warrington, 1991; Takahashi, Kawamura et al., 1995; Tranel, Damasio et al., 1988; Young,   197Newcombe et al., 1993) and congenital prosopagnosia (Duchaine, Parker et al., 2003) (although there are other reports of prosopagnosic patients with impaired perception of expression - (Humphreys, Avidan et al., 2007; Sergent and Signoret, 1992; Stephan, Breen et al., 2006)); however, these reports did not use comparable tests for identity and expression processing and the anatomic detail of their lesions is limited, particularly because fMRI has not been used to study such patients until recently.    Using fMRI in these patients we hoped to correlate perceptual performance with damage to core regions of face processing (i.e.- OFA, FFA, or pSTS). Damage to the fusiform gyri and inferior occipitotemporal cortex has been identified as a critical factor in impairing the perception of facial structure in apperceptive prosopagnosia, which causes subjects to fail to recognize familiar faces (Barton, Press et al., 2002). While the presence of the FFA at this critical location makes it a tantalizing candidate for prosopagnosia, there is evidence for sparing of this functional region in prosopagnosia (Rossion, Caldara et al., 2003). In this patient, damage was observed to the right OFA and left FFA, leading the authors to conclude that a network of face areas is necessary for normal face perception (Rossion, Caldara et al., 2003). Our data tend to support this conclusion. Both R-IOT1 and R-IOT2 show selective impairments in identity processing, yet only R-IOT1 has a damaged right OFA and right FFA, with the lesion in R-IOT2 too medial to affect these regions. In fact, the full face network appears to be spared in R-IOT2. What is the source of his impairments in identity processing? One possibility is that the medial lesion in R-IOT2 affected white matter tracts, such as the inferior longitudinal fasciculus which may be important for communication between the right OFA and FFA or with their left hemispheric counterparts (Catani, Jones et al., 2003; Habib, 1986; Takahashi, Kawamura et al., 1995; Fox, Iaria et al., 2008a). Another possibility is that of a distributed representation for faces in the inferior occipitotemporal cortex as proposed by Haxby et al (2001). Such a representation is comprised of highly selective face regions, and other non-maximal face regions that maintain   198face-responsivity without selectivity, which both contribute to face processing (Haxby, Gobbini et al., 2001). Thus damage to non-maximal face regions (i.e.- not the OFA, FFA, or pSTS) could still disrupt the distributed face representation and thereby impair face processing.   In contrast to these two patients with inferior occipitotemporal damage is R-ST1, who has  a large right-sided lesion involving the superior temporal sulcus and who does not overtly complain of problems in face recognition. Structural and functional MRI showed sparing of the inferior occipitotemporal cortex including the right OFA and FFA, but with damage to the superior temporal sulcus including the face-selective pSTS. R-ST1 was severely impaired on both the Identity-variable and Identity-fixed Expression Tasks, but normal on the Expression-fixed Identity Task, opposite to R-IOT1 and R-IOT2 (Figure 7.5A). However, his perception of identity changes became worse when expression was allowed to vary in the stimuli.    Compared to reports on prosopagnosia, there are far fewer studies of deficits in face expression processing within the literature. Expression deficits in prior reports have been attributed to diffuse bilateral damage (Kurucz, Soni et al., 1980), to right (Adolphs, Damasio et al., 1996) or left (Young, Newcombe et al., 1993) hemisphere lesions, or even selective amygdala damage (Adolphs, Tranel et al., 1994; Brierley, Medford et al., 2004). In a lesion overlap study of patients with deficits in expression recognition a clear right hemisphere bias was demonstrated, with the most common site of lesion being the right temporoparietal junction, in the vicinity of the pSTS (Adolphs, Damasio et al., 1996). In contrast to this study, another large patient series demonstrated selective impairments of expression perception following left hemisphere trauma only (Young, Newcombe et al., 1993). Selective expression impairments were defined as poor performance on expression naming and expression matching tasks, but spared familiar face recognition and unfamiliar face matching (Young, Newcombe et al., 1993). Patients with right hemisphere damage also showed impairments in expression processing, although deficits were preferentially seen in the expression matching task than the expression   199naming task, and were sometimes associated with impairments in familiar face recognition or unfamiliar face matching (Young, Newcombe et al., 1993). Another study, attempting to distinguish between these two aspects of expression perception suggest that amygdala damage leads to impairments of emotion memory, not emotion perception (Brierley, Medford et al., 2004). R-ST1 exhibits the opposite pattern to this patient with amygdala damage, with normal emotion naming and memory as indicated by normal performance on tests of expression naming (Table 7.1), but impaired emotion-matching on the morphed-face discrimination task, indicating the perceptual nature of his problem. R-ST1?s lesion does not damage the amygdala, but extends along the superior temporal sulcus to the pSTS region, which no longer activates on fMRI (Figures 7.3 and 7.4), and thereby supports suggestions that the right STS is involved in the perceptual processing of facial expression (Young, Newcombe et al., 1993). Like previous reports of amygdala damage (Brierley, Medford et al., 2004) the reverse pattern of impaired emotion memory with spared emotion perception was observed in R-AT1, a patient with unilateral right amygdala damage but a spared right pSTS. She performed normally on both Expression versions of the morphed-face discrimination test (emotion perception), but was impaired on the Reading the Mind in the Eyes Test (emotion memory).  R-ST1?s impaired performance on the Expression-variable Identity Task also suggests that the STS region may make a contribution to expression-invariant identity processing. This contribution may be indirect, in that failure to recognize changes in a face as attributable to variations in expression may interfere with the ability to discount these when attempting to match faces for identity. On the other hand, recent fMRI studies show that the right pSTS is sensitive to changes in either facial identity or expression (Winston, Henson et al., 2004; Fox, Iaria et al., 2008b). Therefore an alternative interpretation is that the pSTS is required for tasks that require the integrated analyses of both facial identity and expression. One possible   200interpretation of these data is that impairments in processing expression may interfere with the ability to discount variations in expression when trying to process identity.   Besides R-IOT1, R-IOT2 and R-ST1, we also studied two cases of prosopagnosia resulting from anterior temporal damage, one with extensive bilateral lesions (B-AT1) and another with a right amygdalohippocampectomy (R-AT1), in both of whom the FFA, OFA and STS could be demonstrated bilaterally. Both B-AT1 and R-AT1 showed impaired performance on tests that involve facial memory, such as the Warrington Recognition Memory Test, Famous Face Familiarity Test and the Face Imagery test in R-AT1, but did well on a perceptual test of face matching (Benton Face Recognition Test; Table 7.1), results consistent with a diagnosis of associative rather than apperceptive prosopagnosia, and with the anterior temporal loci of their damage. However, the morphed-face discrimination test revealed that both B-AT1 and R-AT1 were impaired in identity discriminations but not expression discriminations, similar to R-IOT1 and R-IOT2, despite their normal perceptual matching performance. This suggests that the morphed-face discrimination test may be more sensitive to subtler failures in perceiving facial structure. Thus, while the anterior temporal lobes have usually been assigned roles in determining familiarity or linking faces with names or semantic associations (Douville, Woodard et al., 2005; Gobbini and Haxby, 2007; Haxby, Hoffman et al., 2000; Snowden, Thompson et al., 2004; Tsukiura, Fujii et al., 2002; Tsukiura, Mochizuki-Kawai et al., 2006; Glosser, Salvucci et al., 2003), they may also make a contribution to perceptual processes. The associative/apperceptive dichotomy in prosopagnosia is likely a relative rather than an absolute one, as Lissauer (1890) himself contended for agnosia in general, and prior reports have suggested that while patients with associative prosopagnosia may not have the severe deficits in perceiving facial configuration that are present in apperceptive prosopagnosia, they may nevertheless have more subtle deficits in integrating this information (Barton, Zhao et al., 2003; Delvenne, Seron et al.,   2012004). A test using morphed faces may be particularly sensitive to such integrative deficits, since morphing introduces changes across the whole face, all of which may contribute to performance at non-ceiling levels. Furthermore, our results show that such perceptual deficits related to anterior temporal damage remain selective for identity and not expression, similar to the findings in patients with posterior occipitotemporal damage. As in R-IOT2, this anterior temporal damage, while not affecting the core region of the face network, may disrupt face processing through damaging white matter tracts connecting the face network or non-maximal portions of distributed face representations.    In conclusion, by using a non-verbal perceptual test of identity and expression discrimination, matched for level of perceptual difficulty, we showed that impairments in these two functions are dissociable. Selective impairments in discriminating identity can occur after anterior temporal lesions or right inferior occipitotemporal lesions that may or may not affect the OFA and FFA, while selective impairments in discriminating expression can occur with damage to the right superior temporal sulcus that affects the pSTS. These patients provide important lesion data to complement the functional neuroimaging work upon which current neuroanatomic models of face processing are based.    202  7.5  References ADOLPHS R, DAMASIO H, TRANEL D, and DAMASIO AR. Cortical systems for the recognition of emotion in facial expressions. J Neurosci, 16: 7678-7687, 1996. ADOLPHS R, TRANEL D, DAMASIO H, and DAMASIO A. Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala. Nature, 372: 669-72, 1994. AKIYAMA T, KATO M, MURAMATSU T, SAITO F, NAKACHI R, and KASHIMA H. A deficit in discriminating gaze direction in a case with right superior temporal gyrus lesion. Neuropsychologia, 44: 161-70, 2006. ANAKI D, KAUFMAN Y, FREEDMAN M, and MOSCOVITCH M. Associative (prosop)agnosia without (apparent) perceptual deficits: A case-study. Neuropsychologia, 45: 1658-71, 2007. ANDREWS TJ and EWBANK MP. Distinct representations for facial identity and changeable aspects of faces in the human temporal lobe. Neuroimage, 23: 905-13, 2004. BARON-COHEN S, WHEELWRIGHT S, HILL J, RASTE Y, and PLUMB I. The 'reading the mind in the eyes' test revised version: A study with normal adults, and adults with asperger syndrome or high-funtioning autism. Journal of Child Psychiatry and Psychiatry, 42: 241-252, 2001. BARTON J, CHERKASOVA M, and O'CONNOR M. Covert recognition in acquired and developmental prosopagnosia. Neurology, 57: 1161-7, 2001. BARTON JJ. Disorders of face perception and recognition. Neurol Clin, 21: 521-48, 2003. BARTON JJ and CHERKASOVA M. Face imagery and its relation to perception and covert recognition in prosopagnosia. Neurology, 61: 220-5, 2003. BARTON JJ, ZHAO J, and KEENAN JP. Perception of global facial geometry in the inversion effect and prosopagnosia. Neuropsychologia, 41: 1703-11, 2003. BARTON JJS, PRESS DZ, KEENAN JP, and O'CONNOR M. Lesions of the fusiform face area impair perception of facial configuration in prosopagnosia. Neurology, 58: 71-78, 2002. BENTON A and VAN ALLEN M. Prosopagnosia and facial discrimination. Journal of the Neurological Sciences, 15: 167-172, 1972. BOWERS D, BLONDER LX, and HEILMAN KM. Florida affect battery. Secondary Titl.: University of Florida, 1991. BRIERLEY B, MEDFORD N, SHAW P, and DAVID AS. Emotional memory and perception in temporal lobectomy patients with amygdala damage. J Neurol Neurosurg Psychiatry, 75: 593-9, 2004. BRUCE V and YOUNG A. Understanding face recognition. Br J Psychol, 77 ( Pt 3): 305-27, 1986. CATANI M, JONES DK, DONATO R, and FFYTCHE DH. Occipito-temporal connections in the human brain. Brain, 126: 2093-107, 2003. DELVENNE JF, SERON X, COYETTE F, and ROSSION B. Evidence for perceptual deficits in associative visual (prosop)agnosia: A single-case study. Neuropsychologia, 42: 597-612, 2004. DOUVILLE K, WOODARD JL, SEIDENBERG M, MILLER SK, LEVERONI CL, NIELSON KA, FRANCZAK M, ANTUONO P, and RAO SM. Medial temporal lobe activity for recognition of recent and remote famous names: An event-related fmri study. Neuropsychologia, 43: 693-703, 2005. DUCHAINE BC, PARKER H, and NAKAYAMA K. Normal recognition of emotion in a prosopagnosic. Perception, 32: 827-38, 2003. EKMAN P and FRIESEN WV. Constants across cultures in the face and emotion. J Pers Soc Psychol, 17: 124-9, 1971.   203EKMAN P, SORENSON ER, and FRIESEN WV. Pan-cultural elements in facial displays of emotion. Science, 164: 86-8, 1969. FOX CJ, IARIA G, and BARTON JJ. Disconnection in prosopagnosia and face processing. Cortex, (in press), 2008a. FOX CJ, IARIA G, and BARTON JJ. Fusiform face area sensitivity to facial identity categories is modulated by the perceptual task. Secondary Titl. San Francisco, CA, USA, 2008b. GANEL T, VALYEAR KF, GOSHEN-GOTTSTEIN Y, and GOODALE MA. The involvement of the "Fusiform face area" In processing facial expression. Neuropsychologia, 43: 1645-1654, 2005. GLOSSER G, SALVUCCI AE, and CHIARAVALLOTI ND. Naming and recognizing famous faces in temporal lobe epilepsy. Neurology, 61(1):: 81-6, 2003. GOBBINI MI and HAXBY JV. Neural systems for recognition of familiar faces. Neuropsychologia, 45: 32-41, 2007. GOSSELIN F and SCHYNS PG. Bubbles: A technique to reveal the use of information in recognition tasks. Vision Res, 41: 2261-2271, 2001. GROSSI D. La riabilitazione dei disordini della cognizione spaziale. Milano: Masson, 1991. HABIB M. Visual hypoemotionality and prosopagnosia associated with right temporal lobe isolation. Neuropsychologia, 24: 577-82, 1986. HAXBY JV, GOBBINI MI, FUREY ML, ISHAI A, SCHOUTEN JL, and PIETRINI P. Distributed and overlapping representations of faces and objects in the ventral temporal cortex. Science, 293: 2425-2430, 2001. HAXBY JV, HOFFMAN EA, and GOBBINI MI. The distributed human neural system for face perception. Trends Cogn Sci, 4: 223-233, 2000. HOOPER HE. Hooper visual organization test. Secondary Titl. Los Angeles: Western Psychological Services, 1957. HUMPHREYS K, AVIDAN G, and BEHRMANN M. A detailed investigation of facial expression processing in congenital prosopagnosia as compared to acquired prosopagnosia. Exp Brain Res, 176: 356-373, 2007. IARIA G, FOX CJ, and BARTON JJS. Dynamic versus static stimuli for the localization of the cerebral areas involved in face perception. Secondary Titl. Naples, Florida, 2008. JOYCE CA, SCHYNS PG, GOSSELIN F, COTTRELL GW, and ROSSION B. Early selection of diagnostic facial information in the human visual cortex. Vision Res, 46: 800-813, 2006. KANWISHER N, MCDERMOTT J, and CHUN MM. The fusiform face area: A module in human extrastriate cortex specialized for face perception. J Neurosci, 17: 4302-11, 1997. KURUCZ J, FELDMAR G, and WERNER W. Prosopo-affective agnosia associated with chronic organic brain syndrome. J Am Geriatr Soc, 27: 91-5, 1979. KURUCZ J, SONI A, FELDMAR G, and SLADE W. Prosopo-affective agnosia and computerized tomography findings in patients with cerebral disorders. Journal of the American Geriatrics Society, 28: 475-478, 1980. LISSAUER H. Ein fall von seelenblindheit nebst einim beitrage zur theori derselben. Archiv fur Psychiatrie und Nervenkrankheiten, 21: 222-270., 1890. LUNDQVIST D and LITTON JE. The averaged karolinska directed emotional faces-akdef, 1998. MCNEIL JE and WARRINGTON EK. Prosopagnosia: A reclassification. Q J Exp Psychol A, 43: 267-87, 1991. PALERMO R and RHODES G. Are you always on my mind? A review of how face perception and attention interact. Neuropsychologia, 45: 75-92, 2007. POSAMENTIER MT and ABDI H. Processing faces and facial expressions. Neuropsychology Review, 13: 113-143, 2003.   204REITAN RM. Validity of the trail making test as an indicator of organic brain damage. Perceptual and Motor Skills, 8: 271-276, 1958. RICKHAM PP. Human experimentation. Code of ethics of the world medical association. Declaration of helsinki. Br Med J, 2: 177, 1964. ROSSION B, CALDARA R, SEGHIER M, SCHULLER A-M, LAZEYRAS F, and MAYER E. A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing. Brain, 126: 2381-2395, 2003. ROTSHTEIN P, HENSON RN, TREVES A, DRIVER J, and DOLAN RJ. Morphing marilyn into maggie dissociates physical and identity face representations in the brain. Nat Neurosci, 8: 107-13, 2005. SAXE R, BRETT M, and KANWISHER N. Divide and conquer: A defense of functional localizers. Neuroimage, 30: 1088-96; discussion 1097-9, 2006. SERGENT J and SIGNORET JL. Varieties of functional deficits in prosopagnosia. Cereb Cortex, 2: 375-88, 1992. SNOWDEN JS, THOMPSON JC, and NEARY D. Knowledge of famous faces and names in semantic dementia. Brain, 127: 860-72, 2004. SPINNLER H and TOGNONI G. Standardizzazione e taratura italiana di test neuropsicologici. Milano: Ital J Neurol Sci, 1987. STEPHAN BC, BREEN N, and CAINE D. The recognition of emotional expression in prosopagnosia: Decoding whole and part faces. J Int Neuropsychol Soc, 12: 884-95, 2006. TAKAHASHI N, KAWAMURA M, HIRAYAMA K, SHIOTA J, and ISONO O. Prosopagnosia: A clinical and anatomical study of four patients. Cortex, 31: 317-29, 1995. TALAIRACH J and TOURNOUX P. Co-planar stereotaxic atlas of the human brain. New York: Thieme, 1988. TRANEL D, DAMASIO AR, and DAMASIO H. Intact recognition of facial expression, gender, and age in patients with impaired recognition of face identity. Neurology, 38: 690-696., 1988. TSUKIURA T, FUJII T, FUKATSU R, OTSUKI T, OKUDA J, UMETSU A, SUZUKI K, TABUCHI M, YANAGAWA I, NAGASAKA T, KAWASHIMA R, FUKUDA H, TAKAHASHI S, and YAMADORI A. Neural basis of the retrieval of people's names: Evidence from brain-damaged patients and fmri. J Cogn Neurosci, 14: 922-37, 2002. TSUKIURA T, MOCHIZUKI-KAWAI H, and FUJII T. Dissociable roles of the bilateral anterior temporal lobe in face-name associations: An event-related fmri study. Neuroimage, 30: 617-26, 2006. VUILLEUMIER P and POURTOIS G. Distributed and interactive brain mechanisms during emotion face perception: Evidence from functional neuroimaging. Neuropsychologia, 45: 174-94, 2007. WARRINGTON E. Warrington recognition memory test. Secondary Titl. Los Angeles: Western Psychological Services, 1984. WECHSLER D. Wechsler abbreviated scale of intelligence?manual. San Antonio: Psychological Corp, 1999. WILSON B, COCKBURN J, and HALLIGAN PW. Behavioural inattention test. Titchfield: Thames Valley Test Company, 1987. WINSTON JS, HENSON RN, FINE-GOULDEN MR, and DOLAN RJ. Fmri-adaptation reveals dissociable neural representations of identity and expression in face perception. J Neurophysiol, 92: 1830-9, 2004. YOUNG AW, NEWCOMBE F, DE HAAN EH, SMALL M, and HAY DC. Face perception after brain injury. Selective impairments affecting identity and expression. Brain, 116 ( Pt 4): 941-59, 1993.   2058   RESIDUAL SENSITIVITY TO IDENTITY AND EXPRESSION CHANGES IN      ACQUIRED PROSOPAGNOSIA7    8.1  Introduction Prosopagnosia is a neurological disorder which impairs facial recognition (Barton, 2003). The acquired form of this disorder is most commonly associated with damage to the right inferior occipitotemporal cortex (Barton, 2003; Bodamer, 1947; Landis, Cummings et al., 1986). Functional magnetic resonance imaging (fMRI), has revealed a region of cortex which responds most strongly to the presentation of faces than objects at this location, the fusiform face area or FFA (Kanwisher, McDermott et al., 1997), and suggests a possible link between FFA damage and prosopagnosia. However, besides the numerous cases of prosopagnosia which note damage to this region, there are also some which specifically note the sparing of the FFA (Rossion, Caldara et al., 2003). In addition, a congenital form of prosopagnosia presents without any visible neurological damage (Duchaine, Parker et al., 2003), or functional damage to the FFA (Avidan, Hasson et al., 2005).    Besides the FFA, are a number of other regions that respond more strongly to faces than objects in fMRI (Haxby, Hoffman et al., 2000; Ishai, Schmidt et al., 2005; Kanwisher, McDermott et al., 1997). Of these, two other regions have been modeled, along with the FFA, as the core system for face processing; the occipital face area (OFA) located on the inferior occipital gyrus, and a face-selective portion of the posterior superior temporal sulcus (pSTS) (Haxby, Hoffman et al., 2000). It may be that damage to these other regions also results in prosopagnosia, as suggested by Rossion et al. (2003) who present a prosopagnosic with damage to the right OFA and left FFA, and sparing of other core face regions.                                                   7 A version of this chapter has been submitted for publication. Fox C.J., Iaria G., Duchaine B.C., and Barton J.J.S. An fMRI adaptation study of the residual sensitivity to the identity and expression of faces in acquired prosopagnosia..   206  What is the fate of these spared regions of the face network? Do they continue to function normally, only no longer able to translate neural activity into normal behavioral performance? Or is normal functioning disrupted due to damage elsewhere in the face network? The simple identification of face-selective regions in acquired prosopagnosia shows that, to some extent, these spared regions are performing their prescribed functions, by discriminating between faces and objects (Rossion, Caldara et al., 2003). However, current models of face perception attribute more specific roles to each of these regions: initial perception of facial structure in the OFA, perception of facial identity in the FFA, and perception of facial expression in the pSTS (Haxby, Hoffman et al., 2000). While these roles remain the standard a number of studies have suggested possible identity perception in the OFA (Avidan, Hasson et al., 2005; Rossion, Schiltz et al., 2003), expression perception in the FFA (Fox, Iaria et al., 2008a; Ganel, Valyear et al., 2005; Vuilleumier, Richardson et al., 2004), and identity perception in the pSTS (Fox, Iaria et al., 2008a; Winston, Henson et al., 2004).   One method used to assess the functional sensitivities of these various regions is that of fMRI adaptation, which measures reduced BOLD signal in response to repeated presentations of a stimulus (Grill-Spector, Henson et al., 2006). Indeed, this method has been used to demonstrate OFA sensitivity to structural changes in a face (Rotshtein, Henson et al., 2005), FFA sensitivity to identity (Rotshtein, Henson et al., 2005; Winston, Henson et al., 2004) and expression changes (Fox, Iaria et al., 2008a), and pSTS sensitivity to expression and identity changes (Fox, Iaria et al., 2008a; Winston, Henson et al., 2004). To date, only one instance of fMRI adaptation in an acquired prosopagnosic has been reported, wherein it was noted that patient PS did not exhibit any sensitivity to facial identity changes in the right FFA, but did show identity sensitivity in an object-selective portion of the ventral lateral occipital cortex (Dricot, Sorger et al., 2008; Schiltz, Sorger et al., 2006). In contrast to this is the fMRI adaptation study of four congenital prosopagnosics who do demonstrate sensitivity to facial identity changes within both the OFA   207and FFA (Avidan, Hasson et al., 2005). However, these studies are both limited in determining the significance of reported adaptation effects in the single prosopagnosic patient. The adaptation effects for faces and objects in patient PS are compared to a number of controls showing that her effects for faces are numerically, but not necessarily significantly, smaller than the adaptation effect seen in each individual control subject (Dricot, Sorger et al., 2008; Schiltz, Sorger et al., 2006). The adaptation effects seen in the congenital prosopagnosics are significant at the group level, but no mention is made of significant adaptation within the individual patient (Avidan, Hasson et al., 2005). While it may be valid to group congenital prosopagnosics who have no apparent neurological lesion, the heterogeneity of damage in acquired prosopagnosia (Barton, 2003) makes group analyses difficult to interpret. Thus it is important to design an fMRI adaptation study whereby significant sensitivity to identity or expression changes can be determined in the individual patient.   The first goal of the present study is thus to develop a method whereby the significance of adaptation effects can be determined within the single subject. The power of group analyses lies in the averaging of results across a number of subjects, with significance achieved when an effect is consistently observed across most, if not all, individuals (Friston, Holmes et al., 1999). In a parallel fashion, averaging across multiple scans within a single subject can increase the power to detect a significant effect in that subject. Thus, by performing multiple adaptation scans in each individual, and averaging across these scans, the significance of adaptation effects can be determined within the single subject. In Experiment 1 we perform such a study in a small sample of controls to ensure that this is possible in practice.   The second goal of the present study is then to use this method to assess residual sensitivity for identity and expression in a prosopagnosic population. In Experiment 2 we examine four cases of acquired prosopagnosia, each with very different etiologies. Each patient will be tested on a wide array of face tests in order to characterize their particular deficits in face   208processing. Next, the extent of the ?functional lesion? will be mapped out in these individuals using a face localizer. Finally, residual sensitivity to identity and expression changes will be assessed in all spared regions of the core system for face processing (bilateral OFA, FFA, and pSTS), and correlated with any observed deficits in face perception. The most likely regions in which to find residual sensitivity would be the right FFA for identity changes, and the right pSTS for expression changes, although as discussed above this distinction has come under considerable question (Calder and Young, 2005). Furthermore, as residual sensitivity to identity changes was observed in non-traditional areas in patient PS (Dricot, Sorger et al., 2008), it is important to assess all core regions of the face processing network for residual sensitivity to identity or expression changes in any of these areas.    8.2  Experiment 1: Significant adaptation in the single subject.  8.2.1 Methods  8.2.1.1 Participants Three healthy right handed participants (C01-28 year old male, C02-34 year old male, C03-27 year old female) with normal or corrected-to-normal vision and no history of neurological disorders participated. Informed consent was obtained and the protocol was approved by the institutional review boards of the University of British Columbia and Vancouver General Hospital, in accordance with The Code of Ethics of the World Medical Association, Declaration of Helsinki (Rickham, 1964).      2098.2.1.2 Stimuli Face images were selected from the Karolinska Database of Emotional Faces (Lundqvist and Litton, 1998) and from the authors? personal collection. All images were tightly cropped about the face and uniformly sized to 512 by 634 pixels. A standard gray oval was placed over each face which obstructed the neck, hairline and picture background while leaving internal facial features and external face contour unaffected (Figure 8.1). Quartets of face images were selected such that two images were of the same identity displaying the same expression, a third image of the same identity displaying a different expression, and a fourth image of a different identity but which displayed the same expression as the first two images. Forty such quartets were created, twenty using female faces and twenty using male faces. Five facial expressions were included amongst the faces (anger, fear, happiness, sadness, disgust) with each expression appearing as the base expression (displayed in 3 of the 4 images) and the different expression (displayed in 1 of the 4 images) ten times (5 for each gender). Images from each of the 40 face quartets were paired to create the three experimental conditions. The same image was always presented as the first in each pair with the second image varying between conditions: same-identity same-expression, different-identity same-expression, same-identity different-expression. This resulted in 40 unique trials for each of the three experimental conditions.    Finally, six other faces (3 male, 3 female) displaying 3 different expressions (anger, fear, happiness) were selected and formatted in a gray oval as described above. Upright and inverted versions of these six faces were created. Two face pairs were formed for each of the six identities; upright-inverted and inverted-upright. These twelve pairs became target trials in the fMRI adaptation experiment.      210Figure 8.1  Examples of fMRI adaptation trials  In all three experimental conditions the first image was the same. The second image in the pair was either a new picture with the same identity and same expression as the first image, a picture of a different person with the same expression or a picture of the same person with a different expression. An image pair was presented within every TR (2s) and fixation trials were randomly intermixed with experimental trials. Forty similar face quartets were used to create 40 unique trials per experimental condition.          2118.2.1.3 Experimental design An experimental trial consisted of a pair of faces presented within each TR (2s). The first face was presented for 500ms and followed by a 300ms inter-stimulus-interval. This was followed by a 500ms presentation of the second face and a 700ms inter-trial-interval (Figure 8.1). In order to avoid retinal adaptation image location randomly varied from image to image within a region of 50 by 50 pixels. Within each experimental scan 32 of the 40 face quartets were randomly selected, and all 3 experimental trials from these quartets were presented during the scan; 32 trials per condition and 96 trials total. In addition to these experimental trials, 10 of the 12 target trials were randomly selected and included also. Participants were asked to respond to an inverted face with a key press, thereby ensuring attention was directed to the faces. Finally, 48 fixation trials, in which face images were replaced by a fixation cross, were randomly interspersed among experimental and target trials, producing the necessary jittering for the analysis of rapid event-related experimental designs (Grill-Spector, Knouf et al., 2004; Serences, 2004). Six different experimental scans were created and presented to all participants in a random order. Each experimental scan began with 1 and ended with 6 fixation trials.   8.2.1.4 FMRI data acquisition Structural and functional MRI scans were performed on all participants. All scans were acquired in a 3.0 Tesla Philips scanner. Stimuli were presented using Presentation 9.81 software and were rear-projected onto a mirror mounted on the head coil. Whole brain anatomical scans were acquired using a T1-weighted echoplanar imaging (EPI) sequence, consisting of 170 axial slices of 1mm thickness (1mm gap) with an in-plane resolution of 1mm X 1mm (FOV=256). T2-weighted functional scans (TR=2s; TE=30ms) were acquired using an interleaved ascending EPI sequence, consisting of 36 axial slices of 3mm thickness (1mm gap) with an in-plane resolution of 1.875mm X 1.875mm (FOV=240).    212A dynamic localizer, which presented dynamic videos of faces and objects (Iaria, Fox et al., 2008), was used to identify regions comprising the core system of face processing (i.e.- right and left OFA, FFA, and pSTS) (Haxby, Hoffman et al., 2000).  Video-clips of faces all displayed dynamic changes in facial expression (i.e.- from neutral to happy). So that dynamic changes in objects were comparable to those seen in faces, all video-clips of objects displayed types of motion that did not create large translations in position. Patients performed an irrelevant ?one-back task?: that is, to press a button if a video was identical to the previous one. Fixation blocks began and ended the session and were alternated with image blocks; all blocks lasting 12 seconds. Eight blocks of each image category (object, face) were presented in a counterbalanced order. Each image block consisted of 6 video-clips (5 novel and 1 repeated) presented centrally for 2000ms each. Video-clips of objects were gathered from the internet, and video-clips of faces were provided by Chris Benton (Department of Experimental Psychology, University of Bristol, UK). All video-clips were resized to a width of 400 pixels. The dynamic localizer was followed by the presentation of the six experimental scans.   The first volume of each functional scan was discarded to allow for scanner equilibration. All MRI data were analyzed using BrainVoyager QX Version 1.8 (www.brainvoyager.com). Anatomical scans were not preprocessed, but were standardized to Talairach space (Talairach and Tournoux, 1988). Preprocessing of functional scans consisted of corrections for slice scan time acquisition, head motion (trilinear interpolation), and temporal filtering with a high pass filter in order to remove frequencies less than 3 cycles/time course. Functional scans were individually co-registered to their respective anatomical scan, using the first retained functional volume to generate the co-registration matrix.      2138.2.1.5 FMRI data analysis The dynamic localizer time course was analyzed with a single subject GLM, with objects (O) and faces (F) as predictors, and an F>O contrast was overlaid on the whole brain. At a False-Discovery-Rate of q<0.05 (corrected for multiple comparisons), we attempted to identify the core regions of face perception, bilaterally, within each participant (Haxby, Hoffman et al., 2000). Contiguous clusters of face-related voxels located on the lateral temporal portion of the fusiform gyrus were designated as the fusiform face area (FFA), clusters located on the lateral surface of the inferior occipital gyrus were designated as the occipital face area (OFA), and clusters located on the posterior segment of the superior temporal sulcus were designated as the pSTS. In order to maximize face-selectivity in each region-of-interest (ROI) we selected the 50 voxels, contiguous with the peak voxel, that displayed the highest t-value for the F>O contrast (Iaria, Fox et al., 2008). These 50 voxel clusters were then subject to experimental analyses.   Experimental MRI scans were analyzed using a deconvolution analysis that accounts for non-linear summation of the blood oxygen level dependent (BOLD) response in rapid event-related designs. The deconvolution analysis samples BOLD activity at trial onset (time = 0sec) and again 9 more times in 2sec intervals, resulting in an unbiased model of the hemodynamic response (HDR). The inverted target trials were included as a separate condition in the deconvolution analysis, to account for all non-fixation trials, but were not included in subsequent analyses.   Within each ROI, results from the six experimental scans were combined using a multi-study GLM. While single scans make it difficult to determine significant effects in the single subject, averaging across multiple scans enables the assessment of statistical significance in the single subject. Significant adaptation of the HDR may take a number of forms including a reduced HDR-peak due to neural fatigue or a narrowing of the full-HDR due to a facilitated neural response (Grill-Spector, Henson et al., 2006). Within each ROI, the full-HDR was defined   214as the sum of all consecutive time points (collapse across conditions) that showed a significant increase from baseline (p<0.05, 1-tailed). The HDR-peak was defined as the time point exhibiting a maximal increase in BOLD activity, or the average of this time point and adjacent time points that did not significantly differ (p>0.05, 1-tailed). Using these time points, full-HDR and HDR-peak values were determined for each of the three experimental conditions. Different-identity same-expression > same-identity same-expression and same-identity different-expression > same-identity same-expression contrasts were performed, using the multi-study GLM, to assess identity and expression adaptation respectively. The difference values representing these two contrasts are presented graphically. Significant release from adaptation in the different conditions was set at ? <0.05, and would indicate sensitivity of the ROI to changes in identity or expression. As all effects in the full-HDR condition were replicated in the HDR-peak condition, and were indeed stronger when analyzing the HDR-peak alone, we only present the results of the HDR-peak analyses.  8.2.2 Results We were able to identify all six regions of the core system (bilateral OFA, FFA, and pSTS) in each of the three control subjects (Table 8.1). C01 demonstrated a significant release from adaptation to identity changes within the right FFA (Mean+SEM, p; 0.21+0.09, p<0.05). No other regions displayed sensitivity to identity changes, nor did any region show sensitivity to changes in facial expression (p>0.15). C02 did not show sensitivity to identity or expression changes within any of the six core ROIs (p>0.10). C03 demonstrated a significant release from adaptation to changes in identity within the right FFA (0.18+0.07, p<0.01) and the left OFA (0.18+0.08, p<0.05) and a trend in the same direction within the right OFA (0.20+0.12, p=0.09). No other regions displayed sensitivity to identity changes, nor did any regions within C03 show sensitivity to expression changes (Figure 8.2).    215Table 8.1  Dynamically localized core system in four patients and controls  Subject  Region  Maximum t-value  Minimum t-value  X  Y  Z B-IOT1  ROFA 12.16  10.62 48 -76 -5   RFFA LESION         RpSTS 6.48  4.73 57 -43 7  LOFA LESION        LFFA  4.03  2.99  -39 -49 -20  LpSTS  5.95  4.08  -51 -61 -2 R-IOT1  ROFA  LESION           RFFA  LESION           RpSTS  5.52  3.67  57  -40  13   LOFA  6.50  4.85  -37  -82  -20   LFFA  4.73  3.18  -33  -67  -23   LpSTS  7.42  5.23  -42  -40  4 B-AT1  ROFA  12.37  11.18  30  -88  -5   RFFA  13.09  10.25  39  -52  -20   RpSTS  9.67  7.62  45  -49  -2   LOFA  9.43  7.45  -30  -85  -8   LFFA  5.96  5.04  -39  -55  -26   LpSTS  5.9  4.95  -60  -46  4 R-AT1  ROFA  14.88  11.27  27  -70  -20   RFFA  11.29  6.46  36  -58  -11   RpSTS  14.18  10.81  42  -40  4   LOFA  12.92  11.31  -42  -70  -8   LFFA  11.90  9.99  -39  -43  -26   LpSTS  11.66  8.81  -57  -46  13 Controls  ROFA 9.32+2.63 7.38+2.66 37+2 -84+8 -17+3   RFFA 12.23+2.77 9.49+2.26 40+3 -50+6 -21+2   RpSTS 13.35+3.11 10.28+2.53 52+5 -37+3 5+2  LOFA 9.51+1.33 7.29+0.95 -35+5 -73+0 -12+8  LFFA 10.93+1.65 7.52+1.85 -40+2 -47+5 -24+2  LpSTS 12.54+3.17 8.27+1.77 -53+11 -42+9 0+6  Results of the dynamic functional localizer, with brains standardized to Talairach space. The peak 50 voxels were defined as the region-of-interest with maximum and minimum t-values reported.        216Figure 8.2  Single subject adaptation effects in controls A) Control data for the different-identity same-expression > same-identity same-expression contrast. A significant release from adaptation (*) for identity changes was seen within the right FFA of C01 and C03, and within the left OFA of C03. A trend in the same direction (#) was observed in the right OFA of C03. B) No significant release from adaptation was observed for changes in expression, following the same-identity different-expression > same-identity same-expression contrast.    217  The presence of significant adaptation effects within these controls validates this procedure for assessing significant adaptation within the single subject. Of particular interest is the presence of identity sensitivity within regions of inferior occipitotemporal cortex, including the right FFA in 2 controls, an area known to be involved in the processing of facial identity (Haxby, Hoffman et al., 2000; Rotshtein, Henson et al., 2005; Winston, Henson et al., 2004). However, we did not observe expression sensitivity within any region in this control sample. Furthermore, C02 did not display identity or expression sensitivity within any of the core regions, a result that cannot be taken to indicate impairments in this healthy control. Thus, while the assessment of significant positive effects appears viable with this method, we can make no claims as to its reliability. The absence of adaptation effects cannot be taken to reflect impaired processing. With this important caveat we can now examine the damaged brains of a prosopagnosic population for any residual sensitivity to identity or expression changes.    8.3  Experiment 2: Residual sensitivity in a patient population.  8.3.1 Methods  8.3.1.1 Subjects Four patients were included in the current study. Informed consent was obtained and the protocol approved by the institutional review boards of the University of British Columbia and Vancouver General Hospital, in accordance with The Code of Ethics of the World Medical Association, Declaration of Helsinki (Rickham, 1964).   A series of tests were administered to assess the status of face perception in these four patients (Table 8.2). First, the perception of facial identity was assessed with the Benton Facial  Recognition Test (Benton and van Allen, 1972), and with a three alternative forced choice   218Table 8.2  Patient results from the battery of face tests  Modality  Test  Max B-IOT1 R-IOT1 B-AT1 R-AT1 Faces - Identity  Benton Facial Recognition  54  29*  45 45 41  Morph Discrimination 100%36* 83* 72* 56* Faces ? Expression Reading the Mind in the Eyes 36  8*  26  24  19*   Morph Discrimination  100% 39*  92  100  92 Faces - Memory  Words/Faces, WRMT  50/50  43/-*?  41/33*  45/27* 41/17*  Famous Face Recognition (d?) 3.92  0.02*  1.96  1.52*? 1.22*   Face Imagery (%)  100% 76*  82 N/A 71* Faces- Covert Famous Pairs  20  13*  18  11*  8*   Occupation Sorting  41  21*  39  21*  24*  Impairments are indicated in red. (WRMT = Warrington Recognition Memory Test)  ? B-IOT1 refused to perform the face version of this test, stating it would be too difficult. ? Due to poor knowledge of celebrities, a version of this test using personally familiar faces was given to B-AT1.                 219discrimination (chance = 33%) of morphed identity changes (Fox, Iaria et al., 2008b). Next, the perception of facial expression was assessed with the revised Reading the Mind in the Eyes Test (Baron-Cohen, Wheelwright et al., 2001), and with a discrimination of morphed expression changes, balanced in difficulty with the test for morphed identity changes (Fox, Iaria et al., 2008b). Next, facial memory was assessed using the  Warrington Recognition Memory Test (Warrington, 1984), a famous face familiarity test (Barton, Cherkasova et al., 2001), and a facial imagery test (Barton and Cherkasova, 2003). Finally, covert face recognition was assessed through a famous pairs test, wherein famous faces were selected from pairs of familiar/unfamiliar faces, and through the sorting of famous faces by occupation (i.e.- politics and acting) (Barton, Cherkasova et al., 2001).  8.3.1.2 Patient descriptions B-IOT1 (B = bilateral; IOT = inferior occipitotemporal) is a 71 year old right handed male who, 3 yrs prior to testing, suffered a left occipital hemorrhage following heparin treatment of a deep vein thrombosis after knee replacement surgery. Anticoagulation treatment was reversed and was shortly followed by a right inferior occipitotemporal infarct (Figure 8.3). B-IOT1 has subsequently complained of problems with colors, reading, and facial recognition, although voice recognition is normal. B-IOT1 presents with a mild right hemianopia with vision corrected to 20/20 in the remaining visual field. Achromatopsia is apparent with impaired naming for all colors but red, and alexia without agraphia is apparent. B-IOT1 shows profound impairments on all face tests (Table 8.2). Identity perception is impaired on the Benton Facial Recognition Test (29/54) and performance was near chance on the discrimination of morphed identity changes (36%). Expression perception is impaired on the Reading the Mind in the Eyes Test (8/36) and again performance is near chance on the discrimination of morphed expression changes (39%). Facial memory is impaired as indicated by the famous face recognition test (d?=0.02), and the    220Figure 8.3  Structural MRI of the four prosopagnosic patients  Coronal slices of the four patients, standardized to Talairach space. Slices were taken every 12mm, from y = +48mm to y = -84mm. B-IOT1 has multiple lesions, a left inferior frontal lesion (+48 to +24mm), a right inferior occipitotemporal infarct (-24 to -84 mm) and a large left occipital hemorrhage (-60 to -84mm). R-IOT1 has a single right inferior occipitotemporal hemorrhage (-48 to -84m). B-AT1 has large bilateral lesions of the anterior temporal lobes (+12 to -36mm). R-AT1 has a small surgical lesion in the right anterior temporal lobe, affecting the right hippocampus and amygdala (0 to 12mm).                        221   222facial imagery test (76%). Importantly, the imagery test assesses the status of facial memory stores without relying on intact perceptual systems. B-IOT1 refused to take the Faces version of the Warrington Recognition Memory Test, but showed normal performance on the Words version of the same test (43/50). Finally, there was no evidence fot covert face recognition, with impairments on both the famous pairs (13/20) and the occupation sort (21/41) tests. General semantic memory for famous individuals, however, was unaffected. R-IOT1 (R = right) is a 49 year-old left-handed male who, twelve years prior to testing, had an occipital cerebral hemorrhage from rupture of an arterovenous malformation (Figure 8.3). Immediately following this incident, he complained of trouble recognizing hospital workers, and needed to rely on hairstyle, facial hair, or voice for person recognition, a problem which has not resolved. He also has a left superior quadrantanopia (with 20/20 vision in the remaining visual field), and mild topographagnosia (difficulty navigating in new locations). R-IOT1?s self report also indicated the need for letter-by-letter reading immediately following the cerebral hemorrhage, although this had resolved by time of testing. R-IOT1 showed normal performance on most face tests (Table 8.2), but was impaired on the discrimination of morphed identity changes (83%),