Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Opening a window on Bálint syndrome : testing a spatial restriction of attention theory Dalrymple, Kirsten A. 2010

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2010_fall_dalrymple_kirsten.pdf [ 31.26MB ]
Metadata
JSON: 24-1.0071294.json
JSON-LD: 24-1.0071294-ld.json
RDF/XML (Pretty): 24-1.0071294-rdf.xml
RDF/JSON: 24-1.0071294-rdf.json
Turtle: 24-1.0071294-turtle.txt
N-Triples: 24-1.0071294-rdf-ntriples.txt
Original Record: 24-1.0071294-source.json
Full Text
24-1.0071294-fulltext.txt
Citation
24-1.0071294.ris

Full Text

Opening a window on Bálint syndrome: Testing a spatial restriction of attention theory by  KIRSTEN A DALRYMPLE B. Sc. (Hons), Queen’s University, 2004 M. A., University of British Columbia, 2006  A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF  DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE STUDIES (Psychology)  THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver)  September 2010   Kirsten A. Dalrymple 2010  Abstract Bálint syndrome is a disorder of visual attention resulting from bilateral parietooccipital lesions. Patients experience several visual deficits, including an inability to see more than one object at once, and often an inability to see single objects as wholes. This dissertation examines whether impaired object processing in Bálint syndrome results from a restricted spatial area of visual attention, which creates a “restricted visual window” through which patients view the world. In Study 1, brain activity of healthy individuals is recorded while they view hierarchical stimuli, stimuli that patients tend to see at a local, but not global, level. Activity increased when participants identified global letters, suggesting that patients may have difficulty perceiving these objects due to extra processing demands of the global form. Study 2 uses hierarchical letters to investigate whether patients can employ explicit viewing strategies to compensate for their visual deficits. The patient showed difficulty identifying the global level of the letters, and did not employ a strategy to compensate- she appeared to have little control over what she sees. Study 3 employs a manipulation of healthy vision to model these behaviours to understand what underlies this disorder. Restricting healthy individuals to seeing a small visual area (like “tunnel vision”) leads to object identification patterns similar those of Bálint patients, supporting the idea that a restricted area of visual processing may underlie the disorder. Study 4 used photos depicting social scenes to investigate how patients view complex stimuli. Unlike healthy individuals, patients do not look at the eyes of people in the scenes. In Studies 5-6, healthy individuals view these stimuli through the restricted viewing scenario from Study 3 to determine whether restricted viewing can also model patients’ viewing of complex stimuli. Like patients, healthy individuals made reduced fixations on eyes of people in scenes. Study 7 revisits a patient after some recovery. Her scanning of scenes was approaching normal. Manipulations of the restricted viewing paradigm modeled this recovery in healthy individuals, lending further support to the restricted vision model. The dissertation provides insights into normal and abnormal vision, particularly how the brain creates the objects we see.  ii  Table of Contents Abstract......................................................................................................................ii Table of Contents.....................................................................................................iii List of Tables .............................................................................................................v List of Figures ..........................................................................................................vi Acknowledgements ................................................................................................vii Co-Authorship Statement......................................................................................viii Chapter 1 General introduction ...............................................................................1 Bálint syndrome................................................................................................................ 2 Object-based theories of attention.................................................................................. 3 Space-based theories of attention .................................................................................. 5 The co-existence of object-based and space-based attention..................................... 9 Visual attention and Bálint syndrome........................................................................... 11 Testing Bálint syndrome with impoverished stimuli................................................... 11 Modeling restricted attention through a restriction of vision..................................... 14 Testing Bálint syndrome with complex stimuli............................................................ 15 Thesis overview .............................................................................................................. 16 References....................................................................................................................... 20  Part 1: Impoverished stimuli ..................................................................................25 Chapter 2 Event-related potential evidence for a dual-locus model of global/local processing ..........................................................................................26 Method ............................................................................................................................. 29 Results ............................................................................................................................. 32 Discussion....................................................................................................................... 44 References....................................................................................................................... 49  Chapter 3 Global perception in simultanagnosia is not as simple as a game of connect-the-dots .....................................................................................................51 Method ............................................................................................................................. 54 Results ............................................................................................................................. 61 Discussion....................................................................................................................... 68 References....................................................................................................................... 73  Chapter 4 Simulating simultanagnosia: Spatially constricted vision mimics local capture and the global processing deficit...................................................75 Method ............................................................................................................................. 80 Analysis and results ....................................................................................................... 87 Discussion....................................................................................................................... 91 References....................................................................................................................... 98  Part 2: Complex stimuli ........................................................................................101  iii  Chapter 5 Experiencing simultanagnosia through windowed viewing of complex social scenes .........................................................................................102 Study 4 ........................................................................................................................... 105 Method ........................................................................................................................... 105 Results ........................................................................................................................... 114 Discussion..................................................................................................................... 115 Study 5 ........................................................................................................................... 116 Method ........................................................................................................................... 116 Results ........................................................................................................................... 117 Discussion..................................................................................................................... 119 Study 6 ........................................................................................................................... 119 Method ........................................................................................................................... 120 Results ........................................................................................................................... 121 Discussion..................................................................................................................... 122 General discussion....................................................................................................... 123  Chapter 6 Opening a window on attention: Documenting and simulating recovery from simultanagnosia...........................................................................132 Method ........................................................................................................................... 137 Experiment 1: SL........................................................................................................... 143 Results ........................................................................................................................... 144 Experiment 2: Mouse-Contingent windows ............................................................... 148 Method ........................................................................................................................... 149 Results ........................................................................................................................... 151 Discussion..................................................................................................................... 153 References..................................................................................................................... 160  Chapter 7 General discussion .............................................................................163 1. Why are patients with Bálint syndrome impaired at piecing together local elements into a global whole? ..................................................................................... 164 2. Is the global processing deficit in Bálint syndrome directly related to the restriction of visual attention that is thought to underlie this disorder? ................ 165 3. How do patients with Bálint syndrome scan complex stimuli? ........................... 166 4. Can a restricted window of visual attention account for more complex Bálint behaviours?................................................................................................................... 166 5. How does Bálint patient behaviour change with recovery and what can we learn from changes in behaviour over the course of recovery?........................................ 167 Implications of this dissertation .................................................................................. 167 Limitations of this dissertation.................................................................................... 171 Behaviours that suggest that Bálint syndrome involves more than a limited window of attention alone .......................................................................................................... 172 A note on the link between the restricted “window” of attention and the “spotlight” of attention .................................................................................................................... 174 Future directions........................................................................................................... 176 References..................................................................................................................... 178  Appendices............................................................................................................181 Appendix A .................................................................................................................... 182 Appendix B .................................................................................................................... 188 Appendix C .................................................................................................................... 189 Appendix D .................................................................................................................... 190  iv  List of Tables Chapter 2 Table 2.1 RTs for global/local processing of hierarchical letters………………..…....31 Table 2.2 P1 amplitudes for global/local processing of hierarchical letters……..…..38 Table 2.3 N1 amplitudes for global/local processing of hierarchical letters…….…...39 Table 2.4 N2 amplitudes for global/local processing of hierarchical letters……...….40 Table 2.5 P3 amplitudes for global/local processing of hierarchical letters…….…...40 Chapter 3 Table 3.1 SL’s incorrect global responses………………………………………….…..67  v  List of Figures Chapter 1 Figure 1.1 Hierarchical letter stimuli............................................................................8 Figure 1.2 Schematic of modelling an attentional impairment through a limitation of vision.........................................................................................................................15 Chapter 2 Figure 2.1 Large/Sparse and Small/Dense hierarchical letter stimuli…………….….27 Figure 2.2 Waveforms for P1, N1, and N2 ERP components………………………...35 Figure 2.3 P3 data…………………………………………………………………...…….36 Figure 2.4 Mean amplitudes of the P1, N1 and N2 components……………………..37 Chapter 3 Figure 3.1 MRI scans from patients SL and ES……………………………….….……55 Figure 3.2 Navon hierarchical letter stimuli………………………….……….…………57 Figure 3.3 Example of critical difference analysis ……………………..………………59 Figure 3.4 Accuracy for naming global and local letter stimuli……….……………….61 Figure 3.5 Basic eye movement measures…………………………………..…………64 Figure 3.6 Example of patient scan path……………………………………..…………70 Chapter 4 Figure 4.1 Hierarchical Navon letters………………………………………….…….…..77 Figure 4.2 Schematic of gaze-contingent paradigm…………………………….….….79 Figure 4.3 Axial FLAIR sequences of MRI scans of patient SL…………….………...82 Figure 4.4 Accuracy for naming global and local letter stimuli..……..…..…………...88 Chapter 5 Figure 5.1 MRI scans from patients SL, KC, and ES……………………..….………107 Figure 5.2 Example of scene stimuli and regions of interest………….….….…...…109 Figure 5.3 Fixation proportions per region…………………………………………….112 Figure 5.4 Saccade amplitudes, number of fixations, duration per trial……..……..113 Figure 5.5 Fixation proportions on eyes over time……….….….….…….…….…….123 Chapter 6 Figure 6.1 Axial FLAIR sequences of MRI scans of patient SL………..……………138 Figure 6.2 Example of scene stimuli and regions………………..……….…….…….141 Figure 6.3 Fixation proportions………………………….….…….…….….……….…..146 Figure 6.4 Fixation durations, number of fixations, and trial durations…..…...…....147 Appendix A Figure A.1 MRI scans from patient SL .…...............................................................183 Figure A.2 MRI scans from patient KC…….….…………….…………………...…….185 Figure A.3 MRI scans from patient ES ...…………………………………………..….186  vi  Acknowledgements William Shakespeare once wrote, “Our doubts are traitors and make us lose the good we might win, by fearing to attempt.” I would like to extend my deepest and most sincere gratitude to the people in my life who consistently ushered away my doubts, and in doing so, contributed to my success. First and foremost, I would like to thank my supervisor, Alan Kingstone, for his unwavering support, both personally and professionally, throughout my graduate studies. I would also like to thank my co-supervisor, Jason Barton, and my mentors Walter Bischof, Jim Enns, Todd Handy, and Kim Hellemans, for their support and guidance throughout this process. I would like to thank patient SL for her dedicated participation in my research, and for sharing with me her optimism and wisdom for life. Finally, I would like to extend my utmost gratitude to my friends and family, particularly to my parents, Murray and Joan, for their encouragement and support, and for providing me with the skills and confidence to achieve my goals. The research reported in this dissertation was funded by junior and senior graduate fellowships from the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Michael Smith Foundation for Health Research (MSFHR). Additional support was provided through grants awarded to Alan Kingstone and Jason Barton from NSERC and MSFHR.  vii  Co-Authorship Statement I am the primary author on all work presented in this PhD dissertation, and was responsible for the design of experiments, data collection, data analysis, and manuscript preparation. The specific contributions for each chapter are below. Chapter 1: General Introduction. A version of this chapter has been submitted for publication. I am the primary author of this chapter, with intellectual contributions from A. Kingstone. Chapter 2: Event-related potential evidence for a dual-locus model of global/local processing. A version of this chapter has been published. I am primary author of this chapter, and designed and conducted the research, as well as the statistical analyses on all behavioural data. A. Kingstone provided intellectual contributions. T.C. Handy supervised this project, performed statistical analyses on the ERP data, and provided intellectual contributions. Chapter 3: Global perception in simultanagnosia is not as simple as a game of connect-the-dots. A version of this chapter has been published. I am primary author of this chapter, and designed and conducted the research, as well as all statistical analyses. W.F. Bischof assisted with the statistical analyses. D. Cameron programmed the experiment. J.J.S. Barton was the referring Neurologist and provided intellectual contributions. A. Kingstone supervised the project and provided intellectual contributions. Chapter 4: Simulating simultanagnosia: Spatially constricted vision mimics local capture and the global processing deficit. A version of this chapter has been published. I am primary author of this chapter, and designed and conducted the research, as well as all statistical analyses. W.F. Bischof assisted with the statistical analyses. D. Cameron programmed the experiment. J.J.S. Barton was the referring Neurologist and provided intellectual contributions. A. Kingstone supervised the project and provided intellectual contributions. viii  Chapter 5: Experiencing simultanagnosia through windowed viewing of complex social scenes. I am primary author of this chapter, and designed and conducted the research, as well as all statistical analyses. E. Birmingham provided intellectual contributions and the stimuli for the experiments. W.F. Bischof assisted with the statistical analyses. J.J.S. Barton was the referring Neurologist and provided intellectual contributions. A Kingstone supervised the project and provided intellectual contributions. Chapter 6: Opening a window on attention: Documenting and simulating recovery from simultanagnosia. I am primary author of this chapter, and designed and conducted the research, as well as all statistical analyses. E. Birmingham provided intellectual contributions and the stimuli for the experiments. W.F. Bischof assisted with the statistical analyses. J.J.S. Barton was the referring Neurologist and provided intellectual contributions. A. Kingstone supervised the project and provided intellectual contributions.  ix  Chapter 11 General introduction  1  A version of this chapter has been submitted for publication: Dalrymple, KA & Kingstone A. (under review). A world unglued: Piecing together the contributions of object- vs. spacebased attention in Bálint syndrome.  1  Bálint syndrome First described in 1909 by Rezsö Bálint (Balint, 1909), Bálint syndrome is a relatively rare disorder of visual attention that results from bilateral lesions of the parieto-occipital junction (Rafal, 2003). The syndrome is characterized by four primary symptoms: (1) Optic Ataxia, an inability to use visual input to accurately guide reaching; (2) Ocular Apraxia, a deficit in executing voluntary gaze shifts (3) Spatial Disorientation, a deficit in spatial attention and spatial memory; and (4) Simultanagnosia, a restriction of visual attention such that the patient is only aware of a single object at one time (Moreaud, 2003; Rafal, 2003; Rizzo & Vecera, 2002). Despite intact visual acuity, these symptoms create a chaotic visual world, to the point where patients are often described as being functionally blind (Kim & Robertson, 2001). Patients may see “an ant or an elephant, but only one object at a time.” (Rafal, 2001, p.122), and cannot localize these items nor use visual information to interact with them. In an early study of a patient with Bálint syndrome, Holmes and Horrax (1919) were quick to identify the disorder as one of visual attention rather than blindness since their patient had variable perception of objects that fell on fully functioning retinas. “The essential feature was his inability to direct his attention to, and take cognizance of, two or more objects that threw their images on the seeing portion of his retinae. As this occurred no matter on what parts of his retinae the images fell, it must be attributed to a special disturbance or limitation of attention…” (Holmes & Horrax, 1919, p.390). Visual attention is said to be the “glue” that holds our world together, so it is not surprising that Bálint syndrome has been linked to a faulty attentional system, resulting in a visual world experienced in pieces. While it is generally accepted that Bálint syndrome is a disorder of visual attention, authors tend not to elaborate on what is meant by “attention” in this context. It is often described as a “restriction” of visual attention, but how does this restriction manifest itself?  2  Theories of visual attention attempt to explain how attention operates to hold our visual world together. Two prominent theories exist. One theory suggests that attention is object-based, selecting features to create different objects. The other theory suggests that attention is space-based, selecting locations in space, binding together information that occupies a common location. It is unclear what type of attentional deficit underlies Bálint syndrome. For example, if patients can see only one object at a time, is it an object-based deficit, in that they are unable to see all other objects? Or, is it a space-based deficit? After all, if you can only see one location at a time, how can you see more than one object? Object-based and Spacebased theories of attention and how they apply to Bálint syndrome will be reviewed here.  Object-based theories of attention Object-based theories of attention suggest that attention is allocated to objects and object features, and view attention in terms of the number of separate objects that can be perceived at once (Duncan, 1984). According to this theory, there are two stages to this process: A pre-attentive stage, which creates candidate objects from basic elements and Gestalt principles like grouping, and a focal attention stage, which allows more detailed processing of individual objects. This theory predicts that multiple judgments about a single object should be faster than single judgments about multiple objects, since multiple objects must be processed serially. This prediction has been supported empirically by evidence that even when the spatial distribution of the two judgments remains constant, subjects are more accurate when making two judgments about a single object than single judgments about two objects (Duncan, 1984). This “two-object cost” has since been replicated with variations of this basic experiment, for example, subjects have more trouble making spatial judgments about two objects as compared to single objects (Baylis, 1994; Baylis & Driver, 1993). Further supporting an object-based theory of attention is the finding that attention can be directed to object features. Humphreys (1981a) found that subjects 3  could allocate attention on the basis of stimulus colour, with facilitated response to targets that were in an expected colour that differed from the distractor colour. The selection of colour occurred before the selection of target location. Using a negative priming task, Tipper, Brehaut and Driver (1990) found that attention can follow the path of an object, even when the object is temporarily occluded. Finally, functional Magnetic Resonance Imaging (fMRI) and other neurophysiological techniques have supported an object-based prediction that if attention is directed to one attribute of an object, attention will be directed to all attributes of that object. For example, when subjects attend to the motion of an object, there is increased activation in objectsensitive brain areas, suggesting that the object is being processed in its entirety (O'Craven, Downing, & Kanwisher, 1999). One of the defining features of Bálint Syndrome is simultanagnosia, the inability to see more than a single object at one time. While perception of that single object may be intact, the fact that single objects are perceived at the expense of all other objects makes an obvious case for Bálint syndrome resulting from damage to an object-based attentional system. “It is whole objects that are neglected, not spatially determined parts of objects; and the objects that are neglected may occupy the same spatial coordinates as an object that is seen.” (Rafal, 2001, p.128). Much anecdotal evidence from bedside testing points to object-related behavioural deficits in Bálint syndrome. In an early investigation of Bálint syndrome, Holmes and Horrax (1919) report that their patient was able to identify the shape of a square on a page, but when a fixation cross was added, he saw only the cross despite the fact that the cross was occupying the same spatial location as the square. The square was completely neglected. In another, more recent example, Rafal (2001) reports that his patient RX struggled to see both a comb and a ruler at once. When the ruler and the comb were both in view, RX identified only the comb and did not know where the ruler had gone. Further illustrating this tendency to neglect whole objects, Humphreys and Price (1994) provided evidence of non-spatial extinction in two patients who had Bálint syndrome. When a word and an object were presented simultaneously in the  4  same location (i.e. overlapping), the patients reported seeing only the picture on the majority of trials, seldom reporting just the word. Similar effects were demonstrated with closed versus open shapes: patients reported seeing the closed shapes far more often than the open shapes. Since the spatial location of the objects in these examples remain constant, the properties of the object itself must be underlying these effects, influencing what the patients attend to and perceive, regardless of space. Often times the perception of objects in Bálint syndrome can be altered even when spatial information remains constant. As opposed to objects having preserved and preferential treatment in the Bálint brain, evidence is mounting that objects may arrive disjointed and in pieces, much to the confusion of the patient (Rafal, 1997). These object-related behavioural effects may also support the possibility that Bálint syndrome results from damage to an object-based attentional system. One phenomenon that illustrates this decomposition of objects is “local capture” (Karnath, Ferber, Rorden, & Driver, 2000). Here patients are perfectly accurate at identifying the local components of an object, yet fail to see the object as a global whole (e.g. they can see a single tree, but are unable to see the forest). In summary, patients’ visual difficulties that are seemingly object-related suggest that Bálint syndrome may be an object-based deficit (i.e. due to damage to an object-based system). However, based on a careful review of the literature, the case for a space-based deficit is perhaps more convincing. Space-based theory and its application to the Bálint deficits will be reviewed here.  Space-based theories of attention Space-based theories of attention suggest that attention is allocated to a particular location in space. There are many ways of conceptualizing how this may occur (Cheal, Lyon, & Gottlob, 1994), but one of the more popular analogies is of a “spotlight” (Posner, Snyder, & Davidson, 1980; Shulman, Remington, & McLean, 1979; Tsal, 1983). This idea proposes that, like a spotlight, attention can be moved and directed to various locations in space. Just as a spotlight can become brighter when the beam is focused, the size of an attended area can grow and shrink at the 5  expense of attentional acuity. In support of these theories, evidence shows that cueing subjects to the location of a stimulus prior to stimulus onset speeds the subject’s response to that stimulus, suggesting that attention was oriented to that location in space prior to the presentation of the object (Posner, Nissen, & Ogden, 1978). Similarly, increasing the distance between targets and distractors facilitates response to the targets (Eriksen & Hoffman, 1973). When distractors fall within 1 degree of the target, they are processed along with the target and thus interfere. When distractors are moved beyond this area of attentional focus, they are beyond the perimeter of the attentional “spotlight” and therefore interfere to a lesser degree, or, if moved far enough, do not interfere at all. Humphreys (1981b) provided evidence that the size of the attentional spotlight can change from wide-angle to more focused and vice versa, depending on the location of a target. Electrophysiological studies support the concept of space-based attention, showing that there is clear enhancement of visual evoked potentials when subjects direct attention to the location of a stimulus onset, and conversely, amplitudes decrease when attention is directed to the opposite visual field (Eason, 1981; van Voorhis & Hillyard, 1977). Neuropsychological studies involving a disorder of attention called Neglect have demonstrated that individuals with unilateral parietal damage may be unable to direct attention to the left side of space, whether that space is defined as viewer centered, or environment centered (Calvanio, Petrone, & Levine, 1987; Farah, Brunn, Wong, Wallace, & Carpenter, 1990). Evidence is mixed as to whether there is a corresponding object-based neglect (e.g. Farah, et al., 1990), suggesting that neglect may be entirely space-based. Some Bálint behaviours clearly point to a deficit of spatial attention. Spatial disorientation is an inability to locate objects in space, optic ataxia is an inability to use visual information to guide action on objects, and optic apraxia is an inability to accurately guide eye movements to objects. Other behaviours have a subtler link to disordered spatial processing. For example, patients with Bálint syndrome are often reported to have impaired depth perception (e.g. Rafal, 2003): they do not blink or flinch in response to a visual threat (e.g. Holmes & Horrax, 1919), and may bump  6  into objects when walking through a room. This may reflect an inability to judge the relative location of two objects- after all, if you can only see one object at a time, it is impossible to tell where one object lies relative to another (Rafal, 2003). Feature Integration Theory (FIT) proposes that objects are created through the binding of features that occupy the same location in space (Treisman & Gelade, 1980). While features may be processed pre-attentively, focused attention is necessary to combine features into correct perceptions. Patients with Bálint syndrome experience a large number of Illusory Conjunctions (ICs), incorrectly binding features from different objects (Friedman-Hill, Robertson, & Treisman, 1995). According to FIT, the unusually large number of ICs in Bálint syndrome is direct evidence of impaired spatial processing or limited spatial attention, as well as the mislocalization of object features. The notion of the Bálint deficits as resulting from a restricted visual “window” of attention2 is also prevalent in the literature. Bay (1953) suggested that his patient’s simultanagnosia could be accounted for by “shaft vision”, which prevented the patient from seeing the whole picture. Bay described it as a “peripheral constriction”, not unlike “viewing [a] picture through a diaphragm” (p. 545, 546). Similarly, Thaiss and de Bleser (1992) suggested that their patient, TK, may suffer from a rigid reduction of the spatial extent of the visual spotlight. Tyler (1968) also referred to the visual deficit in his patient as “shaft vision” (p. 166), yet in contrast to the description by Thaiss and de Bleser (1992) of a rigid reduction of the visual spotlight, Tyler’s description implies some flexibility. When he measured his patient’s effective visual fields he concluded that they were in fact quite variable: consistent perception occurred within 2° of fixation, and while perception occurred at up to 20°, it quickly fatigued (within 10-30s). Tyler’s description, therefore, suggests a type of  2  I specifically use the term visual “window” of attention to avoid direct analogies with the well-known “spotlight” of attention because the link between the visual attentional limitation in Bálint syndrome and the spotlight of attention is at this point unclear. The possible link between these concepts will be discussed in the General Discussion.  7  shaft vision, with a flexible spatial extent that may widen or shrink. More recently, Michel and Henaff (2004) discuss the simultanagnosic deficit as a shrinkage of the attentional visual field. Based in these reports, it appears as though the size of the visual window is outside of the patient’s control: the window is restricted at rest, and expansion of the window is a demanding process that is easily fatigued. Shalev, Humphreys and Mevorach (2004) found evidence of a reduced, yet flexible window of attention, yet they discovered that the reduced window can be expanded through priming. When viewing hierarchical letters (e.g. a global letter F made up of local letter Ns, Figure 1.1), patients with Bálint syndrome tend to report the local letters, but are unable to see the larger, global letters (Clavagnier, Fruhmann Berger, Klockgether, Moskau, & Karnath, 2006; Dalrymple, Kingstone, & Barton, 2007; Huberle & Karnath, 2006; Karnath, et al., 2000; Shalev, et al., 2004). Shalev et al found that first presenting a large solid letter that takes up the same spatial area as the global aspect of a hierarchical letter improved their patient’s ability to name the global level of that hierarchical letter. These authors interpreted this finding to indicate that the restricted window of attention was temporarily widened by the prime, allowing for explicit processing of the global hierarchical letter.  Figure 1.1 Example of hierarchical letter stimuli typically used to test global/local processing in Bálint syndrome.  8  A space-based view of Bálint syndrome is also supported by the anatomy of the disorder. Bálint syndrome results from bilateral lesions to the parieto-occipital junction, which is part of the dorsal “Where” stream (Goodale & Miller, 1992). The role of the parietal lobes in visual attention (Lynch, Mountcastle, Talbot, & Yin, 1977; Wurtz, Goldberg, & Robinson, 1982) and spatial processing is becoming increasingly clear (Mishkin, 1972; Ungerleider & Mishkin, 1982). Ungerleider and Mishkin’s 1982 influential review “Two cortical visual systems” cites behavioural, electrophysiological, and anatomical evidence, including a systematic ablation study with monkeys, that link the parietal lobes to the processing of space. More recent imaging studies further strengthen this link. Using Positron Emission Tomography (PET) as a measure of regional cerebral blood flow, researchers have shown activation in parietal areas in response to spatial localization tasks (Haxby, et al., 1991). Patients with injuries to the parietal lobes demonstrate complex syndromes such as visuo-spatial neglect, Gerstmann syndrome, and apraxia, all commonly linked to a difficulty processing spatial information. Thus, damage to a space-based system is a logical fit with the current understanding of the anatomy of visual perception.  The co-existence of object-based and space-based attention Duncan (1984) raises the possibility that space-based and object-based theories of attention are not mutually exclusive because even empty space can be thought of as an object, and objects occupy space. This view is echoed by Rafal (1997), who argues that objects are distinguished from other objects based on their spatial location. Similarly, Farah (1990) has argued that space cannot be represented without objects: the location of an object is defined relative to other objects, thus, objects must be perceived relative to each other in order to create the perception of space. The coexistence of object- and space-based mechanisms has been demonstrated by a phenomenon called Inhibition Of Return (IOR) (Posner & Cohen, 1984). IOR was first defined in spatial terms: after shifting attention away from a cued location, subjects are slower to respond to a probe at that cued location  9  compared to a probe at a location that was not cued (Posner & Cohen, 1984). This effect is thought to be adaptive for tasks such as visual search, making search more efficient by suppressing previously attended (searched) locations. Object-based IOR was later demonstrated: when a cued object moves, such that it is no longer in the cued location, subjects show IOR to the object (Tipper, Driver, & Weaver, 1991). Tipper et al. (1994) argued that object- and space-based IOR must coexist in a dynamic world where objects can move to previously attended locations, and previously unattended locations can be occupied by previously attended objects. If space-based and object-based IOR do not work together, objects that move to previously attended locations may be missed, as would locations that contained previously attended objects. Using moving objects, Tipper et al. demonstrated that object-based IOR could be stronger or weaker depending on the final location of the cued object: when the object landed 180° from its original (cued) location, IOR was weaker than when the object landed only 90° from its original location. Also, when the cued object moved to a new location, IOR existed at the originally cued location, even though the cued object was no longer there. Thus object-based and spacebased IOR can co-occur, and their effects appear to be additive. An alternate account of how object- and space-based attention may interact is outlined in the Integrated Competition Hypothesis (Duncan, Humphreys, & Ward, 1997). According to this theory, visual input activates multiple brain systems, within which activations from different objects compete. The competition between objects is integrated between systems such that the winning object in one system becomes the dominant object in others. This competition is directed based on object properties, and the process of selection is flexible, such that many different features can be used to resolve competition. Importantly, these non-spatial factors are combined with spatial influences such that the multiple systems work together on the same area of space. Evidence supporting this theory comes from visual extinction, where patients fail to perceive a contralesional stimulus when there is a competing stimulus on the ipsilesional side of space (Critchley, 1953). The integration of object and spatial attention is an ongoing issue in visual attention research, no doubt  10  critically contributing to the confusion surrounding the true nature of the Bálint deficits.  Visual attention and Bálint syndrome This dissertation looks carefully at the concept of Bálint syndrome as a spaced-based attentional deficit. It is clear from the vocabulary in the literature that many conceptualize the restriction of visual attention in Bálint syndrome as a narrowing of a visual-attentional “window”. As mentioned above, Bay (1953) suggested that his patient’s simultanagnosia could be accounted for by “shaft vision”, which prevented the patient from seeing the whole picture. He described it as a “peripheral constriction”, not unlike “viewing [a] picture through a diaphragm” (p. 545, 546). Similarly, Thaiss and de Bleser (1992) suggested that their patient, TK, may suffer from a rigid reduction of the spatial extent of the visual window. Tyler (1968) also referred to the visual deficit in his patient as “shaft vision” (p. 166) and discussed it as being flexible in size: small at rest, but expandable. Michel and Henaff (2004) discuss the simultanagnosic deficit as a shrinkage of the attentional visual field. Although the concept of a restricted window of attention in Bálint syndrome is gaining popularity, little research has been done to empirically test this concept.  Testing Bálint syndrome with impoverished stimuli As is often the procedure in scientific studies, Bálint syndrome has primarily been studied through the use of carefully controlled, yet arguably stripped down, or “impoverished” stimuli. From the very beginning, patients with Bálint syndrome were tested through the use of simple shapes (e.g. Holmes & Horrax, 1919). One patient was able to identify the shape of a square on a page, but when a fixation cross was added, he saw only the cross despite the fact that the cross was occupying the same spatial location as the square (demonstrating classic simultanagnosia, the inability to see more than one object at a time). Since then, there have been countless examples of studies of patients viewing simple stimuli such as closed versus open shapes (Humphreys, et al., 1994), simultaneously presented words, (Coslett & Saffran, 1991), line drawings (Riddoch & Humphreys, 2004), and so on.  11  One impoverished stimulus that has been frequently used in the study of Bálint syndrome is the hierarchical letter: large, global letters made up of several repetitions of a smaller, local letter. With these stimuli patients are perfectly accurate at identifying the local components of the letters, yet fail to see the global letters (Dalrymple, et al., 2007; Karnath, et al., 2000). Interestingly, the perception of the global object can be improved by changing simple object properties such as the size of the global object, or the spacing of the object parts (Dalrymple, et al., 2007). This remarkable “local capture” behaviour has led to a proliferation of studies investigating global/local perception in Bálint syndrome with different modifications of these letter stimuli. For example I, along with my colleagues, modified the size and density of hierarchical letter stimuli to determine how changes to stimulus properties affect global processing in these patients (Dalrymple, et al., 2007). We found that as the letters got smaller, and as the local elements became more densely packed, the patient’s ability to identify the global letters improved. In another manipulation of hierarchical letters, Shalev et al (2007) found that familiar global letters made up of unfamiliar local letters were easier for patients to perceive than familiar global letters made up of equally familiar local letters. Shalev and her colleagues (2004) also found that first presenting a large solid letter that takes up the same spatial area as the global aspect of a hierarchical letter improved their patient’s ability to name the global level of that hierarchical letter. Again, global and local processing was influenced by simple manipulations of these simple stimuli. One failure in the literature has been to relate these simple behaviours back to the deficit of attention that underlies them. Much of the literature involves characterizing this isolated, and arguably artificially promoted, behaviour, yet falls short of using the behaviour to inform theory of the Bálint deficit. Shalev and Humphreys (2004) make some headway on this front by suggesting that the priming of the global level of a hierarchical stimulus helps improve global level perception by temporarily widening the restricted window of attention that is otherwise too small to facilitate processing of the larger level of the stimulus. One goal of the present thesis is to document and characterize global and local processing in Bálint syndrome, but  12  then, importantly, to follow Shalev and colleagues’ lead by investigating how these behaviours relate to disordered visual attention. Perception of simple hierarchical stimuli is explored in Part 1 of this thesis. In Study 1, I investigate the relative attentional demands of the processing of global and local levels of hierarchical stimuli of different sizes and densities to determine whether the global stimuli that patients with Bálint syndrome struggle with the most are more demanding of their limited attentional resources than the stimuli that they are more easily able to identify. This was done by recording electrical potentials (Event Related Potentials, ERPs) from the brains of healthy participants while they processed the global and local levels of these hierarchical letter stimuli. In Study 2, I investigate why patients3 are not able to piece together the identity of global letters through successive fixations to local elements. While it is clear that a restriction to the area of the spatial window of attention may preclude normal global processing, it is unclear why patients cannot piece together the global shape through perception of the local elements. In Study 3 I investigate the relationship between the restricted window of attention in Bálint syndrome and global processing difficulties by generating restricted window of vision in healthy subjects. Subjects were restricted to seeing only a small portion of the display through a computergenerated gaze-contingent window that is controlled through their eye movements. I investigate whether a literal restriction of vision can lead to similar difficulties with global processing as those resulting from the restriction of visual attention in Bálint syndrome. If so, this suggests a direct relationship between a restricted spatial extent of visual processing (whether attentional or artificial) and impaired global perception in Bálint syndrome. Novel to Part 1 of this thesis, therefore, is a direct attempt to explain disordered perception in Bálint syndrome, rather than simply characterizing it, through informed hypotheses about the attentional mechanisms that underlie the disorder. Beneficially, Part 1 is grounded in an established methodology that allows direct comparison to other findings in the literature.  3  See Appendix A for Case Reports of the patients who participated in the studies of this dissertation.  13  Modeling restricted attention through a restriction of vision It is important to emphasize that the visual deficits in Bálint syndrome are known to be attentional in nature (Holmes & Horrax, 1919). These patients have fully functioning retinas (Holmes & Horrax, 1919), and can see objects of different sizes (Rafal, 2001). What is not known is how that disorder of attention manifests itself. This dissertation applies the concepts of the disordered visual attention in Bálint syndrome that are arising in the literature. Concepts such as “shaft vision” (Bay, 1953; Tyler, 1968) and “viewing [a] picture through a diaphragm” (Bay, 1953, p. 545, 546), imply a spatial context to the restriction of attention. The aim is to test the validity of these concepts by testing whether they adequately capture the true nature of the attentional deficit. The chosen method for testing this is through modeling the attentional disorder in healthy individuals. There are obvious difficulties in experimentally creating a restricted attentional window in healthy individuals. As an alternative, and inspired by the literature, a simulation of the attentional disorder in Bálint syndrome was created through a literal restriction of vision in healthy individuals. This literal restriction of vision aims to create a representation of the visual experience of the patients, but also limits what can be attended by the individuals. That is, by limiting the spatial extent of what can be seen, this model in turn limits what can be attended to, i.e., attention is restricted to what is visible in the window. Therefore, in both patients and healthy individuals, we have a scenario of reduced visual information: in patients, this results from a neurological restriction of the attentional window; in healthy subjects this is created through an artificial restriction of the visual window (Figure 1.2). To reflect this important distinction, specific terminology will be used throughout this dissertation: when discussing patients, the limitation will be described as a restricted or limited window of visual attention; when discussing healthy individuals in the model groups, the limitation will be described as a restricted or limited window of vision. This latter choice of terminology is intended to emphasize that though modeling a disorder of attention, the healthy individuals in the model groups have normal visual attention.  14  Figure 1.2 Schematic of the rationale motivating the choice of a restricted window of vision as a model for a restricted window of attention.  Testing Bálint syndrome with complex stimuli With much focus on traditional experimental methods with controlled, impoverished stimuli, little work has been done to study Bálint syndrome in terms of the perception of more complex stimuli. Using real world objects, Rafal (2001) anecdotally reports that his patient RX struggled to see both a comb and a ruler at once. When the ruler and the comb were both in view, RX identified only the comb and did not know where the ruler had gone. Coslett and Saffran (1991) also anecdotally report of their patient that “she reported watching a movie in which, after a heated argument, she noted to her surprise and consternation that the character she had been watching was suddenly sent reeling across the room, apparently as a consequence of a punch thrown by a character she had never seen.” (p. 1525). One of the gold standard tests for Bálint syndrome is to ask patients to describe a line drawing of a woman and her children in a kitchen (i.e. the Boston Cookie Theft picture, Goodglass & Kaplan, 1983) and to document the patient’s verbal description of the drawing.  15  In a more systematic exploration of Bálint perception with complex stimuli, some researchers have monitored the eye movements of patients while they look at an image. Nyffeler et al (2005) found that their patients scanned uninformative regions of a schematic clock, failing to direct their attention to the hands of the clock and what numbers the hands point to, and rather simply looking at the numbers in succession. Tyler (1968) monitored the eye movements of a patient while she looked at a photograph of dolls. Tyler described her scanning behaviour, but stopped short of quantifying it. Again, very little has been done to extrapolate behaviour to the attentional mechanisms that underlie the behaviour. In Part 2 of this dissertation, I extend this work by monitoring the eye movements of patients while they view complex (social) scenes. Not only do I record where the patients are looking, but I quantify these results by determining to what regions of the scenes the patients allocate their fixations. In Study 4 I examine how the scanning behaviour of patients with Bálint syndrome compares to the scanning behaviour of healthy subjects, and of another brain-damaged patient who does not have Bálint syndrome. In Studies 5 and 6, I determine whether the unusual scanning patterns of the patients relate to their restricted window of attention by determining how healthy subjects viewing the scenes through a literal restricted window of vision scan the same scenes. Finally, in Study 7 I collect follow-up data from a patient several years after her initial testing to investigate how her scanning behaviour changes over time. Importantly, I try to determine how this relates to her restricted window of attention by manipulating the literal window of vision in the restricted viewing simulation. Thus, Part 2 of this thesis brings the investigation of Bálint syndrome to more complex and arguably more real-world stimuli, while still working towards the goal of elucidating the link between behaviour and visual attention.  Thesis overview In summary, the aim of this dissertation, which will follow a manuscript-based format, is to document the perception of patients with Bálint syndrome with simple and more complex tasks and then to determine how the behaviours relate to the  16  restriction of visual attention that is thought to characterize the disorder. It will address five main research questions. The 5 main research questions that will be addressed in this dissertation are: (1) Why are patients with Bálint syndrome impaired at piecing together local elements into a global whole? (2) Is the global processing deficit in Bálint syndrome directly related to the restriction of visual attention that is thought to underlie this disorder? (3) How do patients with Bálint syndrome scan complex stimuli? (4) Can a restricted window of visual attention account for more complex Bálint behaviours? (5) How does Bálint patient behaviour change with recovery and what can we learn from changes in behaviour over the course of recovery? Part 1 of this thesis involves the use of simple hierarchical letter stimuli, which are frequently used in research on Bálint syndrome because they are easily controlled and because patients show a distinct global processing deficit: they perform well at identifying the local level of these stimuli, but struggle to identify the global level (e.g. Dalrymple, et al., 2007; Karnath, et al., 2000). The use of these stimuli allows for a simple empirical measure of disturbed perception (i.e. global report accuracy). Study 1 asks why patients are impaired at naming the global level of hierarchical letters, and investigates this question by measuring the brain activity of healthy subjects while they perform a global/local letter identification task. The primary question specifically asks whether global level perception requires more attentional resources than local level perception, suggesting that patients struggle with the global level of stimuli because these stimuli exceed their depleted attentional resources. The findings show support for this hypothesis: healthy subjects show increased amplitude of an attention-related ERP component when 17  they respond to the global level of hard-to-group stimuli. Having established a basic explanation for impaired global perception in Bálint syndrome, Study 2 asks why patients are unable to piece together the global letter through successive intact perception of local elements. I hypothesized that patients employ a connect-the-dots strategy when trying to identify global stimuli and that any failure to identify the global form would be the result of a mistake in this connect-the-dot pattern. Instead the results show that patient eye movements are not predictive of accuracy on the task, suggesting that no overt strategy is being used. Study 3 asks how the global processing impairment relates to the restriction of visual attention that is said to characterize the disorder and for the first time introduces a new paradigm designed to “model” Bálint behaviour with these stimuli in healthy subjects. This model involves creating a computer-generated restricted viewing condition akin to the restricted window of attention in Bálint syndrome. If the restricted window of attention in Bálint syndrome is directly to blame for the global processing deficit, it could be possible to mimic this deficit in healthy individuals with a simple restricted viewing scenario. Results show a close match between Bálint behaviour and the behaviour of healthy individuals viewing hierarchical letters through a restricted window of attention. Not only does this suggest a link between a restricted spatial area of vision and Bálint behaviours, but it validates the model as a potentially useful way of studying Bálint syndrome. Part 2 of this thesis applies the same principals as Part 1, but with more complex stimuli: natural scenes. That is, Part 2 attempts to document Bálint behaviours and then test their relationship to the restricted window of attention through a model of this restriction with healthy subjects. Study 4 aims to document how patients with Bálint syndrome scan social scenes. Results show that patients allocate an abnormally low proportion of fixations to the eyes of people in social scenes. Studies 5 and 6 aim to investigate the link between the Bálint scanning behaviour and the restricted window of attention by using the restricted window paradigm of Bálint syndrome introduced in Study 3. Healthy participants viewing social scenes through a restricted window of vision also show reduced fixations to the eyes in social scenes, whether moving the window with their eyes (i.e. a “gaze18  contingent” window, Study 5) or with a computer mouse (i.e. a “mouse-contingent window”, Study 6). Study 7 tracks patient behaviour over time to see how it changes with recovery, and again uses the restricted viewing paradigm from Study 3 to try to mimic this recovery behaviour in healthy subjects. With recovery, Bálint patient SL allocates more fixations to the eyes of people in social scenes. This behaviour is closely modeled with a restricted window paradigm in healthy subjects by using a small- and large-size viewing window. This suggests that the patient’s window of attention may expand with recovery. Finally, in the General Discussion, I will discuss limitations of the window model of Bálint syndrome used throughout this dissertation, as well as findings from the Bálint literature that are not easily explained by a restricted window of attention. I will conclude by discussing the implications of this research in terms of theories of attention, the use of simple versus complex stimuli in visual attention research, social attention, neuroanatomy, and patient rehabilitation. I will conclude by discussing some avenues for future research.  19  References Bálint, R. (1909). Seelenlähmung des "Schauens", optische Ataxia, räumliche Störung der Aufmerksamkeit. Monatsschrift für Psychiatrie und Neurologie, 25, 51-81. Bay, E. (1953). Disturbances of visual perception and their examination. Brain, 76(4), 515-550. Baylis, G. (1994). Visual attention and objects: Two-object cost with equal convexity. Journal of Experimental Psychology: Human Perception and Performance, 20(1), 208-212. Baylis, G., & Driver, J. (1993). Visual attention and objects: Evidence for hierarchical coding of location. Journal of Experimental Psychology: Human Perception and Performance, 19(3), 451-470. Calvanio, R., Petrone, P., & Levine, D. (1987). Left visual spatial neglect is both environment-centred and body-centred. Neurology, 37, 1179-1183. Cheal, M., Lyon, D., & Gottlob, L. (1994). A framework for understanding the allocation of attention in location-precued discrimination. The Quarterly Journal of Experimental Psychology Section A, 47(3), 699-739. Clavagnier, S., Fruhmann Berger, M., Klockgether, T., Moskau, S., & Karnath, H.-O. (2006). Restricted ocular exploration does not seem to explain simultanagnosia. Neuropsychologia, 44, 2330-2336. Coslett, H., & Saffran, E. (1991). Simultanagnosia: To see but not two see. Brain, 114, 1523-1545. Critchley, M. (1953). The Parietal Lobes. London: Edward Arnold. Dalrymple, K., Kingstone, A., & Barton, J. (2007). Seeing trees OR seeing forests in simultanagnosia: Attentional capture can be local or global Neuropsychologia, 45, 871-875. Duncan, J. (1984). Selective attention and the organization of visual information. Journal of Experimental Psychology: General, 113(4), 501-517. Duncan, J., Humphreys, G., & Ward, R. (1997). Competitive brain activity in visual attention. Current Opinion in Neurobiology, 7, 255-261. Eason, R. (1981). Visual evoked potential correlates of early neural filtering during selective attention. Bulletin of the Psychonomic Society, 18(4), 203-206.  20  Eriksen, C., & Hoffman, J. (1973). The extent of processing of noise elements during selective encoding of visual displays. Perception and Psychophysics, 14, 155160. Farah, M. (1990). Visual agnosia: Disorders of object recognition and what they tell us about normal vision. Cambridge: MIT Press. Farah, M., Brunn, J., Wong, A., Wallace, M., & Carpenter, P. (1990). Frames of reference for allocating attention to space: Evidence from the neglect syndrome. Neuropsychologia, 28(4), 335-347. Friedman-Hill, S., Robertson, L., & Treisman, A. (1995). Parietal contributions to visual feature binding: Evidence from a patient with bilateral lesions. Science, 269(5225), 853-855. Goodale, M. A., & Miller, M. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15(1), 20-25. Goodglass, H., & Kaplan, E. (1983). The assessment of aphasia and related disorders (2nd ed.). Philadephia: Lea and Febiger. Haxby, J., Grady, C., Horwitz, B., Ungerleider, L., Mishkin, M., Carson, R., et al. (1991). Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proceedings of the National Academy of Sciences USA, 88, 1621-1625. Holmes, G., & Horrax, G. (1919). Disturbances of spatial orientation and visual attention, with loss of stereoscopic vision. Archives of Neurology and Psychiatry, 1, 385-407. Huberle, E., & Karnath, H.-O. (2006). Global shape recognition is modulated by the spatial distance of local elements - Evidence from simultanagnosia. Neuropsychologia, 44(6), 905-911. Humphreys, G. (1981a). Flexibility of attention between stimulus dimensions. Perception and Psychophysics, 30(3), 291-302. Humphreys, G. (1981b). On varying the span of visual attention: Evidence for two modes of spatial attention. Quarterly Journal of Experimental Psychology, 33A, 17-31. Humphreys, G., Romani, C., Olson, A., Riddoch, M., & Duncan, J. (1994). Nonspatial extinction following lesions of the parietal lobe in humans. Nature, 372(6504), 357-359.  21  Karnath, H.-O., Ferber, S., Rorden, C., & Driver, J. (2000). The fate of global information in dorsal simultanagnosia. Neurocase, 6, 295-306. Kim, M., & Robertson, L. (2001). Implicit representations of space after bilateral parietal lobe damage. Journal of Cognitive Neuroscience, 13(8), 1080-1087. Lynch, J., Mountcastle, V., Talbot, W., & Yin, T. (1977). Parietal lobe mechanisms for directed visual attention. Journal of Neurophysiology, 40(2), 362-389. Michel, F., & Henaff, M.-A. (2004). Seeing without the occipito-parietal cortex: Simultanagnosia as a shrinkage of attentional visual field. Behavioural Neurology, 15, 3-13. Mishkin, M. (1972). Cortical visual areas and their interactions. In A. Karczmar & J. Eccles (Eds.), Brain and Human Behaviour. Berlin: Springer. Moreaud, O. (2003). Bálint Syndrome. Archives of Neurology, 60, 1329-1331. Nyffeler, T., Pflugshaupt, T., Hofer, H., Bass, U., Gutbrod, K., von Wartburg, R., et al. (2005). Oculomotor behaviour in simultanagnosia: A longitudinal case study. Neuropsychologia, 43, 1591-1597. O'Craven, K., Downing, P., & Kanwisher, N. (1999). fMRI evidence for objects as the units of attentional selection. Nature, 401, 584-587. Posner, M., & Cohen, Y. (1984). Components of visual orienting. In H. Bouma & D. Bouwhuis (Eds.), Attention and performance (Vol. X). Hillsdale, NJ: Erlbaum. Posner, M., Nissen, M., & Ogden, W. (1978). Attended and unattended processing modes: The role of set for spatial location. In H. L. P. a. I. J. Saltzman (Ed.), Modes of Perceiving and Processing Information. Hillsdale, NJ: Lawrence Erlbaum Associates. Posner, M., Snyder, C., & Davidson, B. (1980). Attention and the detection of signals. Journal of Experimental Psychology: General, 109, 160-174. Rafal, R. (1997). Bálint Syndrome. In T. Feinberg & M. Farah (Eds.), Behavioural Neurology and Neuropsychology (pp. 337-356). USA: McGraw-Hill. Rafal, R. (2001). Bálint’s syndrome. In M. Behrmann (Ed.), Handbook of Neuropsychology (2nd ed., Vol. 4, pp. 121-141): Elsevier ScienceB.V. Rafal, R. (2003). Bálint’s syndrome: A disorder of visual cognition. In M. D'Esposito (Ed.), Neurological foundations of cognitive neuroscience (pp. 27-40). Cambridge, MA: MIT Press.  22  Riddoch, M., & Humphreys, G. (2004). Object identification in simultanagnosia: When wholes are not the sum of their parts. Cognitive Neuropsychology, 21, 423-441. Rizzo, M., & Vecera, S. (2002). Psychoanatomical substrates of Balint's syndrome. Journal of Neurology, Neurosurgery and Psychiatry, 72, 162-178. Shalev, L., Humphreys, G., & Mevorach, C. (2004). Global processing of compound letters in a patient with Bálint’s syndrome. Cognitive Neuropsychology, 22(6), 737-751. Shalev, L., Mevorach, C., & Humphreys, G. (2007). Local capture in Bálint’s syndrome: Effects of grouping and item familiarity. Cognitive Neuropsychology, 24(1), 115-127. Shulman, G., Remington, R., & McLean, J. (1979). Moving attention through visual space. Journal of Experimental Psychology: Human Perception and Performance, 5, 522-526. Thaiss, L., & de Bleser, R. (1992). Visual agnosia: A case of reduced attentional 'spotlight'? Cortex, 28(4), 601-621. Tipper, S., Brehaut, J., & Driver, J. (1990). Selection of moving and static objects for the control of spatially directed action. Journal of Experimental Psychology: Human Perception and Performance, 16, 492-504. Tipper, S., Driver, J., & Weaver, B. (1991). Short report: Object-centered inhibition of return of visual attention. Quarterly Journal of Experimental Psychology, 43A, 289-298. Tipper, S., Weaver, B., Jerreat, M., & Burak, A. (1994). Object-based and environment-based inhibition of return of visual attention. Journal of Experimental Psychology: Human Perception and Performance, 20, 478-499. Treisman, A., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97-136. Tsal, Y. (1983). Movements of attention across the visual field. Journal of Experimental Psychology: Human Perception and Performance, 9, 523-530. Tyler, R. (1968). Abnormalities of perception with defective eye movements (Bálint's Syndrome). Cortex, 4(2), 154-171. Ungerleider, L., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Engle, M. A. Goodale & R. Mansfield (Eds.), Analysis of visual behaviour (pp. 549-586). Cambridge, MA: MIT Press.  23  van Voorhis, S., & Hillyard, S. (1977). Visual evoked potentials and selective attention to points in space. Perception and Psychophysics, 22(1), 54-62. Wurtz, R., Goldberg, M., & Robinson, D. (1982). Brain mechanisms of visual attention. Scientific American, 246, 124-135.  24  Part 1: Impoverished stimuli  25  Chapter 21 Event-related potential evidence for a dual-locus model of global/local processing  1  A version of this chapter has been published. Dalrymple, KA, Kingstone A, & Handy, TC. (2009). Event-related potential evidence for a dual-locus model of global/local processing. Cognitive Neuropsychology, 26 (5), 456-470.  26  Our visual world is composed of an endless hierarchy of stimuli. Scenes are made up of objects, objects are made up of parts, and those parts are made up of features. To accommodate this hierarchy, the visual system is designed to engage the world at multiple levels, and to shift flexibly from one level to another: we can admire a forest at a global level, or shift our awareness to a single tree at a more local level. The dominant tool for investigating global/local processing in the laboratory has been hierarchical letter stimuli, which consist of a large, global letter, made up of several repetitions of a local letter (e.g. Figure 2.1) (Navon, 1977). This stimulus has allowed for some of the key questions regarding global/local processing to be addressed behaviourally. For example, manipulation of stimulus parameters such as size (e.g. Kinchla & Wolfe, 1979) and density (i.e. spacing between local elements) (Martin, 1979), have been shown to alter whether the global or local level of a stimulus appears to have a processing advantage, as indexed by reaction time measures (see Kimchi, 1992, for a review). Such data suggest that stimulus configuration can in fact exert an influence on the balance between global and local level processing.  Figure 2.1. Incongruent Large/Sparse and Small/Dense hierarchical letter stimuli.  27  Taking a novel approach to this issue, Enns and Kingstone (1995) investigated the perceptual time course of processing at global and local levels using a visual search paradigm with hierarchical elements. They used this task because in visual search, perceptual and post-perceptual processes are separable: the baseline measure of a distractor/RT cost function reflects perceptual processes that concern sensory, decision, and response factors; whereas the slope of the function represents post-perceptual processes, primarily the RT cost of committing attention to the distracting items and the time-consuming process of integrating visual items to form a coherent object (Neisser, 1967; Sternberg, 1969; Treisman & Gelade, 1980). Enns and Kingstone created a hierarchical visual search task by asking subjects to search among hierarchical objects for a target at either the global or local level of one of the objects. The size and density of the hierarchical stimuli were manipulated. Enns and Kingstone reasoned that if only early, perceptual, factors were involved in the processing of hierarchical stimuli, RT differences between global and local tasks would only be reflected in baseline measures (i.e. because baseline represents perceptual processes). However, if later, post-perceptual processes were involved differences should emerge in the slopes of the distractor/RT functions. The results showed that with regard to baseline measures, the search for local level items was faster for large and sparse stimuli, while global search patterns were faster for small and dense stimuli. This general pattern reflects the typical level precedence effect that occurs with traditional hierarchical stimuli (Kimchi, 1992). Inspection of the distractor/RT slopes revealed that local target detection was unaffected by stimulus size and density, i.e., the slopes were flat. Interestingly, size and density had a tremendous effect on slopes for the global target detection task, with search for the global targets of large and sparse configurations generating steeper slopes than search for global targets of small and dense configurations. Thus Enns and Kingstone (1995) demonstrated that though both global and local target detection are reflected in the early baseline measures, global target detection seems to be modulated by the post-perceptual attentional aspects of the task. In particular, as stimuli get larger and as inter-element spacing increases, the later post-perceptual demands of the global detection task increase. They proposed that 28  this additional demand represents attentional grouping processes that are not required when grouping occurs automatically, as is the case for identifying the global level of small/dense stimuli, or when grouping is not necessary at all, as is typically the case for identifying local elements. Critically, Enns and Kingstone supported this conclusion by demonstrating that if they created a unique situation where grouping is required at the local level of a stimulus, local slope measures are affected. They concluded that collectively the data supported the interpretation that as displays become increasingly large and sparse, active attentional grouping is required for global level processing. While the conclusions of Enns and Kingstone (1995) are convincing from a behavioural standpoint, their inferences about the attentional demands of the task were drawn from indirect, behaviour-based measures of stimulus processing. Accordingly, our goal here was to re-examine the validity of this assumption by testing whether direct measures of cortical stimulus processing accord with the predictions of Enns and Kingstone. Specifically, we recorded event-related potentials (ERPs) from healthy participants while they made a forced-choice judgment about the identity of either the global or local level of hierarchical letters. At issue was whether ERPs would show evidence that identifying the global level of widely spaced stimuli requires “active” attentional grouping processes, relative to global identification of small/dense stimuli and the identification of local elements.  Method Participants Subjects (n = 12, 3 female) were all right-handed and ranged in age from 21 to 29 years (mean = 24.1 years). All had normal or corrected-to-normal vision. Participants received monetary compensation for their time, and were tested under protocols approved by the Clinical Review Ethics Board of the University of British Columbia.  29  Stimuli and procedures The same hierarchical stimuli as those used by Dalrymple et al (2007) were used in this study, but only the extreme stimuli were chosen in attempts to elicit the strongest effects: large, sparse stimuli that elicited the slowest Global level reaction times, and small, dense stimuli that led to the fastest Global level reaction times in healthy subjects (Figure 2.1). Upper-case Global hierarchical letters N and P were created using repetitions of smaller upper case letters N and P, creating 2 Congruent letters, and 2 Incongruent letters. Global letters of two different Configurations were constructed. Large/Sparse Global letters were made up of 1213 local letters, and measured approximately 12.5 x 13.5 degrees of visual angle. Adjacent local letters were separated by 2.5 degrees of visual angle. Small/Dense Global letters were made up of 47-49 local letters and measured approximately 4.0 x 4.5 degrees of visual angle. Adjacent local letters were separated by 0.01 degrees of visual angle. All letters were black on a white screen. Subjects were seated 57 cm in front of a 17” computer monitor. Each task trial began with the onset of a black fixation dot at the centre of the screen. Between 2700-2900ms after onset of fixation, a hierarchical letter appeared for 100ms and was then replaced with a 300ms visual mask. This mask was replaced by the fixation dot in preparation for the next trial. The task was to discriminate the letter at the designated stimulus level (Global or Local) by button press on a video game controller. Subjects performed all trials at one level before switching to the other, with the order counterbalanced between subjects. They were asked to respond as quickly and accurately as possible while behavioural performance was measured in terms of reaction time (Table 2.1) and accuracy of response (correct vs. incorrect). Trials were blocked by Level (Global or Local) and Configuration (Small/Dense or Large/Sparse). There were 10 trials for each of the 4 letter stimuli, resulting in 40 trials per block. Trials within each block were randomized. There were 5 blocks for each of the two letter Configurations, and each block was performed for Global and Local level responses for a total of 20 blocks and 800 trials.  30  Configuration Small/Dense Large/Sparse  Congruency Congruent Incongruent Congruent Incongruent  Level of Processing Global 489 (±26.8) 488 (±25.2) 522 (±30.5) 565 (±26.6)  Local 530 (±22.8) 586 (±23.3) 512 (±28.4) 526 (±29.2)  Table 2.1. RTs (± standard errors) as a function of stimulus Configuration, Congruency of Global and Local elements, and Level of processing (in ms).  Electrophysiological recording Scalp potentials were recorded from 24 tin electrodes that were evenly distributed across the scalp according to standard 10-20 method of electrode placement and mounted in a custom elastic cap. All electroencephalographic (EEG) activity was recorded relative to the left mastoid, amplified (Grass Instruments, Model 12 Neurodata Acquisition System) with a band-pass of 0.1-30 Hz (1/2 amplitude cutoffs), and digitized on-line at a sampling rate of 256 samples-persecond. To ensure proper eye fixation, vertical and horizontal electro-oculograms (EOGs) were also recorded, the vertical EOG from an electrode inferior to the right eye, and the horizontal EOG from an electrode on the right outer canthus. All electrode impedances were kept below 5 kΩ. Off-line, computerized artifact rejection was used to eliminate trials during which detectable eye movements (> 1°), blinks, muscle potentials, or amplifier blocking occurred. Rejection itself was based on exceeded min-max difference thresholds within a -200 to 600 ms time window around each event (for eye and muscle artifacts), with each participant’s threshold scaled via data visualization to the ambient level of that participant's EEG noise, or 50 contiguous data points with a constant voltage (for amplifier blocking). Following signal-averaging of the EEG, for each subject ERP waveforms were algebraically rereferenced to the average of the left and right mastoid signals, and filtered with a low-pass Gaussian filter (25.6 Hz half-amplitude cut-off) to eliminate high-frequency artifacts in the waveforms (for details on average number of trials rejected per condition, see below).  31  Results When using hierarchical letter stimuli, the Congruent condition presents the same letter at both the Global and Local level and thus it remains ambiguous on these trials as to whether the participant is actually responding to the target level of the stimulus. As a consequence, while initial omnibus analyses of both behavioural and ERP results includes Congruency as a factor of interest, subsequent comparisons were planned a priori to specifically examine data from Incongruent trials only. Behaviour Overall accuracy was at ceiling (99.14%), so this measure was not analyzed further. All reaction times (RTs) are presented in the Table 2.1. An omnibus repeated-measures ANOVA was conducted with factors of Congruency between Global and Local elements (Congruent vs. Incongruent), processing Level (Global vs. Local target), and stimulus Configuration (Large/Sparse vs. Small/Dense). This analysis revealed a significant main effect of stimulus Congruency, F(1, 11) = 19.79, p = 0.001, indicating that overall RTs were longer on Incongruent relative to Congruent trials, and a significant main effect of processing Level, F(1, 11) = 16.04, p = 0.002, indicating that overall RTs were faster for Global level discriminations relative to Local level discriminations. There were also two significant interactions. A 2-way interaction between processing Level and stimulus Configuration, F(1, 11) = 39.24, p < 0.001, indicated that while overall RTs were slower for the Small/Dense Configuration in the Local relative to Global processing condition, overall RTs were slower for the Large/Sparse configuration in the Global relative to Local processing condition. A significant 3-way interaction across all factors, F(1, 11) = 46.91, p < 0.001, indicated that while there was a Congruency effect at both levels of target processing for the Large/Sparse configuration, there was an absence of a Congruency effect for the Small/Dense configuration at the Global level of target processing. To examine the RTs specifically within the Incongruent conditions, a second repeated-measures ANOVA was performed with factors of processing Level (Global 32  vs. Local target) and stimulus Configuration (Large/Sparse vs. Small/Dense). We found a significant main effect of processing Level, F(1, 11) = 30.51, p < 0.001, with overall RTs being faster when discriminating the Global level of the letter stimuli compared to the Local level. There was also a significant interaction between processing Level and stimulus Configuration, F (1,11) = 52.76, p < 0.001, indicating that RTs were faster when discriminating the Global level of Small/Dense letters, but faster when discriminating the Local level of Large/Sparse letters. There was no effect of Configuration, F(1,11) = 0.72, p = 0.41. In sum these RT results confirmed that in the Incongruent condition, the Large/Sparse stimuli produced a Local level processing bias, whereas the Small/Dense stimuli produced a Global processing bias. Having established that these stimulus manipulations produced the desired behavioural effects, we next evaluated our ERP data. ERPs Analysis of the ERP data converged a priori on those visual components sensitive to attentional modulation: the lateral occipital P1, N1, N2 and the parietocentral midline P3 (see Coles & Rugg, 1995). The P1 amplitude typically increases with visual spatial attention, and reflects the magnitude or intensity of sensoryevoked responses in visual cortex (e.g. Mangun & Hillyard, 1991). In contrast, the N1 and N2 amplitudes positively correlate with perceptual difficulty (e.g. Senkowski & Herrmann, 2002; Vogel & Luck, 2000) and we therefore expected the amplitudes of these components to fluctuate with reaction times. Finally, the P3 has been identified as indexing the processing demands of a task, with modulations of the P3 thought to be related to post-perceptual attentional allocation (e.g. Kramer & Strayer, 1988). If Enns and Kingstone's (1995) hypothesis is correct regarding active grouping, it predicted that this component should be specifically sensitive to the Global processing demands of Large/Sparse stimuli. Statistical analyses of the P1 and N1 ERP data were based on peak amplitude measures taken at lateral occipital scalp sites T5/T6 and OL/OR (relative to a -200-0 ms pre-stimulus baseline), the scalp locations where the P1 and N1 have previously been shown to be maximal (e.g. Mangun & Hillyard, 1991). The latency of  33  these components was first identified in the grandaveraged waveform for each electrode (T5/T6, and OL/OR) and condition of interest. For the data points entered into the ANOVA, the amplitude of the single-subject waveforms were then measured within each electrode and condition at the corresponding peak latency identified in the grandaveraged waveform. Because the N2 component had a temporallybroader effect at lateral occipital electrode sites T5/T6 and OL/OR relative to the P1 and N1, this component was measured across all electrodes and conditions via a mean amplitude measure taken over a 50 ms time window beginning at 200 ms post-stimulus. Likewise, the P3 was measured as a mean amplitude across a 375575 post-stimulus time window at midline electrode sites PZ and CZ, where the P3 (or P3b) is maximal (Coles & Rugg, 1995). All component measurements and waveform displays were scaled relative to a -200 to 0 pre-stimulus baseline.  Omnibus ANOVAs. The ERP waveforms for the P1, N1, and N2 components at lateral occipital electrode sites are shown in Figure 2.2 and the P3 waveforms at midline parietal/central sites are shown in Figure 2.3 (a), both as a function of stimulus Configuration, Congruency and processing level. Within these waveforms, we first analyzed each component separately via omnibus repeated-measures ANOVAs that included our central factors of interest: stimulus Configuration (Small/Dense vs. Large/Sparse), stimulus Congruency (Global and Local letters Congruent vs. Incongruent) and target processing level (Local vs. Global target). For the P1, N1, and N2 components factors of electrode pair (OL/OR vs. T5/T6) and hemisphere of electrode site (left vs. right) were also included in the ERP analyses; for the P3, electrode site (PZ vs. CZ) was included as a factor. Notably, there were no systematic effects of these electrode-related factors in the analyses reported below and thus for brevity and clarity we only include the stimulus- and task-related factors in our presentation of the statistical results. The effects reported below are highlighted in Figure 2.4 (for the P1, N1, and N2) and Figure 2.3(b) (for the P3), figures which plot the mean amplitudes of these ERP components as a function of stimulus Configuration, target Congruency, and processing Level, collapsed across electrodes in the corresponding statistical analyses. 34  Figure 2.2. Group-averaged waveforms for P1, N1, and N2 ERP components. Shown are data from lateral occipital-temporal sites OL/OR and T5/T6 as a function of stimulus Configuration and Congruency. In each plot the ERP responses are compared between whether the Global or Local level of the stimulus was processed.  35  Figure 2.3. P3 data. (a) The group-averaged waveforms for the P3 from midline parietal (PZ) and central (CZ) electrode sites, as a function of stimulus Configuration and Congruency. In each plot the ERP responses are compared between whether the Global or Local level of the stimulus was processed. (b) The mean amplitude of the P3 collapsed across CZ and PZ, as a function of stimulus Configuration and Congruency and whether the Global or Local level of the stimulus was processed. This plot highlights the effect of processing level for the Large/Sparse stimulus configuration only.  36  Figure 2.4. The mean amplitudes of the P1, N1 and N2 components. Data are collapsed across lateral occipital electrode sites OL, OR, T5, and T6. These plots highlight that while there was no significant effects found in the P1 (a), the N1 (b) and N2 (c) both showed an increased amplitude when processing the Local vs. Global level of the Small/Dense configuration (left), but an increased amplitude when processing the Global vs. Local level of the Large/Sparse configuration (right). Note: the data for the N2 were normalized such that the smallest value (Local processing of incongruent targets of the Large/Sparse configuration, far right) was set to 0 and all other values scaled relative to this baseline. In this manner the magnitude of differences between conditions was preserved while allowing the data to be plotted such that increased bar height corresponds to a larger N2 amplitude.  37  For the P1 (Table 2.2), there were no significant main effects or interactions across any of these factors (all Fs (1,11) < 2.38, all ps > 0.15); as a consequence, no further considerations of the P1 data were made. Configuration  Congruency Congruent  Small/Dense Incongruent Congruent Large/Sparse Incongruent  Level Global Local Global Local Global Local Global Local  T5 1.31 (0.57) 1.52 (0.41) 1.03 (0.76) 1.54 (0.48) 1.43 (0.45) 1.33 (0.48) 1.27 (0.43) 1.71 (0.50)  Electrode OL T6 3.73 (0.99) 2.63 (0.66) 3.47 (0.56) 2.27 (0.33) 3.75 (0.82) 2.73 (0.33) 3.94 (0.68) 2.58 (0.42) 2.09 (0.66) 1.66 (0.35) 2.02 (0.80) 1.20 (0.43) 2.23 (0.96) 1.70 (0.49) 3.02 (0.83) 2.17 (0.29)  OR 3.00 (1.08) 3.04 (0.76) 2.48 (1.03) 3.53 (0.83) 2.73 (0.73) 2.67 (0.71) 2.47 (0.77) 3.36 (0.84)  Table 2.2. Peak P1 amplitude (± standard errors) as a function of stimulus Configuration, Congruency of Global and Local elements, Level of processing, and scalp electrode location (in µv) For the N1 (Table 2.3), a significant main effect of stimulus Configuration was observed, F(1,11) = 24.16, p = 0.0005, indicating that overall peak N1 amplitudes were greater for the Large/Sparse relative to Small/Dense stimuli. There was also a significant 3-way interaction between Configuration, Congruency, and processing Level, F (1,11) = 33.61, p = 0.0001, indicating that while both stimulus configurations manifest an effect of processing Level in the Incongruent but not Congruent conditions, the effect of processing Level went in opposite directions: for the Large/Sparse configuration, the peak N1 amplitude in the Incongruent condition was greater when processing targets at the Global relative to Local level, while for the Small/Dense configuration, the peak N1 amplitude in the Incongruent condition was greater when processing targets at the Local relative to Global level.  38  Configuration  Congruency Congruent  Small/Dense Incongruent Congruent Large/Sparse Incongruent  Level Global Local Global Local Global Local Global Local  T5 -1.91 (1.63) -1.97 (1.62) -1.47 (1.58) -2.10 (1.63) -4.05 (1.68) -3.99 (1.48) -3.96 (1.60) -3.43 (1.56)  Electrode OL T6 -4.30 (1.65) -1.77 (1.21) -4.81 (1.68) -1.95 (1.29) -3.62 (1.70) -1.32 (1.24) -5.13 (1.71) -2.63 (1.27) -9.45 (2.68) -5.31 (1.75) -9.03 (2.56) -5.16 (1.70) -10.10 (2.92) -5.73 (1.98) -8.16 (2.62) -4.38 (1.72)  OR -5.87 (1.77) -6.73 (1.84) -5.40 (1.79) -6.43 (2.02) -9.48 (2.15) -9.35 (2.06) -9.54 (2.16) -8.50 (2.13)  Table 2.3. Peak N1 amplitude (± standard errors) as a function of stimulus Configuration, Congruency of Global and Local elements, Level of processing, and scalp electrode location (in µv).  For the N2 (Table 2.4), there was significant 2-way interaction between processing Level and stimulus Congruency, F (1,11) = 7.09, p = 0.022, indicating that the mean amplitude of the N2 showed a greater effect of processing Level in the Incongruent relative to Congruent stimulus conditions. There was also a significant 2-way interaction between stimulus Configuration and processing Level, F(1,11) = 11.72, p < 0.0057, indicating that the overall mean N2 amplitude was larger during Global relative to Local processing for Large/Sparse stimuli, but was larger during Local relative to Global processing for Small/Dense stimuli. Finally, there was a 3way interaction between Configuration, Congruency, and processing Level, F(1,11) = 17.31, p = 0.0016, such that the effect of processing Level extended to both the Congruent and Incongruent conditions for the Large/Sparse stimuli, but was limited to the Incongruent only condition for the Small/Dense stimuli.  39  Configuration  Congruency Congruent  Small/Dense Incongruent Congruent Large/Sparse Incongruent  Level Global Local Global Local Global Local Global Local  T5 2.96 (0.97) 1.92 (0.90) 3.32 (1.04) 2.07 (1.05) 2.02 (1.12) 2.24 (0.89) 1.48 (1.00) 2.46 (0.94)  Electrode OL T6 7.27 (1.72) 6.61 (1.02) 5.75 (1.39) 5.79 (0.92) 7.90 (1.86) 6.43 (1.08) 5.38 (0.94) 5.53 (0.94) 6.86 (1.26) 5.65 (0.92) 8.06 (1.67) 6.44 (1.21) 5.33 (1.27) 4.48 (0.80) 9.09 (1.66) 7.26 (1.16)  OR 6.09 (1.77) 2.88 (1.49) 6.11 (1.68) 3.01 (1.49) 4.35 (1.65) 4.66 (1.51) 3.09 (1.57) 5.23 (1.57)  Table 2.4. Mean N2 amplitude (± standard errors) as a function of stimulus Configuration, Congruency of Global and Local elements, Level of processing, and scalp electrode location (in µv). Mean amplitude was measured over a 200-250 time window post-stimulus.  For the P3 (Table 2.5), there was a significant main effect of Congruency, F(1,11) = 26.28, p = 0.0003, indicating an overall larger P3 amplitude for Incongruent relative to Congruent trials. There was also a significant 2-way interaction between stimulus Configuration and processing Level, F (1,11) = 6.77, p = 0.0246, suggesting that while the P3 was larger in amplitude during Global relative to Local processing for the Large/Sparse configuration, this effect was diminished/absent for the Small/Dense configuration.  Configuration  Congruency Congruent  Small/Dense Incongruent Congruent Large/Sparse Incongruent  Level Global Local Global Local Global Local Global Local  Electrode PZ CZ 9.66 (1.32) 9.12 (1.48) 9.48 (1.22) 7.95 (1.25) 10.00 (1.57) 9.96 (1.61) 11.34 (1.47) 9.92 (1.57) 11.96 (1.56) 11.57 (1.84) 10.21 (1.52) 10.12 (1.64) 12.41 (1.65) 12.24 (1.99) 10.67 (1.34) 9.80 (1.34)  Table 2.5. Mean P3 amplitude (± standard errors) as a function of stimulus Configuration, Congruency of Global and Local elements, Level of processing, and scalp electrode location (in µv). Mean amplitude was measured over a 375-575 time window post-stimulus.  40  Based on these initial omnibus results, we wanted to then examine the specific pattern of effects underlying the significant 3-way interactions observed in both the N1 and N2 components. In particular, we wanted to determine if the effects were consistent with increased perceptual/discriminative difficulty when discriminating the Small/Dense letters at a Local level and the Large/Sparse letters at a Global level. Likewise, we also wanted to examine the 2-way interaction observed in the P3 component in order to confirm whether there was an effect of processing Level for the Large/Sparse stimulus Configuration but not the Small/Dense configuration. Accordingly, we thus performed separate statistical analyses on the N1, N2 and P3 within each of the two stimulus configuration levels.  ERPs: Small/Dense configuration. Repeated-measures ANOVAs were performed on the N1, N2, and P3 components generated by the Small/Dense letters that included factors of Congruency (Global and Local letters Congruent vs. Incongruent) and processing Level (Local vs. Global target). For the N1, there was a significant interaction between stimulus Congruency and processing Level, F(1,11) = 16.13, p = 0.002, such that the peak N1 amplitude was greater in the Incongruent condition when processing targets at the Local relative to Global level. For the N2, there was a main effect of processing Level, F(1,11) = 9.55, p = 0.01, indicating that the mean amplitude of the N2 was consistently larger when processing targets at the Local relative to Global level, regardless of Congruency condition. Taken together, these results were thus consistent with significant increases in N1 and N2 amplitude for the Small/Dense letters, specifically when performing a Local discrimination with Incongruent hierarchical letters. For the P3, while there was a significant main effect of Congruency, F(1,11) = 12.37, p < 0.001, indicating that the P3 amplitude was larger for Incongruent relative to Congruent targets, there was no main effect of processing Level nor an interaction between Congruency and processing Level.  ERPs: Large/Sparse configuration. Repeated-measures ANOVAs were also performed on the N1, N2, and P3 components generated by the Large/Sparse  41  letters, ANOVAs that included factors of Congruency (Global and Local letters Congruent vs. Incongruent) and processing Level (Local vs. Global target). For the N1, there was again a significant interaction between Congruency and processing Level, F(1,11) = 11.19, p = 0.007, such that the peak N1 amplitude was greater in the Incongruent condition when processing targets at the Global relative to Local level. For the N2, there was again a significant main effect of processing Level, F(1,11) = 6.17, p = 0.03, indicating that the mean amplitude of the N2 was consistently larger when processing targets at the Global relative to Local level, regardless of Congruency condition. There was also a significant interaction between Congruency and processing Level, F(1,11) = 22.11, p = 0.0006, revealing that this effect of processing Level was greater under Incongruent relative to Congruent target conditions. In short, these results were again consistent with significant increases in N1 and N2 amplitude for the Large/Sparse letters, specifically when performing a Global discrimination with Incongruent hierarchical stimuli. For the P3, there was a significant main effect of processing Level, F(1,11) = 9.97, p = 0.0091, but no main effect of Congruency or an interaction between processing Level and Congruency.  ERPs: Direct comparison of Incongruent conditions. The foregoing analyses converge on the conclusion that the significant interactions between processing Level and Congruency in N1 and N2 amplitude were specifically associated with Incongruent trials. That is, the N1 and N2 appeared to systematically decrease in amplitude when processing (1) Global relative to Local letters in the Incongruent condition of the Small/Dense configuration and (2) Local relative to Global letters in the Incongruent condition of the Large/Sparse configuration. Given this differential direction of processing Level effect between the two stimulus conditions, it predicted that for Incongruent trials there should be a significant interaction in both the N1 and N2 between these two factors. To confirm this prediction, we analyzed the N1 and N2 on Incongruent trials only via repeated-measures ANOVAs with factors of processing Level (Global vs. Local) and stimulus Configuration (Small/Dense vs. Large/Sparse). For the N1, we found a significant interaction between processing 42  Level and stimulus Configuration, F(1,11) = 10.62, p = 0.008; there was also a significant main effect of stimulus Configuration, F(1,11) = 24.02, p < 0.001. The interaction was further investigated with separate 1-way ANOVAs for Large/Sparse and Small/Dense letters, respectively, with factor of processing Level. For Large/Sparse letters there was a significant effect of processing Level, F(1,11) = 8.16, p = 0.0156, indicating that the N1 amplitudes were smaller for Local compared to Global level discriminations. For Small/Dense letters there was a trend to significance, F(1,11) = 4.37, p = 0.0605, indicating that the N1 amplitudes were marginally smaller for the Global compared to the Local level discriminations. For the N2 there was a significant interaction between processing Level and stimulus Configuration, F(1,11) = 15.87, p = 0.0021. The interaction was further investigated with separate 1-way ANOVAs for Large/Sparse and Small/Dense letters, respectively, with factor of processing Level. For Large/Sparse letters there was a significant effect of processing Level, F(1,11) = 11.13, p = 0.0066, indicating that the N2 amplitudes were smaller for Local compared to Global level discriminations. For Small/Dense letters there was also a significant effect of processing Level, F(1,11) = 11.95, p = 0.0054, indicating that the N2 amplitudes were smaller for the Global compared to the Local level discriminations. Together, these results indicate that there is indeed a systematic decrease in amplitude for these components for the Local relative to Global letters in the Incongruent condition of the Large/Sparse configuration and, conversely, Global relative to Local letters in the Incongruent condition of the Small/Dense configuration. This interaction parallels the RT data such that the decrease in amplitudes occurs for the conditions for which subjects had the smaller RTs. We performed similar analyses for the P3, again with factors of processing Level and stimulus Configuration. Here we found a significant interaction between processing Level and stimulus Configuration, F(1,11) = 7.42, p = 0.0198. This interaction was further investigated with separate 1-way ANOVAs for each Configuration, with factor of processing Level. While there was no effect of processing Level for the Small/Dense configuration, F(1,11) = 0.60, p = 0.4553, 43  there was a significant effect of processing Level for the Large/Sparse configuration, F(1,11) = 6.03, p = 0.032, indicating that for Large/Sparse letters alone there was a larger P3 amplitude for Global relative to Local level discriminations.  Discussion Our experiment was designed to investigate the perceptual time course of global/local processing, and in particular, to test the validity of the conclusion drawn by Enns and Kingstone (1995) that identifying the global level of widely spaced hierarchical stimuli requires greater “active” attentional grouping processes relative to either the global identification of small/dense stimuli or the identification of local elements. In this regard, we found two critical results. First, we found that modulations of the early N1 and N2 ERP components correlated with behavioural RT effects: N1/N2 amplitudes were smaller when processing the level of the stimulus that subjects respond to more quickly. Specifically, there were smaller N1/N2 amplitudes when subjects responded to (1) the local level of large/sparse stimuli and (2) to the global level of small/dense stimuli. Second, and more importantly, the amplitude of the later P3 component was selectively increased for global level discriminations of large/sparse stimuli only, with larger peak amplitudes for global discriminations of these hard-to-group items versus global discriminations of small/dense stimuli, and any local level discriminations. This result is again consistent with the behavioural results of Enns and Kingstone (1995). In particular, they showed that the baseline search times for both local and global targets are affected by size and density manipulations, while the distractor/RT slope is only affected by these manipulations for conditions where active grouping is required; for example, search for the global targets of large/sparse configurations generates a steeper slope than search for global targets of small/dense configurations. Enns and Kingstone proposed that the baseline measure for a visual search represents early, perceptual, processing of targets, while the distractor/RT slope represents later post-perceptual processing relating to attention-mediated grouping. Thus, they suggest that while both global and local processing occur at an early stage (as also suggested by our N1/N2 data), active grouping requires 44  additional later stage processing, which, in the present case, occurs when global discriminations are made about targets that are large and made up of widely spaced elements that need to be effortfully grouped into a coherent whole. Thus, Enns and Kingstone’s conclusion precisely aligns with our finding here that P3 amplitude was selectively increased for these specific stimuli. Our P3 findings are consistent with other ERP findings from experiments using hierarchical letters. For example, Volberg and Hübner (2004) found overall increased amplitudes of the P3 when participants responded to the global, compared to the local, level of hierarchical letters. Although it is difficult to make cross-study comparisons, the stimuli in the Volberg and Hübner experiment had larger inter-element spacing than those used in our small/dense condition, suggesting the possibility that their stimuli, like our large/sparse stimuli, required active grouping to be processed at a global level. Also consistent with our results, Volberg and Hübner found that their effects were only present with incongruent stimuli, highlighting the fact that conflict between global and local levels are crucial for these effects. These authors suggest that when stimuli are congruent, letter identity alone can be processed. However, when stimuli are incongruent, target level must also be processed, leading to modulation of the P3 component with target level. Given that the P3 has been identified as indexing post-perceptual attentional allocation (e.g. Kramer & Strayer, 1988), it is possible that modulations of this component reflect other attentionally demanding processes beyond active grouping. One excellent suggestion is that the P3 reflects the active suppression of the more salient, but irrelevant, level of the stimulus (i.e. the local level of large/sparse stimuli when global judgments are made, and the global level of small/dense stimuli when local judgments are made). If this were the case, one would predict modulation of the P3 much like the modulation seen here with the N1/N2 components, that is, an interaction between processing level and stimulus configuration such that there is an enhancement of the P3 component corresponding to judgments about the level that does not take precedence. While this possibility is extremely reasonable at a  45  theoretical level, it is not supported by the current data set. We see no modulation of the P3 for judgments at either the global or local level of small/dense (easy to group) stimuli, yet we do see an enhancement of the P3 for global judgments only for large/sparse stimuli (hard to group). Thus, while it is still possible that suppression of the more salient, but irrelevant level of the stimulus is an active, attentionally demanding process, this proposal is not captured or reflected by the current ERP measures2. It is worthwhile considering whether other hypotheses regarding the role of the P3 are supported by our results. Verleger, Jaskowski, and Wascher (2005) suggest that the P3b reflects a process of monitoring the transformation of a response decision into action. This possibility can be evaluated in light of the present results because our task requires participants to make a decision regarding the identity of the target letter and to transform that decision into an appropriate manual response (i.e. a button press). Our results show an enhancement of the P3 to global level judgments of large/sparse stimuli alone. It is unclear why this condition would recruit monitoring of the transformation of stimulus classification to action while global level judgments of small/dense stimuli, and local level judgments in general, would not. Contrary to our results, Verleger et al’s interpretation of the P3 would predict equal enhancement of the P3 regardless of target level or stimulus configuration. Thus a post-perceptual attentional allocation account of the P3 component is still preferred in light of the present data. These data not only speak to the perceptual time course of the processing of hierarchical stimuli in healthy subjects, but they also have implications for special neurological populations. Simultanagnosia is a rare neuropsychological disorder that results in a restriction of visual attention such that the patient is only aware of a single object at any one time (Moreaud, 2003; Rafal, 2003; Rizzo & Vecera, 2002),  2  Thank you to L. Shalev for pointing out a second alternative interpretation of our results, that the P3 effect reflects the controlled spreading of attention across a large area. This is an interesting possibility for future research, which could be tested by comparing P3 modulation in response to our large/sparse stimuli to P3 modulation in response to new, large/dense, stimuli.  46  yet paradoxically, patients are often unable to see single objects when those objects are made up of smaller local elements (Dalrymple, et al., 2007; Karnath, Ferber, Rorden, & Driver, 2000). Many behavioural studies have shown that hierarchical stimuli are not processed normally by patients with simultanagnosia, with patients processing only the local level of some stimuli (Clavagnier, Fruhmann Berger, Klockgether, Moskau, & Karnath, 2006; Dalrymple, et al., 2007; Huberle & Karnath, 2006; Karnath, et al., 2000; Shalev, Mevorach, & Humphreys, 2007). Previously this effect has been interpreted in terms of local level processing: some researchers have suggested that simultanagnosics are unable to process the global level of stimuli because they are “captured” by the local level of those stimuli (Karnath, et al., 2000) or that patients may have trouble disengaging attention from the local elements in order to process the global level of the stimulus (Clavagnier, et al., 2006). It is now clear that patients actually produce a fair amount of exploratory eye movements when viewing these stimuli and that a disengage deficit is unlikely (Clavagnier, et al., 2006). Our data further this by suggesting global processing itself may be disrupted due to the high attentional demands of active grouping. What is the role of the local elements in grouping? With such a complex relationship between global and local processing, the local elements of these stimuli may influence the ease at which grouping occurs. For example, Shalev, Mevorach, and Humphreys (2007) found that familiarity with the local elements of hierarchical stimuli influenced global level perception in simultanagnosic patient GK. GK performed well at naming the global level of a familiar global shape (English letter) when he was unfamiliar with the local elements that made up that shape (Hebrew letters). However, when GK was trained to learn Hebrew letters such that the local elements were familiar to him, his performance on the global English letters declined. Thus while attentionally demanding global level grouping may underlie the difficulties with global level processing in simultanagnosia, grouping itself may be influenced by the identity or properties of the local components. Indeed, as Enns and  47  Kingstone (1995) show, when grouping is required at the local level of the stimulus, attention is required there too. In conclusion, our results support a dual-locus model of global/local processing. We show early modulation of the N1/N2 that mirrors behavioural reaction time and level precedence effects with hierarchical stimuli. At a later stage of processing, the P3 component, which indexes post-perceptual attentional allocation, shows an enhancement during active grouping of the global stimulus. These findings support the conclusions drawn by Enns and Kingstone (1995), that active grouping is an attentionally demanding process. These findings speak to visual processing in healthy subjects, but also carry implications for neuropsychological populations; for instance, impaired grouping abilities may underlie the global level processing deficit in patients with simultanagnosia because these grouping processes may place demands on an already limited attentional system.  48  References Clavagnier, S., Fruhmann Berger, M., Klockgether, T., Moskau, S., & Karnath, H.-O. (2006). Restricted ocular exploration does not seem to explain simultanagnosia. Neuropsychologia, 44, 2330-2336. Coles, M., & Rugg, M. (1995). Event-related potentials: An introduction. In M. Rugg & M. Coles (Eds.), Electrophysiology of mind: Event-related brain potentials and cognition. New York: Oxford University Press. Dalrymple, K., Kingstone, A., & Barton, J. (2007). Seeing trees OR seeing forests in simultanagnosia: Attentional capture can be local or global Neuropsychologia, 45, 871-875. Enns, J., & Kingstone, A. (1995). Access to global and local properties in visual search for compound stimuli. Psychological Science, 6, 283-291. Huberle, E., & Karnath, H.-O. (2006). Global shape recognition is modulated by the spatial distance of local elements - Evidence from simultanagnosia. Neuropsychologia, 44(6), 905-911. Karnath, H.-O., Ferber, S., Rorden, C., & Driver, J. (2000). The fate of global information in dorsal simultanagnosia. Neurocase, 6, 295-306. Kimchi, R. (1992). Primacy of wholistic processing and Global/Local paradigm: A critical review. Psychological Bulletin, 112(1), 24-38. Kinchla, R., & Wolfe, J. (1979). The order of visual processing: "Top-down," "bottomup," or "middle-out". Perception and Psychophysics, 33, 1-10. Kramer, A., & Strayer, D. (1988). Assessing the development of automatic processing: An application of dual-task and event-related brain potential methodologies. Biological Psychology, 26, 231-267. Mangun, G. R., & Hillyard, S. A. (1991). Modulations of sensory-evoled brain potentials indicate changes in perceptual processing during visual-spatial priming. Journal of Experimental Psychology, 17(4), 1057-1074. Martin, M. (1979). Local and global processing: The role of sparsity. Memory and Cognition, 7(6), 476-484. Moreaud, O. (2003). Balint Syndrome. Archives of Neurology, 60, 1329-1331. Navon, D. (1977). Forest before trees: The precedence of global features in visual perception. Cognitive Psychology, 9, 353-383. Neisser, U. (1967). Cognitive psychology. New York: Appleton-Century-Crofts.  49  Rafal, R. (2003). Balint's syndrome: A disorder of visual cognition. In M. D'Esposito (Ed.), Neurological foundations of cognitive neuroscience (pp. 27-40). Cambridge, MA: MIT Press. Rizzo, M., & Vecera, S. (2002). Psychoanatomical substrates of Balint's syndrome. Journal of Neurology, Neurosurgery and Psychiatry, 72, 162-178. Senkowski, D., & Herrmann, C. S. (2002). Effects of task difficulty on evoked gamma activity and ERPs in a visual discrimination task. Clinical Neurophysiology, 113, 1742-1753. Shalev, L., Mevorach, C., & Humphreys, G. (2007). Local capture in Balint's syndrome: Effects of grouping and item familiarity. Cognitive Neuropsychology, 24(1), 115-127. Sternberg, S. (1969). The discovery of processing stages: Extensions of Donders' method. In W. Koster (Ed.), Attention and Performance II (pp. 276-315). Amsterdam: North Holland. Treisman, A., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97-136. Verleger, R., Jaskowski, P., & Wascher, E. (2005). Evidence for an integrative role of P3b in linking reaction to perception. Journal of Psychophysiology, 19(3), 165-181. Vogel, E. K., & Luck, S. J. (2000). The visual N1 component as an index of a discrimination process. Psychophysiology, 37, 190-203. Volberg, G., & Hubner, R. (2004). On the role of response conflicts and stimulus position for hemispheric differences in global/local processing: An ERP study. Neuropsychologia, 42, 1805-1813.  50  Chapter 31 Global perception in simultanagnosia is not as simple as a game of connect-the-dots  1  A version of this chapter has been published. Dalrymple, KA, Bischof, WF, Cameron, D, Barton, JJS, & Kingstone A. (2009). Global perception in simultanagnosia is not as simple as a game of connect-the-dots. Vision Research, 49, 1901-1908.  51  Simultanagnosia is a rare neuropsychological disorder that reflects in part a restriction of visuospatial attention. Classically a patient with simultanagnosia is said to be aware of only a single object at any one time (Moreaud, 2003; Rafal, 2003; Rizzo & Vecera, 2002). It typically occurs with bilateral lesions of the parieto-occipital junction, and is often a component of Bálint syndrome (Rizzo, 1993). Some studies have shown that simultanagnosia can be associated with an abnormal direction of attention towards smaller, local, elements of a scene at the expense of larger global elements, a phenomenon called ‘local capture’ (Dalrymple, Kingstone, & Barton, 2007; Huberle & Karnath, 2006; Karnath, Ferber, Rorden, & Driver, 2000). This is reflected in the narrative of simultanagnosic patients describing visual scenes, which suggests a piecemeal approach and a failure to integrate the individual elements into a coherent whole (Dalrymple, et al., 2007; Duncan, et al., 2003; Humphreys & Price, 1994; Rafal, 2003). Local capture has been investigated experimentally with hierarchical stimuli (Dalrymple, et al., 2007; Karnath, et al., 2000; Rafal, 1997), such as large ‘global’ letters made up of several repetitions of smaller ‘local’ letters (Navon, 1977). Simultanagnosic patients identify local letters well, but are poor and inconsistent at identifying global letters, sometimes naming them successfully on one trial, only to fail on the next (Dalrymple, et al., 2007; Huberle & Karnath, 2006; Karnath, et al., 2000). By monitoring the eye movements of simultanagnosics while they identified the local and global levels of hierarchical stimuli, Clavagnier and colleagues (2006) concluded that inconsistent global level identification in simultanagnosia is not due to an inability to disengage from the local elements of the stimulus, because the patients fixated multiple areas of the stimuli, rather than staying fixed on a single local letter. Patients made significantly more eye movements than control subjects and almost appear to trace the contour of the global letter with their eyes. If their inability to process the global level of these stimuli cannot be accounted for by an inability to disengage from local elements, what is an alternative explanation for their difficulties with this task? 52  Previously we tested a simultanagnosic patient (SL) while she identified the local and global levels of hierarchical stimuli (Dalrymple, et al., 2007). SL’s unsuccessful attempts at identifying the global level of hierarchical letters were often characterized by a close approximation of the global letter shape (e.g. reporting P when the global letter was a B). This suggests that SL’s success or failure may depend on specific exploration of critical parts of a stimulus that distinguish the true letter from the mistakenly reported one. While healthy subjects can see all elements of the hierarchical stimulus at once, restricted visual attention may limit simultanagnosic patients to seeing only portions of the stimulus in the vicinity of their current fixation. Accordingly, their perception at a global level would require assembly and integration of the local elements processed in sequential fixations (i.e. “connecting-the-dots”). In support of this hypothetical assembly of global identities from local elements, simultanagnosic patients are better at naming the global form of hierarchical letters that are small and contain densely packed local elements (Dalrymple, et al., 2007; Huberle & Karnath, 2006), conditions that place more neighbouring local elements within a spatially constricted attentional window. Furthermore, patients are better at identifying the global letter when using a finger to passively trace the global shape (Karnath, et al., 2000), also suggesting the implementation of a ‘connect-the-dots’ strategy. We monitored the eye movements of a simultanagnosic patient (SL) while she performed local and global identification tasks in separate trials to determine whether successful global letter identification requires thorough scanning and assembly of local elements (i.e. “connecting the dots”). If this hypothesis is correct, SL’s incorrect global responses will be close shape approximations to the actual identity of the global letter. Furthermore, this hypothesis predicts that SL’s unsuccessful global trials will be characterized by a failure to scan the critical region of the stimulus letter that distinguishes it from other possible letter identities.  53  Method Participants Patient SL: Case report Patient SL is a 49 year-old right-handed woman, with 12 years of education. She had idiopathic cerebral vasculitis resulting in bilateral parietal and lateral occipital infarcts (Figure 3.1). She had been treated with cyclophosphamide and prednisone for her vasculitis, but had completed these 4 months prior to her testing. At the time of testing she was on carbamazepine for a single seizure suffered several months prior. She presented with left hemi-neglect, as assessed with the Sunnybrook Neglect Assessment Battery (Leibovitch, et al., 1998), left inferior quadrantanopia, and Bálint syndrome, with ocular motor apraxia, optic ataxia, and simultanagnosia, though her acuity was 20/25 in both eyes. Her optic ataxia was evident in that she often mis-reached for objects, and failed to orient her grasp correctly to the axes of objects such as pencils. Her simultanagnosia was evidenced through tests with four complex displays of visual scenes. For example, she could report elements of the Boston Cookie Theft picture (Goodglass & Kaplan, 1983), but was unable to make sense of the whole scene. She initially reported seeing only “a boy’s face… eyes,” without reporting the mother on the right side of the display or the second child in the scene, nor did she describe the action in the scene. Neuropsychological evaluation showed normal attention, language, and verbal memory functions. Her reading was in the borderline impaired range and she tended to guess words based on the first or last letters. She was successful at recognizing simple line drawings of objects and could correctly identify colours and simple shapes.  54  a.  b.  Figure 3.1. Axial FLAIR sequences of MRI scans of a) patient SL, and b) patient ES. White areas indicate hyperintense signal in damaged brain regions. R = Right, L = Left side of the brain. At the time of testing, several months after onset, SL no longer showed left hemi-neglect or quadrantanopia and had no defects in saccadic targeting and generation, as was confirmed by her rapid and accurate saccades during the calibration of the eye monitor. However she still showed optic ataxia when using the left hand to point to targets. This was a specific sensorimotor transformation for the contralateral hand, and therefore not a due to a general difficulty with perceptual localization (which would affect both hands). Patient SL has been discussed in previous reports (i.e. Dalrymple, et al., 2007; Malcolm & Barton, 2007) Control subjects Healthy control participants (N = 8; 5 male) were volunteers from the community, who ranged in age from 40-57 years (mean = 51 years). All participants reported normal or corrected-to-normal vision and gave informed consent prior to participation in the experiments, which were performed in accordance with the ethical guidelines of UBC.  55  Brain damaged control Patient ES: Case report As an additional control, we tested another patient, ES, who had also suffered bilateral posterior occipitoparietal damage, but never had simultanagnosia. ES matched SL well in age, gender, the chronic phase at testing, and probable pathology, since she also has an underlying condition that is associated with vasculitis. She is a 47 year-old woman with systemic lupus erythematosis. She was tested several months after presenting with flashing lights and transient visual loss for 30 minutes, followed by headache. Her visual examination was normal, but MR imaging revealed bilateral occipital and parietal lesions consistent with either vasculitis or posterior leucoencephalopathy. Subsequently she had a seizure, and was treated with phenytoin for 9 months. At her most recent visit she was taking prednisone, chloroquine, and mycophenolate mofetil. Her visual acuity without correction at far was 20/20 in both eyes. Confrontation showed full visual fields. Fixation, pursuit and saccades were normal. There was no nystagmus, oculomotor apraxia or optic ataxia. There was no simultanagnosia as demonstrated by normal report on the Boston Cookie Theft picture. Experiment Stimuli and apparatus Hierarchical letters (global upper-case letters made up of several repetitions of smaller, local upper-case letters) were presented on a 33x24.5 cm monitor corresponding to 36.5°x27.5° at the viewing distance of 50 cm. All letters were black uppercase and on a white background. All letters of the alphabet were eligible for use except local letters M, O, W, because their adjacent elements overlapped in dense global displays. Global and local letters were never the same. Stimuli were sampled with replacement (letters could recur within the same block) preventing subjects from deducing the identity of the letters based on which letters had been displayed. Letters were created in three different sizes and densities, for a total of 9 different Size x Density combinations (Figure 3.2; see also Dalrymple, et al., 2007). Global letters averaged, 17.4°x 15.3° for Large stimuli, 8.9° x 7.0° for Medium, and 56  5.9° x 4.7° for Small. Inter-element spacing ranged from 3.3° for Large/Sparse stimuli to 0.06° for Small/Dense stimuli.  Figure 3.2. Examples of the Navon hierarchical letters of each size and density used. Size refers to the dimensions of the global stimulus. Density refers to the degree to which the global letter is packed with local elements (more dense = more local elements) EyeLink II and 1000 systems detected saccades with an amplitude of at least 0.5° using an acceleration threshold of 9500°/s2 and a velocity threshold of 30°/s. Fixations were defined as the epochs between successive saccades. Procedure For SL each block started with the experimenter stating that the target letter was Global or Local. On each trial the task was to fixate a central circle, which then disappeared and after 500ms it was replaced by a target letter. SL named the target as quickly as possible. The experimenter keyed in the response (to avoid inaccurate reaching by SL) and this terminated the trial and triggered the next trial's fixation circle. The procedure was identical for Control subjects, except that they pressed a spacebar when the target was identified and entered their response on the keyboard. ES verbally reported her response, at which point the experimenter  57  pressed the spacebar to terminate the trial and subsequently entered the response on the keyboard. Trials were blocked by Size-Density configuration, and by task (Global or Local). Thus subjects performed 9 blocks of Global target letters and 9 blocks of Local target letters. Each block consisted of 11 trials. Patients SL and ES performed the Global before Local blocks. For Controls, the order was counterbalanced across participants. For all participants the trials within each block, and the blocks within a target level, were randomized. Analysis The data from one control was excluded from analysis because she did not complete all conditions. For all ANOVAs all of SL and ES’s trials for each block were used; the average measures per block were used for each healthy control participant. All alpha levels were set to p < 0.05. We analyzed the data set at three different levels: A. Accuracy and basic eye movement measures Basic performance on the Local and Global letter identification tasks was assessed, in terms of accuracy and the key scanning measures of number of fixations, fixation duration, and saccade amplitude. B. Critical difference analysis This analysis was designed to determine whether SL’s incorrect Global reports were due to failures to scan the segments of the actual letters that made them distinct from her reported letter. This analysis determined whether, during unsuccessful trials, SL failed to fixate the part of a Global letter that made it distinct from other letters of the alphabet. For all Global trials that the patient identified unsuccessfully, we compared the shape of the actual Global letter to the shape of the letter that the patient reported. We then determined what part of the actual letter was ‘Critically Different’ from the reported letter. For example, if the actual letter was ‘R’, but the patient reported ‘P’, the area that is critically different between the two  58  letters is the diagonal line of the ‘R’. The Critical Difference between two letters can be the presence of a letter segment, such as the presence of the diagonal line on the letter ‘R’, the absence of a segment, such as the absence of a diagonal line on the letter ‘P’, or a combination of the presence and absence of various segments (i.e. Letters ‘M’ and ‘N’: ‘M’ can be distinguished from ‘N’ by the presence of a segment in the top right side of the ‘M’, and the absence of a segment on the bottom right of the ‘M’) (Figure 3.3).  a.  b.  c.  d.  Figure 3.3. Example of steps involved in determining fixation overlap with critical area: a) identifying the actual letter viewed and tracing an area of 1° around each local element; b) identifying the reported letter and tracing an area of 1° around each element; c) identifying the critical area: the area of the actual letter and the reported letter that does not overlap; d) determining the area of overlap between SL’s fixations (defined by 1° around each fixation) and the critical area.  The shape and area of Global letters were defined by drawing a circle with a radius of 1 degree around each local letter. The patient’s erroneous response letter on a given unsuccessful trial was overlaid on the actual letter that was presented on  59  that trial. This determined the ‘Critical Difference Area’ between the two letters. Next, the patient’s fixations on that trial were overlaid on the Critical Difference Area to determine what proportion of the fixations for that trial landed within this area.  C. SL Global Task - Successful vs Unsuccessful trials We compared SL’s eye movements on unsuccessful global trials to her eye movements during successful global trials to determine whether any other differences exist in her scanning technique that could account for her global processing deficit. While such comparisons have been made before in two patients with Balint syndrome (Clavagnier, et al., 2006), Local and Global levels were identified simultaneously in the same trial in that prior report, and thus it was not possible to determine whether eye movements were being driven by processes involved in Local or in Global letter identification. In contrast, our patient performed Local and Global identification on separate trials. We also included new measures of scanning behaviour aimed at assessing where the patient’s eye movements were distributed. This included the following: (i) The amount of letter area2 fixated, expressed as a proportion of the total letter area (the total proportion of the letter stimuli covered by each fixation, summed over all fixations and divided by the total amount of letter area, in pixels); (ii) the average letter area covered per fixation; (iii) the total proportion of the background covered by each fixation, summed over all fixations and divided by the total amount of background area, in pixels.  2  All area measures were calculated by drawing a circle with radius of 1 degree of visual angle around each local letter of the stimulus and around each fixation.  60  Results A. Accuracy and basic eye movement measures SL vs Controls Accuracy We compared SL’s Accuracy to that of Controls using a 4-way ANOVA with factors of Subject (Patient vs Controls), Level (Global vs Local), Density (Sparse, Medium Density, Dense), and Size (Small, Medium, Large). There was a significant effect of Subject, F(1,4) = 11.74, p < 0.001; Level, F(1,4) = 6.52, p = 0.011; and Density, F(2,4) = 7.64, p < 0.001. There was also a significant Subject by Level interaction, F(1,4) = 4.86, p = 0.028; Subject by Density interaction, F(2,4) = 8.18, p < 0.001; and Level by Density interaction, F(2,4) = 4.43, p = 0.013. No other results were significant. To understand these interactions, Local and Global performance was analyzed with separate 3-way ANOVAs (see Figure 3.4).  a.  Figure 3.4. a) SL’s accuracy (%) for naming global and local letters.  61  b.  Figure 3.4. b) Overall accuracy for SL, ES, and healthy Controls, for identifying letters at the global and local levels. Accuracy here is collapsed over stimulus size and density because the performance of ES and healthy controls was at ceiling for all conditions.  Local task. A 3-way ANOVA with factors of Subject, Density and Size revealed no significant main effects or interactions, indicating that SL did not differ from Controls in terms of Accuracy for naming the Local level of letters (Patient = 96% vs. Controls = 99%, F(1,4) = 1.44, p = 0.232). Global task. Controls were near-perfect at naming the Global letters, and significantly better than SL overall (Patient = 84% vs Controls = 98%; F(1,4) = 10.57, p = 0.001). There was a significant main effect of Density, F(2,4) = 7.70, p < 0.001. Bonferroni Multiple Comparisons revealed that subjects were significantly worse at the Sparse letters (mean = 76%), compared to the Medium Density (mean = 94%) or Dense letters (mean = 99%). However, a significant Subject by Density interaction, F(2,4) = 6.89, p = 0.001, indicates that this effect was driven by SL's accuracy pattern: while her accuracy varied with stimulus Density, F(2,4) = 11.50, p < 0.001, Sparse = 61%, Medium Density = 91%, Dense = 100%, the accuracy of Control  62  subjects did not, F(2,4) = 0.34, p = 0.710. Eye movements Number of fixations, mean fixation duration, and mean saccade amplitude were analyzed with Subject, Level, Density and Size as factors. Number of fixations There were main effects of Subject, F(1,4) = 27.82, p < 0.001, indicating that SL made significantly more fixations than Control subjects (13.63 vs 3.80); Level, F(1,4) = 4.06, p = 0.045, indicating that subjects made more fixations for Global than Local letters (11.99 vs 7.58); and Density, F(2,4) = 3.21, p = 0.042 with Bonferroni tests revealing that subjects made more fixations when identifying Sparse than Dense letters (13.63 vs. 7.23) with Medium Density (8.49) not differing from the two extremes. No other effects were significant. Fixation duration There were main effects of Subject, F(1,4) = 225.11, p < 0.001, indicating that SL's durations were shorter than the fixations of Controls (457.00 vs. 264.60 ms); and Letter Size, F(2,4) = 4.25, p = 0.015. Bonferroni tests revealed that durations were longer when identifying Small vs. Large letters (365.28 vs. 319.21 ms), with Medium letters (336.03 ms) not different from either Small or Large letters. No other effects were significant. Saccade amplitude There were main effects of Subject, F(1,4) = 262.30, p < 0.001, indicating that SL's saccades were shorter than Controls (5.71° vs. 24.11°); and Level, F(1,4) = 4.64, p = 0.032, indicating that saccades were larger overall for Global than Local letters (14.20° vs. 11.68°). No other effects were significant. In summary, for both Local and Global targets, SL makes more fixations, of smaller duration and amplitude than Controls.  63  Brain damage control In order to help disambiguate specific profiles of simultanagnosia from general effects of brain damage and related health problems, we tested one brain damaged control subject, ES. ES was chosen because she matched SL in terms of age, gender, and approximate lesion location, but ES was never simultanagnosic. This allows us to determine whether simultanagnosia itself is crucial to the behavioural effects observed in SL. We compared ES to our healthy control group and to SL on measures of accuracy, number of fixations, fixation durations, and saccadic amplitude. We restricted our analyses to look only at effects of Subject for each of the Global and Local tasks, respectively. Any main effects of Subject were followed up with a Bonferroni multiple comparisons test. Accuracy is presented in Figure 3.4, while eye movement profiles are presented in Figure 3.5. a.  b.  c.  Figure 3.5. Basic eye movement results for patients SL, ES, and healthy Control subjects for each of the global and local letter identification tasks: a) mean number of fixations; b) mean fixation duration; c) mean saccade amplitude. Error bars represent standard error from the mean.  64  ES vs Controls vs SL: Global Accuracy We compared ES’s accuracy to that of Controls and SL using a 1-way ANOVA with factors of Subject (ES vs Controls vs SL). There was a significant effect of Subject, F(2,255) = 9.70, p < 0.001, indicating that SL’s accuracy at the Global task was significantly worse than ES and Controls, who did not differ from each other (SL = 84%, ES = 98%, Controls = 98%). Eye movements There was a main effect of Subject for all eye movement measures, number of fixations F(2,255) = 12.75, p < 0.001; fixation duration, F(2,255) = 93.08, p < 0.001; saccade amplitude, F(2,255) = 107.89, p < 0.001. SL made significantly more fixations than ES and Controls, who did not differ from each other (SL = 17.31, ES = 6.47, Controls = 3.88). In duration, Controls made significantly longer fixations than ES and SL, who did not differ from each other (SL = 265.09, ES = 284.04, Controls = 464.30ms). SL had significantly shorter saccades than ES, who in turn had significantly shorter saccades than Controls (SL = 6.80°, ES = 11.58°, Controls = 25.51°). ES vs Controls vs SL: Local Accuracy There was no significant effect of Subject, F(2,260) = 1.48, p = 0.229, indicating that SL’s accuracy at the Local task equivalent to that of ES and of Controls, who also did not differ from each other (SL = 96%, ES = 99%, Controls = 99%). Eye movements There was a main effect of Subject for all eye movement measures: number of fixations F(2,260) = 97.09, p < 0.001; fixation duration, F(2,260) = 59.16, p < 0.001; saccade amplitude, F(2,260) = 115.26, p < 0.001. SL made significantly more fixations than ES, who made more fixations than Controls (SL = 10.05, ES = 5.65, Controls = 3.71). Controls made significantly longer fixations than ES and SL, who 65  did not differ from each other (SL = 264.12, ES = 299.95, Controls = 449.70ms). SL had significantly shorter saccades than ES, who had significantly shorter saccades than Controls (SL = 4.66°, ES = 12.06°, Controls = 22.74°). In summary, ES had normal accuracy for both Global and Local letter identification. For eye movements on both tasks, ES made either similar or only slightly greater numbers of fixations than Controls, which were far fewer than those made by SL. However, the brevity of ES’s fixations was more similar to SL than to Controls, and her saccades were smaller than those of Controls, though not as small as those of SL. B. Critical difference analysis SL’s incorrect global responses are reported in Table 3.1. As predicted, SL’s reports are close approximations of the actual global letter shape (e.g. reporting ‘C’ for the letter ‘O’). To analyze whether her incorrect responses were due to a failure to scan the segments of the actual letters that made them distinct we designed and implemented our “Critical Difference” analysis. SL made errors on a total of 15 out of 96 Global trials. The overlap of the patient’s fixations and the Critical Difference Area varied between 0% and 88%. Of note, she did fixate at least a portion of the Critical Difference Area on 87% of unsuccessful global trials.  66  Stimulus Size Medium Large Large Large Large Large Small Small Small Large Large Medium Medium Medium Medium  Stimulus Density Medium Sparse Sparse Sparse Sparse Sparse Sparse Sparse Sparse Medium Medium Sparse Sparse Sparse Sparse  Local letter L P R Q Q H V F T F F X D I H  Global Letter M A F A W P C V W Q C C S R N  Patient Proportion of critical Response area fixated N 0.511 H 0.439 P 0.124 H 0.159 H 0.195 F 0.029 O 0.000 U 0.178 H 0.550 O 0.059 O 0.000 O 0.875 B 0.109 F 0.861 H 0.239 Mean 0.289  Table 3.1. Summary of patient’s unsuccessful global trials and proportion of critical area fixated for each unsuccessful global trial (in order of viewing). Local letter identity, global letter identity, and patient response are indicated as well as proportion of critical area scanned. Mean, high and low proportions are in bold. C. SL Global task - Successful vs. Unsuccessful trials We performed 1-way ANOVAs with factor of Success (Successful vs Unsuccessful) for each eye movement measure. SL made more fixations during Unsuccessful compared to Successful global trials (37.4 vs.13.59, F(1,95) = 8.52, p = 0.004). However, SL’s fixation durations and amplitudes did not vary with performance success. SL made more fixations during Unsuccessful trials; where was she distributing these fixations? Analysis revealed that SL proportioned more of her fixations to the background during Unsuccessful than Successful global trials (0.063 vs. 0.030, F(1, 95) = 17.99, p < 0.001). There was no effect of letter area fixated and letter area encompassed per fixation. Thus, SL makes more fixations and scans more of the background during Unsuccessful compared to Successful Global trials.  67  Discussion We tested the hypothesis that impaired perception of global forms in simultanagnosia results from a failure to adequately scan and then assemble local elements into a global whole. Our results confirm that SL has preserved perception of local elements and impaired global perception. In support of our hypothesis, SL’s incorrect responses were often close letter-shape approximations of the correct letter, suggesting that part of her global-level problems may be the result of inadequate collection of local element data to differentiate similar letter forms. Our “Critical Difference” analysis, however, did not support this hypothesis. This analysis examined whether SL was fixating the segments that distinguish the correct letter from her incorrect answer on unsuccessful global trials. SL scanned the critical difference area on all but two of her unsuccessful trials: for example, on one trial she reported that the letter ‘F’ was a letter ‘P’, despite scanning the open portion of the ‘F’ that would rule out the possibility that it was a ‘P’. Conversely, a visual inspection of SL’s trials indicated that there were no instances where she scanned the entire global form in order to correctly identify the global letter. Together, these findings indicate that fixating, or “connecting the dots” of the distinctive parts of the global shape, is neither necessary, nor sufficient, for SL to correctly discern their identity. These results have several implications. First, SL’s errors in identifying global letters despite fixating the portion of the letter that would invalidate the error could be due to a failure to process this information to a level of conscious awareness. Other research has shown that simultanagnosics can “look, but not see”: stimuli disappear suddenly from their awareness despite steady fixation (Rizzo & Hurtig, 1987). Rizzo and Hurtig suggest that this abnormal disruption of visual awareness allows stimuli to be processed to a point where they can influence ocular motor movements, while not reaching a level of conscious processing. Thus, SL may be fixating the letter stimuli, but the information gathered during those fixations is not being processed to a level sufficient for identification of the global form.  68  Alternatively SL may acquire information where she is fixating, but may not correctly integrate it spatially with the information about other local elements. Rizzo and Robin (1990) have suggested that simultanagnosics may suffer from an inability to maintain continuous visuospatial attention across an array and that this could prevent multiple elements from being integrated so that spatial information is registered relative to each other. Similarly, Cooper and Humphreys (2000) have suggested that simultanagnosics have poor representations of the spatial relationship between elements such that, with no relative spatial information, they may even re-select the same object multiple times. This is consistent with SL’s scanning patterns that show that she frequently revisits some elements, while failing to scan others (e.g. Figure 3.6). Indeed, SL’s increased fixation number may reflect persistent searching when a definitive (and correct) decision has not yet emerged, due to uncertainty on incorrect trials. The increased scanning of the empty background is of interest, and one interpretation of this result is that SL’s failures are related to this excessive background scanning, perhaps due to getting ‘lost’ in empty space. However, SL fixated as much of the letter stimuli on unsuccessful as on successful global trials, so in principle she had ample opportunities to acquire local elements for global structure in both situations.  69  Figure 3.6. Example of patient scan path for global trial. On this trial the patient incorrectly reported the global letter as a “B”. Circles represent fixations. The size of circle represents the duration of the fixation, with larger circles representing longer fixations. Lines represent eye movements.  SL’s successful performance on global trials is also informative. There are no instances where she scans all local elements. While it is possible that her occasional successes in global identification are due to successful assembly of the global shape from the local elements that she scans serially, her successful global letter identification could also be evidence of residual true global processing, rather than a feature-by-feature strategy. Other simultanagnosic patients have, like SL, been reported to struggle to name the global letters on some trials while successfully naming them on other trials (Clavagnier, et al., 2006; Himmelbach, Erb, Klockgether, Moskau, & Karnath, 2008; Karnath, et al., 2000; Shalev, Humphreys, & Mevorach, 2004; Shalev, Mevorach, & Humphreys, 2007). This behaviour has been correlated with fluctuations of brain activation in the posterior parietal cortex, the area affected  70  in simultanagnosia (Himmelbach, et al., 2008), suggesting a mechanism for preserved but unreliable global level perception in these patients. In order to help disambiguate specific profiles of simultanagnosia from general effects of cerebral lesions and related health problems, we tested a second brain-damaged patient, ES. ES was age-matched, had bilateral occipitoparietal lesions, but was never simultanagnosic. Like SL, ES showed abnormal eye movements on our basic measures, making, in general, more fixations than healthy controls, fixations of shorter duration, and saccades of shorter amplitude, yet unlike SL, ES’s performance was not impaired at either the global or local letter identification task. Therefore SL’s impaired ability to report the global level of our hierarchical stimuli is likely a specific manifestation of her simultanagnosia, rather than a general result from bilateral posterior brain damage, or to her abnormal eye movements. Indeed our results ultimately show that aside from the amount of background area scanned during global trials, SL’s eye movements do not seem to predict her performance on this task. We and others have shown that simultanagnosics are better at identifying global forms that are small and made up of densely packed local elements, yet are poor at identifying global forms that are large and made up of widely spaced local elements (Dalrymple, et al., 2007; Huberle & Karnath, 2006). We replicated this finding, which suggests that forms that are easily grouped are available to the patient for conscious report. Other Gestalt rules, such as collinearity and closure, also predict patient performance (Cooper & Humphreys, 2000). The manipulations that promote global level perception in these patients are those that allow preattentive processing of the global shape (Cooper & Humphreys, 2000; Enns & Kingstone, 1995). Thus, it is possible that if a stimulus is pre-attentively packaged as a whole (e.g. small/dense letters), it will be explicitly available to the individual. The large/sparse letters, on the other hand, require attention to be grouped. Considering SL’s limited attentional resources, this could make explicit report of the global form difficult.  71  With access to only a small portion of the scene at one time, increased interelement spacing may also increase the demands on visual short term memory (VSTM), suggesting another mechanism for SL’s poor performance in these conditions. The capacity of visual short-term memory has been estimated to be limited to approximately 4 items (Luck & Vogel, 1997). Huberle and Karnath (2006) argue that if simultanagnosia is linked to a limitation of VSTM, the addition of more elements (and therefore VSTM load) would lead to impaired performance, the reverse of the improvement that is seen when patients view hierarchical letters made up of several, densely packed elements compared to a few, sparse elements. Using Bundesen’s Theory of Visual Attention (TVA) Duncan et al (2003) also suggested that VSTM capacity was not the primary deficit in simultanagnosia, instead suggesting a limitation of processing capacity. This is consistent with suggestions from others, who argue that simultanagnosia reflects deficits in sustained visual attention (Luria, 1959; Rizzo & Robin, 1990). Further research is necessary to clarify the role of VSTM in simultanagnosia. Conclusions Our results show that our simultanagnosic patient’s failures of global level perception did not result from a failure to scan the parts of those letters that make them distinct from other, similar letters. Furthermore, scanning each individual element of the hierarchical letters was not necessary for successful global level perception. It is therefore unlikely that impaired global level perception in simultanagnosia is related to a failure of a strategic connect-the-dots pattern aimed at piecing together the identity of global letters. Rather, we found that unsuccessful global trials are characterized by excessive searching behaviour, reflected by increased eye fixations and greater coverage of background area. Beyond this, our patient’s eye movements did not seem to predict her performance, suggesting that disrupted eye movements may not be the cause of her difficulties with the global letters, but rather the consequence of a breakdown in information processing.  72  References Clavagnier, S., Fruhmann Berger, M., Klockgether, T., Moskau, S., & Karnath, H.-O. (2006). Restricted ocular exploration does not seem to explain simultanagnosia. Neuropsychologia, 44, 2330-2336. Cooper, A., & Humphreys, G. (2000). Coding space within but not between objects: Evidence from Balint's syndrome. Neuropsychologia, 38, 723-733. Dalrymple, K., Kingstone, A., & Barton, J. (2007). Seeing trees OR seeing forests in simultanagnosia: Attentional capture can be local or global Neuropsychologia, 45, 871-875. Duncan, J., Bundesen, C., Olson, A., Humphreys, G., Ward, R., Kyllingsbaek, S., et al. (2003). Attentional functions in dorsal and ventral simultanagnosia. Cognitive Neuropsychology, 20(8), 675-701. Enns, J., & Kingstone, A. (1995). Access to global and local properties in visual search for compound stimuli. Psychological Science, 6, 283-291. Goodglass, H., & Kaplan, E. (1983). The assessment of aphasia and related disorders (2nd ed.). Philadephia: Lea and Febiger. Himmelbach, M., Erb, M., Klockgether, T., Moskau, S., & Karnath, H.-O. (2009). fMRI of global visual perception in simultanagnosia. Neuropsychologia, 47, 1173-1177. Huberle, E., & Karnath, H.-O. (2006). Global shape recognition is modulated by the spatial distance of local elements - Evidence from simultanagnosia. Neuropsychologia, 44(6), 905-911. Humphreys, G., & Price, C. (1994). Visual feature discrimination in simultanagnosia: A study of two cases. Cognitive Neuropsychology, 11(4), 393-434. Karnath, H.-O., Ferber, S., Rorden, C., & Driver, J. (2000). The fate of global information in dorsal simultanagnosia. Neurocase, 6, 295-306. Leibovitch, F., Black, S., Caldwell, C., Ebert, P., Ehrlich, L., & Szalai, J. (1998). Brain-behaviour correlations in hemispatial neglect using CT and SPECT: The Sunnybrook Strong Study. Neurology, 50, 901-908. Luck, S., & Vogel, E. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390, 279-281. Luria, A. (1959). Disorders of "simultaneous perception" in a case of bilateral occipito-parietal brain injury. Brain, 82, 437-449. Malcolm, G., & Barton, J. J. S. (2007). "Sequence Agnosia" in Balint's Syndrome: Defects in Visuotemporal processing after bilateral parietal damage. Journal of Cognitive Neuroscience, 19(1), 102-108. 73  Moreaud, O. (2003). Balint Syndrome. Archives of Neurology, 60, 1329-1331. Navon, D. (1977). Forest before trees: The precedence of global features in visual perception. Cognitive Psychology, 9, 353-383. Rafal, R. (1997). Balint Syndrome. In T. Feinberg & M. Farah (Eds.), Behavioural Neurology and Neuropsychology (pp. 337-356). USA: McGraw-Hill. Rafal, R. (2003). Balint's syndrome: A disorder of visual cognition. In M. D'Esposito (Ed.), Neurological foundations of cognitive neuroscience (pp. 27-40). Cambridge, MA: MIT Press. Rizzo, M. (1993). Balint's syndrome and associated visuospatial disorders. Philadelphia: WB Saunders. Rizzo, M., & Hurtig, R. (1987). Looking but not seeing: Attention, perception, and eye movements in simultanagnosia. Neurology, 37(10), 1642-1648. Rizzo, M., & Robin, D. (1990). Simultanagnosia: A defect of sustained attention yields insights on visual information processing. Neurology, 40, 447-455. Rizzo, M., & Vecera, S. (2002). Psychoanatomical substrates of Balint's syndrome. Journal of Neurology, Neurosurgery and Psychiatry, 72, 162-178. Shalev, L., Humphreys, G., & Mevorach, C. (2004). Global processing of compound letters in a patient with Balint's syndrome. Cognitive Neuropsychology, 22(6), 737-751. Shalev, L., Mevorach, C., & Humphreys, G. (2007). Local capture in Balint's syndrome: Effects of grouping and item familiarity. Cognitive Neuropsychology, 24(1), 115-127.  74  Chapter 41 Simulating simultanagnosia: Spatially constricted vision mimics local capture and the global processing deficit  1  A version of this chapter has been accepted for publication. Dalrymple, KA, Bischof, WF, Cameron, D, Barton, JJS, & Kingstone A. (2010). Simulating simultanagnosia: spatially constricted vision mimics local capture and the global processing deficit. Experimental Brain Research, 202, 445-455.  75  Bálint syndrome is a neurological disorder that typically results from bilateral lesions to the parieto-occipital junction (Balint, 1909). It is characterized by four primary symptoms: (1) dorsal simultanagnosia2: a restricted window of visual attention resulting in an inability to see more than a small perceptual area at one time; (2) spatial disorientation: an inability to locate objects in space; (3) optic ataxia: an inability to use visual information to guide accurate reaching towards objects; and (4) ocular apraxia: an inability to voluntarily execute accurate eye movements (Moreaud, 2003; Rafal, 2003; Rizzo & Vecera, 2002). In everyday life, patients with Bálint syndrome routinely describe scenes in a piecemeal fashion. This can be demonstrated in the laboratory by presenting patients with hierarchical stimuli, which are global shapes (such as letters) made up of several local elements (e.g. other letters, see Figure 4.1). When faced with such stimuli, patients with Bálint syndrome are remarkably poor at identifying the global form despite normal accuracy for reporting the identity of the local elements (Karnath, Ferber, Rorden, & Driver, 2000).  2  Patients with Bálint syndrome present with dorsal (as opposed to ventral) simultanagnosia (Farah, 1990) which is an attentional limitation that prevents patients from seeing more than one object at a time. Thus, we define simultanagnosia in this context strictly as a reduction in attentional processing capacity that is in part reflected by a failure to process or maintain attention to a larger region of the visual field outside of the current focus of attention, as opposed to the more general definition offered by Wolpert (1924), which also includes difficulty with the interpretation of the global concept of a scene or a figure.  76  Figure 4.1 Examples of the Navon hierarchical letters of each size and density used. Size refers to the dimensions of the global stimulus. Density refers to the degree to which the global letter is packed with local elements (more dense = more local elements and less inter-item space)  Simultanagnosia is understood to play a key role in this global processing deficit, but whether it can account for the global processing deficit entirely remains unclear. An alternate view posits that the restricted attentional window of simultanagnosia, is insufficient for explaining the global processing deficit. While it is generally agreed that a restricted spatial area, or “window”, of visual attention in simultanagnosia can preclude “normal” global processing, according to this position it should in theory be possible for an individual whose only limitation was a restricted window of visual attentional processing to reconstruct a global picture from the serial perception of local elements. That is, with a restricted window of attention, it should still be possible to 1) locate local elements relative to each other, 2) remember their locations, and 3) integrate those elements into the global picture. If these processes are intact, patients with simultanagnosia should show longer reaction times, but good accuracy for global report, making simultanagnosia alone insufficient for explaining the inability to derive global shape. However, simultanagnosic patients are unable to deduce the identity of global forms, even with unlimited viewing time. This has suggested to others that there may be an additional impairment underlying 77  this inability to derive global shape, beyond a simple reduction of visual area (e.g. Farah, 1990; Tyler, 1968). One candidate for an additional deficit is the inability to commit visual attention to each element in order to mark the local elements relative to each other. Tyler (1968) proposed precisely such a mechanism. He wrote that the “presence of small ‘effective’ fields combined with bilateral parietal ‘attention’ defects would seem to be the ideal substrate” (p.168). Echoing this position, Farah (1990) conducted a thought experiment and proposed that if a healthy subject were seated in a dark room and saw a sequence of flashes on a screen, s/he should be able to “keep track of” their relative locations. “However,” she argues, “this ‘keeping track of’ previous locations presumably involves allocating attention to them, something that dorsal simultanagnosics cannot do.” (p. 44). Thus, these researchers suggest that there must be an additional attentional impairment, beyond the restricted attentional window, that contributes to the inability to derive global shape that is typically associated with simultanagnosia. Despite this convincing reasoning, or perhaps because of it, no investigation to-date has tested whether this view of an additional attentional deficit beyond a restriction of the attentional window in simultanagnosia is accurate. The alternative possibility is that the restriction of the attentional window in simultanagnosia could on its own be sufficient to impair the ability to derive global shape. Clearly a restricted window of attention would impair the normal perception of global forms, but could it also impair the ability to piece together the global form from successive perception of local elements? The aim of the present study was to provide precisely this test. We tested healthy individuals with normal brains and normal visual perception on a global – local letter identification task under “simulated” conditions of simultanagnosia. This simulation was achieved by creating a gaze-contingent display to mimic a metaphorical “restricted window of attention” with a literal window of vision, as others have suggested (e.g. Bay, 1953; Thaiss & de Bleser, 1992; Tyler, 1968).  78  The visual experience was created by use of an eye monitor and a gazecontingent aperture on a computer screen (Figure 4.2). Gaze-contingent displays have been used in the past with a variety of tasks, such as reading (McConkie & Rayner, 1975), visual search (Pomplun, Reingold, & Shen, 2001), and scene exploration (Loschky, McConkie, Yang, & Miller, 2005). Critically, gaze-contingent displays only reveal a small portion of the stimulus at one time through a computercreated “window” that exposes only what the subject is looking at directly. Subjects can move the window by moving their eyes, and can explore the stimuli however they wish.  Figure 4.2 Schematic of the gaze-contingent paradigm. Dotted squares represent the gaze-contingent window, which in reality had an invisible (white on white) border. For illustrative purposes the entire stimulus is visible (light gray items), but in practice only items falling within the window at a given time were visible to participants (black items). Arrows show hypothetical path of the window. Darker elements represent more recent window locations  In the present experiment participants viewed hierarchical Navon letters (Navon, 1977), stimuli which have been used in a number of investigations of patients with simultanagnosia (e.g. Clavagnier, Fruhmann Berger, Klockgether, 79  Moskau, & Karnath, 2006; Dalrymple, Kingstone, & Barton, 2007; Karnath, et al., 2000; Shalev, Humphreys, & Mevorach, 2004). Their task was to identify the local, and, more importantly, the global letters. When simultanagnosic patients perform this hierarchical letter identification task under natural viewing conditions, they tend to do very well at identifying the local letters, yet perform poorly at identifying the global letters when those letters are large and made of widely-spaced local elements (Dalrymple, et al., 2007). Their performance improves for small global letters that are made up of densely packed local elements. According to Tyler’s (1968) reasoning, and Farah’s (1990) thought experiment, healthy participants, who do not have a deficit of visual attention, should keep track of individual local elements viewed through a narrowed window and successfully deduce the global letters. However, if the spatial constriction of visual processing associated with simultanagnosia is alone sufficient to impair global processing -- even through serial perception of individual elements -- narrowing the visual window of healthy participants should disrupt integration of local elements, yielding global-level processing deficits highly similar to those of a simultanagnosic patient.  Method Participants Gaze-Contingent group This group viewed hierarchical letters under conditions of limited visual information induced through a gaze-contingent display. Participants (n = 24, 12 male) were undergraduate students at the University of British Columbia who ranged in age from 18 to 42 years (mean = 20.7 years). All participants reported normal or corrected-to-normal vision and gave informed consent prior to participation in the experiments, which were performed in accordance with the ethical guidelines of the University of British Columbia.  80  Full-view control group This group viewed hierarchical letters under natural (unrestricted) viewing conditions. Full-View Control participants (n = 8; 1 male) were undergraduate students at the University of British Columbia who ranged in age from 17-24 years (mean = 19.4 years). All participants reported normal or corrected-to-normal vision and gave informed consent prior to participation in the experiments, which were performed in accordance with the ethical guidelines of the University of British Columbia. Patient SL Patient SL suffers from Bálint syndrome, which is characterized by ocular motor apraxia, optic ataxia, spatial disorientation, and dorsal simultanagnosia. She is a 48 year-old right-handed woman, with 12 years of education. She had idiopathic cerebral vasculitis resulting in bilateral parietal and lateral occipital infarcts (Figure 4.3). Her visual exam showed Snellen acuity of 20/25 in each eye. Her neurological exam showed left hemi-neglect, left inferior quadrantanopia, and Bálint syndrome. Her dorsal simultanagnosia was evidenced through tests with four complex displays of visual scenes. For example, she could report elements of the Boston Cookie Theft picture (Goodglass & Kaplan, 1983), but was unable to make sense of the whole scene. She initially reported seeing only “a boy’s face… eyes,” without reporting the mother on the right side of the display or the second child in the scene, nor did she describe the action in the scene. At the time of testing, several weeks after her stroke, she no longer showed left hemi-neglect or quadrantanopia, yet still showed optic ataxia when using the left hand to point to targets and she was still simultanagnosic. Patient SL and her data reported here has been discussed in a previous report (i.e. Dalrymple, et al., 2007).  81  Figure 4.3 Axial FLAIR sequences of MRI scans of patient SL within 1 month of testing. White areas indicate hypertense signal in damaged brain regions. R = Right, L = Left side of the brain  Stimuli and apparatus Hierarchical letters (global letters made up of multiple repetitions of smaller, local letters) were produced dynamically by the computer, which used a series of screen coordinates to place repetitions of a local letter into the configuration of a given global letter. The screen coordinates for each global letter were determined by the experimenter, who designed the uppercase letters on a 17 x 17 grid. Local letters were displayed in uppercase Times New Roman font and were black on a white background. All letters of the alphabet were eligible for use as global letters. Most letters of the alphabet were eligible for use as local letters, with the exception of letters M, O, and W, with which adjacent elements overlapped when stimuli were most densely packed. Note that this means that chance level accuracy for Global report is 3.8% (1/26), and 4.3% for Local report. Global and local letters were pseudo-randomly paired so that hierarchical letters were always incongruent. Global and local letters were sampled with replacement, such that letters could re-occur 82  within the same block, preventing participants from deducing the identity of the letters based on which letters had already been displayed. Letters were created in three different sizes and densities, for a total of 9 different Size x Density combinations (Figure 4.1), identical to those used in our previous experiment (i.e. Dalrymple, et al., 2007). Exact stimulus dimensions depended on which letters appeared at the Global and Local levels, as well as the Size and Density of the stimulus. Global letters were, on average, 17.4° x 15.3° for Large stimuli, 8.9° x 7.0° for Medium, and 5.9° x 4.7° for Small stimuli. Local letters were, on average, 1.0° x 0.7° for Large stimuli, 0.6° x 0.3° for Medium, and 0.5° x 0.3° for Small stimuli. Interelement spacing ranged from 3.3° for Large/Sparse stimuli to 0.06° for Small/Dense stimuli, calculated by measuring the distance between the edges of adjacent local elements. Full-View Control participants and patient SL viewed these stimuli under natural (unrestricted) viewing conditions (simply looking at letters on the screen while their eye movements were monitored). For the Gaze-Contingent group, a 2°x2° (square) gaze-driven aperture was generated by the computer, and revealed the portion of the stimulus image at the gazed-at location only. This window size was chosen because, using similar stimuli with patient SL, we estimated the size of her attentional window to be 1.25° for threshold identification of global letters, (Dalrymple, et al., 2007), similar to estimates of 2° to 4° in other patients (Tyler, 1968). The stimulus image consisted of the black letters on a white background. The non-gazed at area of the screen (area not covered by the moving aperture) was also white, matching the background colour revealed by the aperture. Thus, the hard edges of the aperture were not perceptible. For the most part this created an effect of seeing one local element at a time, though partial elements could be visible when the aperture edges overlapped with the elements, and multiple elements could be visible when elements were densely packed and fell within the window. For all participants, letters were displayed on a 33 x 24.5 cm screen corresponding to 36.5° x 27.5° at the viewing distance of 50 cm. Eye movements for the Gaze-Contingent and Full View groups, but not SL, were monitored using the  83  EyeLink II eye tracking system (SR Research Ltd., www.eyelinkinfo.com). The online saccade detector of the eye tracker was set to detect saccades with an amplitude of at least 0.5°, using an acceleration threshold of 9500°/s2 and a velocity threshold of 30°/s. A high-speed camera tracked the left eye, while a second camera tracked and compensated for head position by monitoring 4 infrared sensors placed on the corners of the display monitor. Cameras were mounted and held in place by a lightweight headband, which was placed and secured on the participants. Two computers were used in the experimental setup and were connected to each other via Ethernet, allowing for real-time transfer of saccade and gaze position data as well as response information. One computer collected the data from the eye tracker and displayed an image of the participant’s eye and calibration information. The other computer displayed the stimuli and recorded keypress responses. Procedure Gaze-Contingent group Prior to the set up of the apparatus, participants viewed a hierarchical digit (a global digit made up of repetitions of a local digit), and were asked to name the global and local digits to confirm that they understood the task. Digits were used as example and practice stimuli so that participants did not learn what the letters looked like before the experiment. Participants were informed that during the experiment they would be only seeing a small portion of the stimulus at one time, and that what was revealed was always contingent on where they were looking on the screen. Since many participants were unfamiliar with the concept of gaze-contingent displays, during the first practice trial the experimenter asked participants to follow her finger as she moved it across the screen, allowing participants to see how moving their eyes moved the gaze-contingent aperture (which moved along with the experimenter’s finger in concert with the participant’s eye movements). Participants were seated 50 cm from the screen of the display computer with their chin supported by a chin rest. They were asked to remove any eyewear unless it was necessary for reading letters on a computer screen. The eye monitor was placed on the participant’s head and securely fastened with a lightweight headband. 84  Eye movements were recorded monocularly from the left eye. The experimenter verified that the camera did not obstruct the participant’s view of the screen, and that the pupil was in view of the camera, even when the participant made eye movements to the far corners of the screen. The eye monitor was calibrated using a 9-dot array. Calibration was verified using the same procedure. After successful calibration and verification, the experimenter initiated a short block of practice trials. This consisted of 3 trials of global digits made up of repetitions of a local digit. The stimuli for these practice trials consisted of global digits 1, 4, and 7, with any digit from 1-9 at the local level presented at randomly chosen sizes and densities. Practice trials were performed with the gaze-contingent aperture. Participants were asked to name the global digit to ensure that they were performing the task properly. Upon successful completion of the practice trials, the experiment began. Each block started with the experimenter informing the participant of whether the task was to name the letters at the Global or Local level throughout the upcoming block. The participants then initiated the block by keypress. Each trial began with a fixation circle, which participants had to fixate accurately in order for the trial to proceed. When the participant accurately fixated the circle, they were able to initiate the stimulus onset by keypress. The fixation circle was removed for 500 ms at which point the stimulus appeared on the screen. Participants were given a maximum of 3 minutes per trial to identify the letter. When participants believed they knew the identity of the letter, they pressed space bar to terminate the trial. This led to a screen that prompted the participant to look down at the keyboard and to carefully enter their response. If they reached the time limit, the stimulus was replaced by a screen that informed them that they had run out of time and that they should enter a response based on what they thought the letter might have been. When the response was entered, a fixation circle appeared in preparation for the next trial. Participants were asked to press the space bar when they knew the identity of the letter to avoid recording eye movements as they searched the keyboard for their response. Participants did not receive feedback about their  85  performance. Trials were blocked by Size x Density configuration, and by Level (Global or Local). Participants performed 9 blocks during which they identified letters at the Local level (one for each size x density combination) and 3 blocks during which they identified letters of one size at the Global level. Participants were not asked to perform all 9 blocks of the Global trials because of anticipated participant fatigue. The size of the Global letters was counterbalanced across participants. Participants saw Global letters of all three densities, which were blocked and presented in random order. Each block consisted of 11 trials. Half the participants performed the Global identification blocks first, while the other half performed the Local identification blocks first. The order of the blocks within a level was randomized, as well as the trials within each block. Full-view control group All procedures were identical to the Gaze-Contingent group except that trials were performed under natural (unrestricted) viewing conditions and that participants performed the Global level task for all sizes and densities (9 blocks). The eye monitor was used with this group to match the procedure used with the GazeContingent group, but in this condition the eye monitor did not create a gazecontingent window. There was no time limit for full-view control participants to respond, but participants never reached the 3 minute maximum imposed on the Gaze-Contingent group. Patient SL All procedures were identical to the Gaze-Contingent group except that trials were performed under natural (unrestricted) viewing conditions without an eye monitor, and that SL performed the Global level task for all sizes and densities (9 blocks). Also, there was no time limit for SL, who made verbal responses that were entered by keypress by the experimenter.  86  Analysis and results We first performed a visual inspection of all eye movement plots for the GazeContingent and Full-View Controls and removed any trials for which there was a clear shift of the eye movements indicative of poor drift correction for that trial. Accuracy3 was calculated for all remaining trials by assigning a value of 0 to an incorrect response and 1 to a correct response. From this, a percent accuracy for each Size x Density condition was calculated (e.g. 8/11 = 0.727 or 72.7%). We then compared the accuracy for the Gaze-Contingent group to the accuracy of the FullView Control group in separate 3-way ANOVAs for Global and Local trials. The analysis of Local trials included the between-subjects factor of Group (GazeContingent vs. Full-View Controls), and within-subjects factors of Size (Small, Medium, Large) and Density (Sparse, Medium Density, Dense). Some participants from the Gaze-Contingent group did not complete all local conditions due to technical difficulties. Because participants in the Gaze-Contingent group performed global trials of one size only, the analysis of Global trials included the betweensubject factors of Group (Gaze-Contingent vs. Full-View Controls) and Size (Small, Medium, Large) and a within-subjects factor of Density (Sparse, Medium Density, Dense). Where appropriate, main effects and interactions were investigated with ttests. SL’s accuracy was calculated in the same way as it was for the GazeContingent and Full-view groups. To determine whether SL’s accuracy was significantly different from these groups, we performed Bayesian Standardized Difference tests (Crawford, J.R. (2009). SingleBayes. [Computer Software]. Retrieved September 10, 2009, from http://www.abdn.ac.uk/~psy086/dept/SingleCaseMethodsComputerPrograms.HTM) (Crawford, Garthwaite, & Howell, 2009; Sokal & Rohlf, 1995) comparing her accuracy to the accuracy of the Gaze-Contingent and Full-View Controls,  3  I do not report reaction times (RTs) because patient SL’s responses were entered by the experimenter and are therefore unreliable. The Gaze-Contingent group had long trial durations whereas the Full-View group responded almost instantly, hence any differences in RTs for these groups are relatively uninformative.  87  respectively, for each Size x Density condition. All alpha levels were set to p < 0.05. To account for multiple comparisons, p values were also compared to a Bonferonnicorrected alpha, but results were unaffected. When performance was at ceiling, ttests could not be performed because of zero variance. Local letter processing Figure 4.4(a) illustrates the accuracy data for the Full-View Controls, the Gaze-Contingent group, and SL for naming the Local letters in each Size-Density condition. a.  b.  Figure 4.4 Accuracy (%) for Gaze-Contingent Group, Full-View Controls, and SL, for identifying (a) Local letters, and (b) Global letters for each Density x Size condition.  Gaze-Contingent vs. Full-view controls Accuracy was perfect or near perfect for both groups (Gaze-Contingent = 99.4%; Full-View Controls = 100%), and therefore there were no significant main effects of Group, Size, or Density, and there were no interactions.  88  SL vs. Full-view controls The Full-View Control group performed at ceiling for all Local trials. SL also performed at ceiling for all but one condition (Small-Sparse = 90.9%). SL vs. Gaze-Contingent group SL’s accuracy was at ceiling for all conditions except Small-Sparse letters. In this condition, she performed significantly worse than the Gaze-Contingent group, SL = 90.9% vs. Gaze-Contingent = 99.6%, t (22) = -4.43, p < 0.001. The GazeContingent group was perfect or near-perfect for all conditions, and therefore did not differ from SL in any of the remaining conditions (all p > 0.10). Both SL and the Gaze-Contingent group reached ceiling for Small-Dense and Medium-Medium Density letters and therefore did not differ from each other in these conditions either. Summary The groups did not differ from each other in any way, except that SL performed worse than the normative groups on the Small-Sparse Local letters. Global letter processing Figure 4.4(b) shows the accuracy for each group for identifying Global letters. Gaze-Contingent vs. Full-view controls There was a main effect of Group, F(1,26) = 62.86, p < 0.001, (GazeContingent = 66.4%; Full-View Controls = 99.4%), reflecting the fact that the FullView Control group was significantly more accurate overall than the GazeContingent group. There was a main effect of Density, F(2,51) = 47.41, p < 0.001, (Sparse = 69.1%; Medium Density = 88.0%; Dense = 91.5%), reflecting the fact that, overall, participants performed worse in the Sparse conditions, compared to the Medium Density or Dense conditions: Sparse vs. Medium Density, t (47) = -5.50, p < 0.001; Sparse vs. Dense, t (47) = -6.13, p < 0.001; Medium Density vs. Dense, t (47) = -1.13 , p = 0.264. There was no main effect of Size, F(2,26) = 1.48, p = 0.247, (Small = 88.7%; Medium = 81.1%; Large = 79.7%). There was a Group x Density 89  interaction, F(2,51) = 39.29, p < 0.001. No other interactions were significant. The Group x Density interaction was further explored by a series of paired t-tests for each group as follows. These t-tests were compared to a Bonferroni corrected alpha (α’= 0.017) to account for multiple comparisons. Full-view controls The Full-View Control group showed no difference in accuracy for letters of different densities, Sparse = 98.1%, Medium = 100%, Dense = 100%; Sparse vs. Medium Density, t (23) = -2.00, p = 0.057; Sparse vs. Dense, t (23) = -2.00, p = 0.057; Medium Density vs. Dense, (at ceiling). Gaze-Contingent The accuracy of the Gaze-Contingent group was significantly worse for Sparse letters compared to Medium Density and Dense letters, Sparse = 40.2%, Medium Density = 76.1%, Dense = 83.2%; Sparse vs. Medium Density, t (23) = 7.62, p < 0.001; Sparse vs. Dense, t (23) = -10.28, p < 0.001; Medium Density vs. Density, t (23) = -1.13, p = 0.269. SL vs. Full-view controls SL performed significantly worse than the Full-View Control group for all Sparse letter conditions: Small-Sparse, t (8) = -15.74, p < 0.001; Medium-Sparse, t (8) = -23.77, p < 0.001; Large-Sparse, t (8) = -10.87, p < 0.001, and while the FullView Control group was at ceiling for all other conditions, SL was not. SL vs. Gaze-Contingent group SL’s accuracy did not differ from the Gaze-Contingent group’s accuracy in any condition (all p > 0.10). Summary The Full-View Control group performed better than the Gaze-Contingent group overall. Although the Full-View Control group was unaffected by the density of the letters, performing at ceiling, or near-ceiling, the Gaze-Contingent group was affected by stimulus density, performing worse on Sparse letters than Medium  90  Density or Dense letters. SL did not differ from the Gaze-Contingent group in any of the conditions.  Discussion Our results show that when restricted to seeing only a small portion of a stimulus at one time, healthy individuals are impaired at piecing together the global identity of hierarchical stimuli. In contrast to previous predictions in the literature, subjects were unable to keep track of the locations of previously viewed elements, as evidenced by their impaired ability to identify global letters, particularly when those letters are large and made up of widely spaced local elements. Furthermore, healthy participants under restricted viewing conditions show accuracy patterns that are remarkably similar to those of a patient with Bálint syndrome performing the same task under natural viewing conditions, suggesting that the global level impairment in Bálint syndrome is not due to an additional attentional deficit unique to the disorder, but may instead be the result of the narrowed attentional window, combined with normal limits to visual attention. Our gaze-contingent paradigm was designed to test whether the constriction of the spatial area of visual processing in simultanagnosia is alone sufficient to impair global level processing, including the ability to synthesize global shape by the serial perception of individual local elements. The ability to keep track of the relative location of stimuli relies on visual attention and working memory (Farah, 1990). Since visual attention is impaired in simultanagnosics but intact in healthy participants, one might expect that healthy participants would have little difficulty keeping track of the relative location of stimuli that are exposed over time by movement of the window across the letters. In contrast to this prediction, we found that even with normal visual attention, healthy participants viewing hierarchical letters through a restricted viewing window are unable to keep track of local elements in order to derive the identity of the global letter, suggesting that there are normal limitations on the use of visual working memory and attention to integrate global information under such viewing conditions.  91  Not only did our gaze-contingent window manipulation mimic SL’s overall accuracy at the global task, but it also produced similar patterns of performance across different stimulus densities. Previously, we suggested that inter-element spacing, rather than stimulus size per se, is key for determining global level performance in patients with Bálint syndrome (Dalrymple, et al., 2007). Like SL, our gaze-contingent group was most impaired at identifying global stimuli with large inter-element spacing, stimuli that would allow fewer local elements to fall in a narrowed viewing window compared to stimuli with local elements that are more densely packed. Our findings support the growing body of evidence that suggests that Visual Short Term Memory (VSTM) is normal in simultanagnosia (e.g. Duncan, et al., 2003; Huberle & Karnath, 2006). The finding that simultanagnosics struggle with largesparse letter stimuli, which have large inter-element spacing and which may therefore increase the VSTM load by increasing time between the perception of successive elements, suggests the possibility that a global processing deficit can be linked to a VSTM impairment. However, it has been argued that if simultanagnosia is linked to a limitation of VSTM, the addition of more elements (and therefore VSTM load) would lead to further decrements in performance, a prediction that is contradicted by the improvement that is seen when patients view hierarchical letters made up of several, densely packed elements compared to a few, sparse elements (Huberle & Karnath, 2006). The present findings support this notion because participants in our gaze-contingent group, who have normal VSTM, had similar inaccuracies with global report to our simultanagnosic patient SL. Furthermore, based on Budensen’s Theory of Visual Attention, others have suggested that rather than reduced VSTM capacity, the primary deficit in simultanagnosia is a limitation of processing capacity (Duncan, et al., 2003). Our findings with healthy participants support this idea and suggest that the inability to derive global shape in simultanagnosia may reflect a restricted window of visual processing combined with normal limits in general visual processing capacity that allow integration of visual information across time.  92  The visual deficits in Bálint syndrome were identified early on as being attentional in nature. Holmes and Horrax (1919) described the disorder as one of visual attention rather than blindness because their patient had variable perception of objects that fell on fully functioning retinas. “The essential feature was his inability to direct his attention to, and take cognizance of, two or more objects that threw their images on the seeing portion of his retinae. As this occurred no matter on what parts of his retinae the images fell, it must be attributed to a special disturbance or limitation of attention…” (Holmes & Horrax, 1919, p.390). The global processing deficits in simultanagnosia have been hypothesized to be related to “local capture”, that is, patients being “locked” on the local elements of an object at the expense of the global whole (Karnath, et al., 2000), perhaps due to an inability to disengage attention from those local elements (Farah, 1990). However, more recent evidence shows that despite poor report of the global level hierarchical stimuli, patients scan these stimuli extensively, arguing against an inability to disengage from individual parts of the stimulus (Clavagnier, et al., 2006; Dalrymple, Bischof, Cameron, Barton, & Kingstone, 2009). In those reports, eye movements were not predictive of the success at global level report in these patients. Others have described the attentional limitation in Bálint syndrome as being related to a restricted window of visual processing (e.g. Bay, 1953; Shalev & Humphreys, 2002; Shalev, et al., 2004; Shalev, Mevorach, & Humphreys, 2007; Thaiss & de Bleser, 1992; Tyler, 1968). Bay described it as a “peripheral constriction”, not unlike “viewing [a] picture through a diaphragm” (p. 545, 546). Thaiss and de Bleser (1992) suggested that their patient may suffer from a rigid reduction of the spatial extent of the visual “spotlight”. Tyler (1968) referred to the visual deficit in his patient as “shaft vision” (p. 166), yet implied some flexibility: when Tyler measured his patient’s effective visual fields he concluded that they were quite variable: while items were consistently perceived within 2° of fixation, perception could also occur for items at up to 20° eccentricity, though this more peripheral processing quickly fatigued, within 10-30s.  93  More recently, the flexibility of the restricted window of attention has been tested empirically with hierarchical stimuli. The ability to expand the window of attention from local to more global stimuli is partly determined by the stimulus itself. For example, patients viewing hierarchical letters made up of unfamiliar local elements (Hebrew letters), showed good performance for naming the global letters compared to when local items were familiar (English letters) (Shalev, et al., 2007). Similarly, priming the global level of a hierarchical letter with a solid letter that is the same size as the global level of the stimulus can improve global level report in simultanagnosia (Shalev, et al., 2004), in theory by expanding the window of attention prior to the presentation of the target stimulus. Finally, when viewing “globally biased” stimuli (hierarchical faces) patients actually show a type of “global capture”, in that they see only the global level of the stimulus without awareness of the local level (Dalrymple, et al., 2007). Based on these and other findings, some have suggested that the primary deficit in simultanagnosia could specifically involve an inability to expand a restricted window of attention (Shalev & Humphreys, 2002), with the default state being a relatively small area of useful visual field (e.g. global capture seems to occur less commonly than local capture in simultanagnosia). With global capture of hierarchical faces, the expansion of the window may occur at the expense of attentional acuity: patients can see the global face, but not the individual elements that make up the face. It is possible that SL’s successful global level report in this task reflects successful expansion of her restricted window of attention. However, the fact that participants in our gaze-contingent group who had a rigid restriction of vision show the same accuracy patterns as SL suggests instead that her attentional window remained fixed in this task. Indeed, one of the strengths of our model is its parsimony in that it models the simultanagnosic behaviours with a simple restriction of the visual window. This provides strong support for the idea that the restriction of attention in simultanagnosia is alone sufficient to explain the global processing deficit, regardless of how the restriction itself is manifested.  94  The idea of a restricted window of attention as a mechanism for simultanagnosia is consistent with the present results, that a literal restriction of vision is sufficient to cause Bálint-like global processing deficits in healthy participants viewing hierarchical letters. However, although the size of the gazecontingent window in our experiment appears to be an appropriate choice for mimicking patient behaviour with hierarchical letter stimuli, these results do not mean that this window size will replicate simultanagnosic performance with all stimuli, given the data showing that the attentional window varies with factors like priming at the global level and salience at the local level. For example, we do not anticipate that a rigid restricted window would lead to global capture behaviour in healthy participants viewing hierarchical face stimuli (Dalrymple, et al., 2007). However, it may be possible to design paradigms to simulate other hypothesized properties of the simultanagnosic window of attention with different stimuli. For example, the global capture effects patients experience with hierarchical faces may be simulated in healthy subjects by use of a gaze-contingent window that is small, but expandable, but with limited processing capacity mimicked by decreasing the spatial resolution within this larger window, limiting the processing of local elements. Our paradigm is primarily designed as a starting point in testing concepts related to the restricted window of attention in simultanagnosia, and provides multiple avenues for future studies of the properties of the visual attention window in simultanagnosia. In contrast to a restricted window of attention, one alternate explanation for the global perceptual deficits in simultanagnosia is that they result from damage to the right hemisphere, which has been implicated in playing a role in global processing, while the left hemisphere has been implicated in local processing (e.g. Delis, Robertson, & Efron, 1986; Robertson, Lamb, & Knight, 1988; van Kleeck, 1989). While damage to the right hemisphere could be related to the global processing deficit in simultanagnosia, simultanagnosia results from bilateral damage, leaving no a priori reason to predict preferential processing of one stimulus level over the other. Furthermore, we have shown with patient SL that “global capture” can occur with globally-biased stimuli, (i.e. hierarchical faces, Dalrymple et al, 2007), demonstrating that simultanagnosia is not characterized by a local 95  preference per se, as may be the case with right hemisphere damage alone. Rather than a selective global deficit from right hemisphere damage, we suggest that patients have preference for the local elements of hierarchical letters because these elements fit into a narrowed window of visual attention. This explanation is consistent with the improvement in global level report seen with patients, and with our gaze-contingent group, for global letters that are smaller and more densely packed. These stimuli allow for more of the global stimulus to occupy the narrowed window of vision at one time and therefore lead to better global level report. We thank an anonymous reviewer for suggesting an alternative explanation for SL’s impaired global report, that it is the result of poor ocular motor control from the other deficits of Bálint syndrome. SL indeed shows evidence of difficulties with the accurate execution of voluntary eye movements (ocular motor apraxia). However, Clavagnier and colleagues (2006) monitored the eye movements of two simultanagnosic patients while they identified the global and local levels of hierarchical letter stimuli and found that the patients’ eye movements were abnormal but not predictive of performance on the letter identification task. We recently performed a comparable experiment with SL and similarly found that her eye movements were not predictive of her accuracy at identifying global or local letter stimuli (Dalrymple, et al., 2009). Rather, we found that SL’s abnormal eye movements were the consequence rather than the cause of her difficulties with global level report suggesting that her ocular motor apraxia was not to blame. Our current findings with healthy participants in the gaze-contingent paradigm are again consistent with this conclusion. These participants do no have ocular motor deficiencies, yet showed global level report difficulties similar to SL’s. This further supports the idea that seeing only a small portion of the stimulus at one time, rather than disordered eye movements, is crucial to difficulties with global level report. Having demonstrated that our manipulation was successful in creating Bálintlike accuracy patterns for this particular 2-dimensional task, it might be possible to extend this effect to other domains. Patients with Bálint syndrome suffer from other visual-spatial deficits, such as spatial disorientation, optic ataxia, and ocular apraxia  96  (Holmes & Horrax, 1919). While some have considered the possibility that simultanagnosia is also responsible for these other deficits (A. R. Luria, E. N. Pravdina-Vinarskaya, & A. L. Yarbus, 1962), others argue that each deficit is dissociable from the others (Cummings, Houlihan, & Hill, 1986; Hecaen & de Ajuriaguerra, 1954; Luria, A., Pravdina-Vinarskaya, E., & Yarbus, A., 1962), and modern neuroimaging suggests that the anatomic substrates of each differ. Nevertheless, it remains possible that simultanagnosic limitations of processing in two and three dimensions may affect visual reaching and saccadic targeting. For instance, it is possible that participants placed in a 3-dimensional environment where they were restricted to seeing a single object at one time would have difficulties reaching for the objects, akin to optic ataxia. Extending our simulation to a 3D environment may allow us to determine the contribution of limited windows of processing to reaching and ocular motor deficits in Bálint syndrome. In summary, we have shown that when restricted to seeing only a small portion of a display at one time, healthy participants show difficulties with global level perception of hierarchical stimuli. This behaviour is well documented in patients with simultanagnosia, who suffer from a constriction of the spatial extent of their visual window of attention. Even with protracted viewing times, our participants, like simultanagnosics, were unable to correctly derive the global level of hierarchical stimuli, suggesting a difficulty keeping track of the relative locations of the individual elements viewed through serial fixations. Our findings suggest that the constriction of the spatial area of visual processing in simultanagnosia may itself contribute to this difficulty, rather than some additional impairment of visual attention resulting from parietal damage. We propose that parietal damage may cause a restriction of the spatial extent of the attentional window that limits the amount of visual information that can be processed at one time, and this restriction of visual information, combined with normal limits to visual processing, affects the ability to keep track of the relative position of the elements of a scene leading to a piecemeal view of the world.  97  References Balint, R. (1909). Seelenlähmung des "Schauens", optische Ataxia, räumliche Störung der Aufmerksamkeit. Monatsschrift für Psychiatrie und Neurologie, 25, 51-81. Bay, E. (1953). Disturbances of visual perception and their examination. Brain, 76(4), 515-550. Clavagnier, S., Fruhmann Berger, M., Klockgether, T., Moskau, S., & Karnath, H.-O. (2006). Restricted ocular exploration does not seem to explain simultanagnosia. Neuropsychologia, 44, 2330-2336. Crawford, J., Garthwaite, P., & Howell, D. (2009). Neuropsychologia, 47(13), 26902695. Cummings, J., Houlihan, J., & Hill, M. (1986). The pattern of reading deterioration in dementia of the Alzheimer type. Brain and Language, 29, 315-323. Dalrymple, K., Bischof, W., Cameron, D., Barton, J., & Kingstone, A. (2009). Global perception in simultanagnosia is not as simple as a game of connect-the-dots. Vision Research, 49, 1901-1908. Dalrymple, K., Kingstone, A., & Barton, J. (2007). Seeing trees OR seeing forests in simultanagnosia: Attentional capture can be local or global Neuropsychologia, 45, 871-875. Delis, D., Robertson, L., & Efron, R. (1986). Hemispheric specialization of memory for visual hierarchical stimuli. Neuropsychologia, 24, 205-214. Duncan, J., Bundesen, C., Olson, A., Humphreys, G., Ward, R., Kyllingsbaek, S., et al. (2003). Attentional functions in dorsal and ventral simultanagnosia. Cognitive Neuropsychology, 20(8), 675-701. Farah, M. (1990). Visual agnosia: Disorders of object recognition and what they tell us about normal vision. Cambridge: MIT Press. Goodglass, H., & Kaplan, E. (1983). The assessment of aphasia and related disorders (2nd ed.). Philadephia: Lea and Febiger. Hecaen, H., & de Ajuriaguerra, J. (1954). Balint's syndrome (psychic paralysis of visual fixation) and its minor forms. Brain, 77, 373-400. Holmes, G., & Horrax, G. (1919). Disturbances of spatial orientation and visual attention, with loss of stereoscopic vision. Archives of Neurology and Psychiatry, 1, 385-407. Huberle, E., & Karnath, H.-O. (2006). Global shape recognition is modulated by the spatial distance of local elements - Evidence from simultanagnosia. Neuropsychologia, 44(6), 905-911. 98  Karnath, H.-O., Ferber, S., Rorden, C., & Driver, J. (2000). The fate of global information in dorsal simultanagnosia. Neurocase, 6, 295-306. Loschky, L., McConkie, G., Yang, J., & Miller, M. (2005). The limits of visual resolution in natural scene viewing. Visual Cognition, 12(6), 1057-1092. Luria, A., Pravdina-Vinarskaya, E., & Yarbus, A. (1962). Disturbances of ocular movement in a case of simultanagosia. Brain, 86, 219-228. Luria, A. R., Pravdina-Vinarskaya, E. N., & Yarbus, A. L. (1962). Disturbances of ocular movement in a case of simultanagnosia. Brain, 86, 219-228. McConkie, G., & Rayner, K. (1975). The span of the effective stimulus during a fixation in reading. Perception and Psychophysics, 17, 578-586. Moreaud, O. (2003). Balint Syndrome. Archives of Neurology, 60, 1329-1331. Navon, D. (1977). Forest before trees: The precedence of global features in visual perception. Cognitive Psychology, 9, 353-383. Pomplun, M., Reingold, E., & Shen, J. (2001). Investigating the visual span in comparative search: the effects of task difficulty and divided attention. Cognition, 81, B57-B67. Rafal, R. (2003). Balint's syndrome: A disorder of visual cognition. In M. D'Esposito (Ed.), Neurological foundations of cognitive neuroscience (pp. 27-40). Cambridge, MA: MIT Press. Rizzo, M., & Vecera, S. (2002). Psychoanatomical substrates of Balint's syndrome. Journal of Neurology, Neurosurgery and Psychiatry, 72, 162-178. Robertson, L., Lamb, M., & Knight, R. (1988). Effects of lesions of temporal-parietal junction on perceptual and attentional processing in humans. The Journal of Neuroscience, 8, 3757-3769. Shalev, L., & Humphreys, G. (2002). Implicit location encoding via stored representations of familiar objects: Neuropsychological evidence. Cognitive Neuropsychology, 19(8), 721-744. Shalev, L., Humphreys, G., & Mevorach, C. (2004). Global processing of compound letters in a patient with Balint's syndrome. Cognitive Neuropsychology, 22(6), 737-751. Shalev, L., Mevorach, C., & Humphreys, G. (2007). Local capture in Balint's syndrome: Effects of grouping and item familiarity. Cognitive Neuropsychology, 24(1), 115-127. Sokal, R., & Rohlf, J. (1995). Biometry. San Francisco, CA: W.H. Freeman. Thaiss, L., & de Bleser, R. (1992). Visual agnosia: A case of reduced attentional 'spotlight'? Cortex, 28(4), 601-621. 99  Tyler, R. (1968). Abnormalities of perception with defective eye movements (Balint's Syndrome). Cortex, 4(2), 154-171. van Kleeck, M. (1989). Hemispheric differences in global versus local processing of hierarchical visual stimuli by normal subjects: New data and a meta-analysis of previous studies. Neuropsychologia, 27(9), 1165-1178. Wolpert, I. (1924). Die simultanagnosie: Störung der Geamtauffassung. Zeitschr Ges Neurol Psychiatrie, 93, 397-415.  100  Part 2: Complex stimuli  101  Chapter 51 Experiencing simultanagnosia through windowed viewing of complex social scenes  1  A version of this chapter has been submitted for publication. Dalrymple, KA, Birmingham, E, Bischof, WF, Barton, JJS, & Kingstone A. Experiencing simultanagnosia through windowed viewing of complex social scenes  102  Bálint syndrome is a complex disorder of visual attention that typically results from bilateral lesions to the parieto-occipital junction (Balint, 1909). It is characterized by four primary symptoms: (1) simultanagnosia: a restricted spotlight of visual attention resulting in an inability to see more than one object at a time; (2) spatial disorientation: an inability to locate objects in space; (3) optic ataxia: an inability to use visual information to guide accurate reaching towards objects; and (4) ocular motor apraxia: an inability to voluntarily execute accurate eye movements (Moreaud, 2003; Rafal, 2003; Rizzo & Vecera, 2002). These symptoms can be so severe that patients appear “functionally blind” (Kim & Robertson, 2001), with little or no understanding of the fragmented world they perceive. Their perception is often ‘captured’ by local elements of a scene, in that they report local details of a scene or object at the expense of the global whole (Karnath, Ferber, Rorden, & Driver, 2000). For example, when viewing impoverished stimuli like hierarchical letters, which consist of global letters made up of several repetitions of a local letter, patients report only the local letters of large, sparse, stimuli (Dalrymple, Kingstone, & Barton, 2007; Huberle & Karnath, 2006). Recently we showed that the local capture of simultanagnosia could be modeled in healthy subjects by using an artificially limited field of vision to simulate a spatially restricted zone of attentional processing (Dalrymple, Bischof, Cameron, Barton, & Kingstone, 2010). Subjects viewed hierarchical letters through a gazecontingent aperture, which allowed them to see only a small part of the stimulus at one time. Healthy subjects under this limitation showed patterns of inaccuracy that were highly similar to those seen in a patient (SL), in that they identified local elements well but not the global letters, particularly when those letters were large and had widely spaced local elements, despite having unlimited time to accomplish the task. This suggests that a narrowed window of visual processing - from either a small visual window in our simulation in healthy subjects or a constricted focus of attention in simultanagnosia, coupled with normal limits in the ability to integrate spatial information across fixations, is sufficient to account for the poor global report in Bálint syndrome.  103  A handful of studies have investigated where simultanagnosics look when they explore these and other impoverished stimuli (e.g. Clavagnier, Fruhmann Berger, Klockgether, Moskau, & Karnath, 2006; Dalrymple, Bischof, Cameron, Barton, & Kingstone, 2009; Nyffeler, et al., 2005; Tyler, 1968), but little work has systematically investigated how simultanagnosics scan more complex stimuli, such as social scenes. Tyler (1968) recorded the eye movements of a patient who looked at a photograph of five dolls and a line drawing of a desert scene, but these stimuli were still quite simple, and the eye movement patterns were described but not quantified. It remains unclear how patients with simultanagnosia explore more complex stimuli, which may better represent their experience in the real world. The purpose of the present study was two-fold. Our first aim was to assess and quantify how simultanagnosic patients scan complex social stimuli. Social scenes were used because much is known about how healthy individuals explore these scenes (e.g. Birmingham, Bischof, & Kingstone, 2007, 2008a, 2008b; Smilek, Birmingham, Cameron, Bischof, & Kingstone, 2006). Our second aim was to assess how their scanning behaviour relates to the spatial constriction of visual processing that underlies the simultanagnosic deficit, by using our gaze-contingent paradigm to model this restriction of processing in healthy subjects. If a spatial constriction of processing itself is pivotal to the visual exploration in simultanagnosia, our paradigm of seeing only a small portion of a stimulus at a time should generate simultanagnosic behaviours in healthy subjects engaging in a complex task, such as describing complex scenes. In Study 4, we studied the distribution of fixations made during scanning and reporting of complex scenes, and compared two patients with Bálint syndrome (SL and KC) to healthy subjects and to a brain damaged control (ES). In Study 5 we investigated how healthy subjects scan the scenes while viewing them through a gaze-contingent aperture that allowed them to see only a small portion of the scene at one time. We then compared their scanning patterns to those of SL, KC, and the control subjects from Study 4. In Study 6 we compared our gaze-contingent paradigm to a mouse-contingent paradigm to determine whether the restricted  104  viewing window itself is crucial to produce the Bálint-like behaviours, or whether our results from Study 5 were specifically related to controlling the window with the eyes. If our paradigm is valid and a spatially constricted window of visual processing indeed contributes to complex ocular motor behaviours seen in Bálint syndrome, we predict that healthy subjects viewing scenes through a small aperture should show scanning patterns that more closely match SL and KC’s scanning patterns than those of control subjects with unrestricted viewing. This should be the case regardless of the method of controlling the viewing aperture (gaze-contingent vs. mouse-contingent).  Study 4 The goal of this Study was to establish the scanning patterns of healthy subjects viewing social scenes under normal viewing conditions and to characterize the scan patterns made by two simultanagnosic patients viewing the same stimuli.  Method Participants Control subjects Control participants (n = 8; 5 male) were undergraduate students at the University of British Columbia who ranged in age from 17-34 years (mean = 22 years). All participants reported normal or corrected-to-normal vision and gave informed consent prior to participation in the experiments, which were performed in accordance with the ethical guidelines of the University of British Columbia. Case reports SL is a 48 year-old right-handed woman, with 12 years of education. She had idiopathic cerebral vasculitis resulting in bilateral parietal and lateral occipital infarcts (Figure 5.1a). She presented with left hemi-neglect, as assessed with the Sunnybrook Neglect Assessment Battery (Leibovitch, et al., 1998) and Bálint syndrome, with ocular motor apraxia, optic ataxia, and simultanagnosia. Her visual exam showed Snellen acuity of 20/25 in both eyes, and a left inferior  105  quadrantanopia. Her optic ataxia was manifested by misreaching for objects and failure to orient her grasp correctly to the axes of objects such as pencils. Her simultanagnosia was evident in testing with four complex displays of visual scenes. For example, she could report elements of the Boston Cookie Theft picture (Goodglass & Kaplan, 1983), but was unable to make sense of the whole scene. She initially reported seeing only “a boy’s face… eyes,” without reporting the mother on the right side of the display or the second child in the scene, nor did she describe the action in the scene. Neuropsychological evaluation showed normal attention, language, and verbal memory functions. Her reading was in the borderline impaired range and she tended to guess words based on the first or last letters. She was successful at recognizing simple line drawings of objects and could correctly identify colours and simple shapes. At the time of testing, SL had completed treatment with cyclophosphamide and prednisone 4 months prior, but was still taking carbamazepine for a single seizure suffered several months prior. She no longer had left hemi-neglect, quadrantanopia, or defects in saccadic targeting and generation, as confirmed by her rapid and accurate saccades during the calibration of the eye monitor. She still had optic ataxia when using the left hand to point to targets. This was a specific sensorimotor transformation for the contralateral hand, not a general difficulty with perceptual localization, which would affect both hands. Patient SL has been discussed in previous reports (i.e. Dalrymple, et al., 2007; Malcolm & Barton, 2007).  106  a.  b.  c.  Figure 5.1. MRI scans of patients SL (a), KC (b), and control patient ES (c).  107  KC is a 55 year-old man with posterior reversible leucoencephalopathy syndrome in the setting of Crohn’s disease being treated with omisartan as part of an experimental trial. He was seen 2 months prior to testing for fluctuating visual symptoms of several weeks’ duration. He stated that he “could see but not perceive”. He could see things and recognize them, but had trouble locating and searching for household objects, and could not reach for items accurately. He saw an “echo” or multiple ghosts of objects when he stared at them. His reading was slow and at his worst he had trouble recognizing faces and difficulty with locating objects in depth. His acuity with correction at far was 20/30 od and 20/40 os, which improved by pinhole to 20/25-1 os. Confrontation showed full visual fields. Fixation was steady; pursuit and VOR cancellation were normal. He showed normal initiation of saccades and saccadic accuracy. There was no nystagmus. His reaching was accurate, and he showed correct grasp orientation to objects. Reading was slow, but accurate and without a word-length effect. Line cancellation and object cancellation showed significant errors, but these were not lateralized. His recognition of line drawings was normal. With the Boston Cookie Theft picture, he eventually named all objects but did so slowly in a fragmented fashion and did not relate the items to each other. In summary, his clinical findings were consistent with optic ataxia, simultanagnosia (though milder than SL), and palinopsia. His MRI showed bilateral parietooccipital and right posterior occipital white matter FLAIR hyperintensities, as well as a small left occipital cortical infarct (Figure 5.1b). Patient ES, like SL and KC, also suffered bilateral posterior occipitoparietal damage (Figure 5.1c). Unlike the other two patients, however, ES never had signs of simultanagnosia or symptoms suggestive of any component of Balint syndrome. She is a 47 year-old woman with systemic lupus erythematosis, tested several months after presenting with flashing lights, transient visual loss and headache. Her visual examination was normal, but MR imaging revealed bilateral lesions consistent with either vasculitis or posterior leucoencephalopathy. Subsequently she had a seizure, and was treated with phenytoin for 9 months. At her most recent visit she was taking prednisone, chloroquine, and mycophenolate mofetil. Her visual acuity without correction at far was 20/20 in both eyes. Confrontation showed full visual fields. 108  Fixation, pursuit and saccades were normal. There was no oculomotor apraxia, optic ataxia, or simultanagnosia as shown by normal report on the Boston Cookie Theft picture. ES matched SL particularly well in age, gender, the chronic phase at testing, and probable pathology, since she also has an underlying condition that is associated with vasculitis. Stimuli and apparatus Full color images were taken with a digital camera in different rooms in the Psychology building at the University of British Columbia. Image size was 36.5 x 27.5 (cm) corresponding to 40.1° x 30.8° at the viewing distance of 50 cm, and image resolution was 800 x 600 pixels. Twelve scenes were used in the present experiment. Scenes contained 3 persons in either an “interactive” or “noninteractive” state. All scenes were comparable in terms of their basic layout: each room had a table, chairs, objects, and background items (e.g. see Figure 5.2a).  a.  b.  Figure 5.2. Example of scene stimuli (a) and regions of interest (b).  109  Eye movements were monitored using the EyeLink II system (SR Research Ltd., www.eyelinkinfo.com). The Eyelink II has a temporal resolution of 4 ms (sampling rate 250 Hz) and a spatial resolution of 0.5°. One high-speed camera tracked the left eye, while a second camera tracked and compensated for head position by monitoring 4 infrared sensors placed on the corners of the display monitor. Cameras were mounted and held in place by a lightweight headband, which was placed and secured on the subjects. Patients ES and KC were tested at a later date on the Eyelink 1000 system2, which differs from the Eyelink II in that it has a temporal resolution of 1ms (sampling rate 1000Hz) and mounts the cameras on the desktop, rather than on a headband. Two computers were used in the experimental setup and were connected to each other via Ethernet, allowing for real-time transfer of saccade and gaze position data. The Experimenter computer collected the data from the eye tracker and displayed an image of the participant’s eye and calibration information. The Display computer displayed the stimuli and recorded key-presses. Procedure Subjects were seated 50 cm from the screen of the display computer with their chin supported by a chin rest. The eye monitor was placed on the subject’s head and securely fastened with a lightweight headband. Eye movements were recorded monocularly from the left eye. The eye monitor was calibrated using a 9dot array. Calibration was validated using the same procedure. After successful calibration and validation, the subject was asked to fixate a dot at centre-screen in order to correct for drift in gaze position. Once the dot was fixated, the experimenter initiated the onset of the scene image by key-press. The scenes were presented in random order. Subjects were asked to verbally describe the scene while a digital voice recorder recorded their response. They had unlimited 2  The Eyelink 1000 was used for ES and KC because of equipment upgrades that took place after initial testing with the other groups. Both Eyelink systems have the same 0.5° Gaze Accuracy, and the difference between the Eyelink II and Eyelink 1000 sampling rates does not affect the analysis of fixation frequencies, therefore the use of two different systems should not impact our results.  110  time to describe the scene, and indicated that their description was complete by saying “Next”. At this point, a keypress initiated the next trial. Subjects each viewed 8 social scenes. Analysis Fixation proportions For each image, an outline was drawn around each region of interest (e.g. "eye") and each region’s pixel coordinates and area were recorded. We defined the following regions: Eye, Head (excluding eyes), Body (including arms, torso and legs), foreground Objects (e.g., tables, chairs, objects on the table) and Background (e.g., walls, shelves, items on the walls). Figure 5.2b illustrates these regions for one scene. To compensate for the different sizes of these regions, we computed area-normalized fixation proportions (Birmingham, et al., 2008a; Smilek, et al., 2006), by first dividing the number of fixations in each region by the area of the region separately for each image and each participant, and then computing proportions based on these normalized data3. The data from one control subject was excluded because of a large offset in eye position due to a problem with the head-mounted camera. To compare each patient’s fixations to the control group, we performed 2-tailed t-tests on the fixation proportions for each region. In any instance where Levene’s test for Equal Variance was violated, a more conservative, Aspen-Welch Unequal Variance t-test was performed and the adjusted degrees of freedom are reported. T-test p values were compared to a corrected α=0.01, to account for multiple comparisons. These results are presented in Figure 5.3.  3  We also calculated and analyzed normalized fixation durations for each region ((total duration of fixations to a region/area of that region) * sum over all regions), but as these results mirrored to the results from the fixation proportions analyses, they are not reported. Saccadic distributions for each group showed no systematic differences, so these too, are not reported further.  111  Figure 5.3. Fixation proportions for Control subjects and patients SL, KC, and ES, who viewed the scenes under natural viewing conditions; Gaze-Contingent subjects who viewed the scenes through a small gaze-contingent aperture; and MouseContingent subjects who viewed the scenes through a small mouse-contingent aperture. Fixation proportions are normalized for the size of the region. Error bars represent standard error. Mean saccade amplitude, average number of fixations per trial and average trial durations In addition to the fixation proportions, we calculated the mean saccade amplitude (degrees of visual angle), average number of fixations per trial (to all regions), and the average time spent describing each scene (i.e. the average duration of each trial) in milliseconds (Figure 5.4). We compared SL, KC, and ES to Controls on these measures with 2-tailed t-tests. In any instance where Levene’s test for Equal Variance was violated, a more conservative, Aspen-Welch Unequal Variance t-test was performed and the adjusted degrees of freedom are reported. All p values were compared to a corrected α value of 0.01, to account for multiple comparisons.  112  a.  b.  c.  Figure 5.4. Mean saccade amplitudes (in degrees of visual angle) (a), average number of fixations per trial (b) and average time spent describing each scene (in seconds) (c) for Control subjects (C) and patients SL, KC, and ES, who viewed the scenes under natural viewing conditions; Gaze-Contingent (G-C) subjects who viewed the scenes through a small gaze-contingent aperture; and Mouse-Contingent (M-C) subjects who viewed the scenes through a small mouse-contingent aperture. Error bars represent standard error.  113  Results Summary of main findings. Control subjects had higher proportion of fixations on the Eye region compared to SL and KC. Neither SL nor KC differed from Control subjects in the proportion of fixations on the Head region, but both patients had a significantly larger proportion of fixations on the Body region compared to Controls. Neither of these patients differed from Controls in the proportion of fixations on the Object region, but SL had more fixations than Controls to the Background region. Control patient ES did not differ from the Control group in terms of her fixations to the different regions. Thus, while Control subjects make a disproportionately high number of fixations on the Eye region, SL and KC’s fixations are more distributed across regions, with abnormally low proportions of fixations to the Eye region. All patients made smaller saccades than Controls. SL made more fixations and took longer to describe the scenes than Controls, but KC and ES did not differ from Controls in terms of these measures. These results are presented in detail below. SL vs. Controls. Control subjects had a higher proportion of fixations on the Eye region than SL, t(62) = 5.29, p < 0.001, but SL had a higher proportion of fixations on the Body, t(7.51) = -4.88, p = 0.001, and Background, t(62) = -6.57, p < 0.001, regions, compared to Controls. These groups did not differ in terms of fixations on the Head, t(62) = -1.14, p = 0.260, or Object, t(7.46) = -2.44, p = 0.043, regions. Compared to Controls, SL made significantly smaller saccades t(22.14) = 7.39, p < 0.001, more fixations per trial, t(62) = -4.06, p < 0.001, and took significantly longer to describe each scene, t(62) = -5.13, p < 0.001. KC vs Controls. Control subjects had a higher proportion of fixations on the Eye region than KC, t(62) = -3.60, p < 0.001, but KC had a higher proportion of fixations on the Body, t(62) = 6.66, p < 0.001, than Controls. These groups did not differ in the proportion of fixations on the Head, t(62) = 2.15, p = 0.036, Object, t(62) = -0.47, p = 0.639, and Background, t(62) = 0.35, p = 0.729, regions. KC made significantly smaller saccades than Controls, t(28.30) = 11.30, p < 0.001, but he did not differ from them in terms of the average number of fixations per scene, t(62) = -1.51, p = 0.135, or the average duration of each trial t(62) = -1.96, p = 0.055. 114  ES vs. Controls. ES and the healthy Control subjects did not differ in terms of the proportion of fixations to any region: Eye, t(62) = 1.02, p = 0.312; Head, t(62) = 0.38, p = 0.708; Body, t(62) = -1.15, p= 0.256; Object, t(62) = -1.14, p = 0.170; Background, t(62) = -1.41, p = 0.165, nor did they differ in terms of the average number of fixations per scene, t(62) = 0.27, p = 0.788, or the average time spent describing the scenes, t(62) = 1.32, p = 0.192. ES’ saccades were significantly smaller than those of Controls, t(25.37) = 7.20, p < 0.001.  Discussion We monitored the eye movements of two simultanagnosic patients, SL and KC, while they viewed social scenes to determine whether they allocate their fixations differently from control subjects. Our regions of interest analysis showed that SL and KC had an abnormally low proportion of fixations on the eye region of the scenes compared to control subjects. This behaviour is particularly noteworthy because it is well documented that healthy subjects typically allocate disproportionately high numbers of fixations to eyes in social scenes (Birmingham, et al., 2007, 2008a, 2008b; Smilek, et al., 2006). SL, but not KC, took longer to describe the scenes, and made more fixations per trial compared to healthy control subjects. The fixation distributions of control patient ES, who had similar lesions to SL and KC, but no simultanagnosia, did not differ from those of healthy controls in any way, nor did her average number of fixations per trial, or her average time spent describing each scene. Interestingly, all patients made significantly smaller saccades compared to the healthy control group. This finding of abnormally small saccades for patients SL and KC is consistent with the piecemeal exploration of visual stimuli associated with simultanagnosia (Holmes & Horrax, 1919). However, the finding that ES, who is not simultanagnosic, also showed small saccades relative to healthy controls suggests that small saccadic amplitudes is not sufficient to characterize simultanagnosia. Rather, simultanagnosic exploration of scenes seems to be characterized both by small saccades and a specific reduction in fixations to the eyes.  115  Study 5 This study was designed to test whether healthy subjects viewing social scenes through a spatially constricted viewing window would show fixation patterns similar to those of a patient with simultanagnosia. By using an eye-monitor to produce a small gaze-contingent aperture, healthy subjects saw only a small portion of the scenes around their point of fixation at any point in time. Gaze-contingent displays have been used in the past with a variety of tasks, such as reading (McConkie & Rayner, 1975), visual search (Pomplun, Reingold, & Shen, 2001), and scene exploration (Loschky, McConkie, Yang, & Miller, 2005) and even to simulate simultanagnosic behaviour with simple stimuli (Dalrymple, et al., 2010). If the findings from our simultanagnosic patients in Study 4 are related to a restricted window of visual processing, then healthy subjects viewing scenes through a restricted gaze-contingent visual window should show similar fixation proportions on the different regions. That is, they should show reduced fixations on the eyes in social scenes compared to the control subjects of Study 4, who viewed the scenes under unrestricted viewing conditions.  Method Participants Subjects (n = 14, 4 male) were undergraduate students at the University of British Columbia who ranged in age from 18 to 24 years (mean = 20 years). All participants reported normal or corrected-to-normal vision and gave informed consent prior to participation in the experiments, which were performed in accordance with the ethical guidelines of the University of British Columbia. Stimuli, apparatus and procedure The stimuli and procedure for Study 5 were the same as Study 4, except that participants viewed the scenes through a gaze-contingent aperture. A 2°x2° square aperture was generated by the computer, and revealed the portion of the stimulus centered on the point of fixation, the screen being white elsewhere: the aperture  116  moved as the subject moved their fixation across the scene. Subjects underwent a practice trial in which they were instructed to start from a circle at centre screen labeled “Start” and follow a line from that circle until they reached a second circle labeled “End”. They were then instructed to freely search the screen for a hidden object on the screen. This was designed to familiarize them with, and teach them how to control, the gaze-contingent aperture. Once they located the hidden object and felt comfortable with the apparatus, the task began. Like subjects in Study 4, the Gaze-Contingent group was asked to describe each scene. Analysis We compared the data from the Gaze-Contingent group to SL, KC, and the Control group from Study 4 (because ES did not differ from healthy controls with unrestricted viewing conditions in terms of her fixation proportions, her data was not analyzed further). For all measures, t-tests were used to compare the GazeContingent group to SL, and KC, respectively. T-tests were also used to compare the Gaze-Contingent group to the Control group in terms of proportions of fixations to each region. Mean saccade amplitude, average number of fixations per trial, and average duration per trial for these groups were compared using 1-way repeated measures ANOVAs with between-subjects factor of Group.4  Results Summary of main findings. The Gaze-Contingent group differed from the Control group in the same ways that SL and KC differed from the Control group - most notably by having an abnormally low proportion of fixations on the Eye region and abnormally small saccade amplitudes. The Gaze-Contingent group and KC did not differ from each other in fixations to the Eye region, but the Gaze-Contingent group had somewhat more fixations on this region compared to SL. The Gaze-Contingent 4  For all experiments t-tests were used to compare individuals to groups (e.g. SL to Controls) and ANOVAs were used to compare groups to groups (e.g. Gaze-Contingent group to Controls).  117  group took longer and made more fixations than Controls, but did not differ from SL or KC on these measures. These results are presented in detail below. Gaze-Contingent vs. Controls. Compared to the Control group, the Gaze-Contingent group had a lower proportion of fixations on the Eye region, t(166) = 5.88, p < 0.001, but a higher proportion on the Body, t(166) = -7.26, p < 0.001, Object, t(166) = -3.88, p < 0.001 and Background, t(163.48) = -9.86, p < 0.001, regions. There was no difference between these groups in the proportion of fixations on the Head region, t(80.74) = -0.18, p = 0.855. The Gaze-Contingent group made significantly smaller saccades, F(1,19) = 265.88, p < 0.001, significantly more fixations per trial, F(1,19) = 16.63, p < 0.001, and took significantly more time to describe each scene than Controls, F(1,19) = 16.80, p < 0.001. Gaze-Contingent vs. SL. Compared to SL, the Gaze-Contingent group had a higher proportion of fixations on the Eye region, t(118) = 3.63, p < 0.001, but a lower proportion of fixations on the Body region, t(118) = -4.52, p < 0.001. SL and the Gaze-Contingent group did not differ in the proportion of fixations on the Head, t(118) = -1.58, p = 0.117, Object, t(118) = -0.81, p = 0.419, or Background, t(118) = 1.00, p = 0.319, regions. The Gaze-Contingent group made smaller saccades than SL, t(118)= -3.00, p = 0.003, but did not differ from her in terms of the number of fixations made per trial, t(118) = -0.25, p = 0.800, or in the average time spent describing each scene, t(118) = -2.99, p = 0.765. Gaze-Contingent vs KC. The Gaze-Contingent group and KC did not differ in the proportion of fixations on the Eye, t(118) = -1.57, p = 0.118, Head, t(7.36)=1.96,p=0.088, Body, t(118) = 2.61, p = 0.032, or Object, t(118) = -1.75, p = 0.083 regions, but the Gaze-Contingent group had higher fixation proportions on the Background region compared to KC, t(118) = -3.12, p = 0.002. The Gaze-Contingent group and KC did not differ in terms of mean saccade amplitudes, t(12.78) = -1.65, p = 0.124, average number of fixations made per trial, t(118) = 2.08, p = 0.040, or in the average time spent describing each scene, t(118) = 2.10, p = 0.038.  118  Discussion This experiment was designed to test whether a constricted field of visual processing when healthy subjects viewed social scenes would simulate the scanning behaviour of a patient with simultanagnosia. We predicted that healthy subjects under these conditions would show low fixation proportions to the eyes in social scenes, similar to simultanagnosic patients SL and KC. Our results show that, like those patients, our gaze-contingent group fixated the eyes significantly less than the control subjects under natural viewing conditions. The gaze-contingent group matched KC in terms of fixations to almost all regions. All groups fixated the head region equally. The gaze-contingent group made smaller saccades than controls and SL, but these saccades did not differ in amplitude from those of KC. The gazecontingent group made more fixations per trial and spent longer describing each scene than controls, but, again, did not differ from SL or KC on these measures. These results suggest that the restriction of visual information imposed on the gaze-contingent group made them perform more similarly to our simultanagnosic patients SL and KC, than to control subjects who viewed the scenes under natural viewing conditions. In particular, like SL and KC, the gaze-contingent group made an abnormally low proportion of fixations to the Eye region, while allocating an abnormally high proportion of fixations to the Body, Object, and Background regions. Furthermore, the saccade amplitude data suggest that the gaze-contingent group explored the scenes in a similar way to the patients – i.e., with a series of small saccades.  Study 6 Before concluding from Study 5 that a spatially constricted view of the scenes was responsible for the abnormal scanning patterns in healthy subjects, we must consider the possibility that abnormal scanning may be due to the artificial method of controlling the window through eye movements. Specifically, the movement of the gaze-contingent aperture must be controlled through eye movements to “empty space”. Because any area of the display that is outside the viewing aperture is blank, eye movements cannot be initiated based on information in the periphery and this 119  could have led to the abnormal scanning patterns of the gaze-contingent group. We tested this possibility in Study 6, by replicating Study 5 with a mouse-contingent rather than a gaze-contingent window. Using a mouse to control the movement of the window allows subjects to move the window to a location prior to initiating an eye movement. This would therefore minimize any anomalies in fixation patterns that might be due to the unusual situation of initiating eye movements to empty space, while still confining them to processing information within a small window at any given point in time.  Method Participants Subjects (n = 17, 7 male) were undergraduate students at the University of British Columbia who ranged in age from 17 to 23 years (mean = 19 years). All participants reported normal or corrected-to-normal vision and gave informed consent prior to participation in the experiments, which were performed in accordance with the ethical guidelines of the University of British Columbia. Stimuli, apparatus, and procedure The stimuli and procedure were identical to those used in Study 5, but subjects now controlled the aperture with a computer mouse. A 2°x2° (square) aperture was generated by the computer, and revealed the portion of the stimulus image at the location of the mouse. The aperture was initially placed at the central fixation point at the beginning of each trial, and subjects could move the aperture by moving the mouse. We monitored where subjects looked on the screen while moving this window around. Analysis Results were analyzed as in Study 5. 1-way repeated measures ANOVAs with between-subjects factor of Group compared the Mouse-Contingent, GazeContingent and Control groups in terms of proportions of fixations for each region (i.e. separate ANOVAs for each region), and in terms of mean saccade amplitude,  120  average number of fixations per trial, and the average time spent describing each scene. Bonferonni Multiple Comparison t-tests were used to follow up any main effects.  Results Summary of main findings: The Mouse-Contingent group did not differ from the Gaze-Contingent group in any way, showing similar proportions of fixations on each of the five regions. These results are reported below. Mouse-Contingent vs. Gaze-Contingent vs. Controls: The ANOVAs revealed that while the Mouse-Contingent group did not differ from the Gaze-Contingent group in any way, these groups both differed from the Control group. There was a main effect of Group for the proportion of fixations on the Eye region, F(2,35) = 21.70, p < 0.001, indicating that the Control group had a higher proportion of fixations on the Eye region than the other two groups. The Mouse-Contingent and Gaze-Contingent groups both had a higher proportion of fixations on Body, F(2,35) = 15.87, p < 0.001, Object, F(2,35) = 13.91, p < 0.001, and Background, F(2,35) = 12.40, p < 0.001, regions compared to the Control group. There were no differences between any groups in the proportion of fixations on the Head region, F(2,35) = 2.87, p = 0.070. There was a main effect of Group for the mean saccade amplitude, F(35) = 252.33, p < 0.001, number of fixations per trial, F(2,35) = 12.75, p < 0.001, and the average duration of each trial, F(2,35) = 9.87, p < 0.001. Bonferonni t-tests revealed that all groups differed from each other in terms of saccade amplitudes, with Controls making the largest saccades, and the Mouse-Contingent group making the smallest saccades. In contrast, the Mouse-Contingent and Gaze-Contingent groups differed from Controls, but not from each other, in terms of the number of fixations per trial and average trial duration. Specifically, these groups made more fixations, and took longer to describe the scenes than the Controls.  121  Discussion Our results show that the mouse-contingent and gaze-contingent groups did not differ from each other in terms of the distribution of their fixations among the regions, average number of fixations per trial, or average time spent describing each scene, thought the mouse-contingent group did make smaller saccades. This suggests that the low proportion of fixations on the eyes in Study 5 is not due to the generation of saccades to empty space, but due to limiting visual processing to a small portion of the scene at any moment in time. Supplementary temporal analysis We conducted a supplementary analysis to determine whether the scanning patterns observed for each group were stable over time. We compared the cumulative proportion of fixations to the Eye region for each patient and model group to that of the control group at fixed intervals of 5, 15, 30, 45, and 60 seconds. This allows for a temporal assessment of when the differences between these groups and controls emerge. At each time interval, we conducted 2-tailed t-tests to compare each group to the control group. P values were compared to a corrected α’ = 0.01 to account for multiple comparisons. These analyses revealed that the Control group fixated the eyes significantly more than the SL, KC, and the Gaze- and Mouse-Contingent groups as early as 5 seconds, and that this effect persisted for each remaining interval (all ps < 0.001). ES did not differ from Controls at any time interval (all ps > 0.01). Our temporal analysis of all groups therefore indicates that the abnormally low proportions of fixations on the eyes by the simultanagnosic patients and model groups starts early and persists throughout the scene viewing. Again, patient ES, who did not have simultanagnosia, did not differ from controls on this measure at any time. These results are presented in Figure 5.5.  122  Figure 5.5. Cumulative fixation proportions to the Eye region for Controls, ES, SL, KC, and the Gaze- and Mouse-Contingent groups over time. Error bars represent standard error.  General discussion In Study 4 we showed that, unlike healthy control subjects from this and other studies (Birmingham, et al., 2008a, 2008b; Smilek, et al., 2006), simultanagnosic patients SL and KC made very few fixations on the eyes in social scenes. The control patient ES, who has similar bilateral posterior lesions to SL and KC, but never had simultanagnosia, did not differ from healthy control subjects in any way other than making smaller saccades. This suggests that it is simultanagnosia, rather than non-specific effects of brain damage, that generates the reduced fixations on the eyes. In Study 5 we showed that healthy subjects who view scenes through a gaze-contingent aperture produce similar scanning behaviour to SL and KC both in terms of where they look (i.e. reduced fixations on the eyes) and how they move their eyes (i.e. abnormally small saccade amplitudes). Study 6 replicated this result with a mouse-contingent aperture, indicating that this effect is due to spatial restriction of visual processing, rather than being specific to ocular motor control, or to the general effects of parietal damage. Our temporal analysis of the fixation  123  proportions on the eyes region indicates that while healthy control subjects and control patient ES look at the eyes of people in the social scenes early and consistently, patients SL and KC, and the gaze- and mouse-contingent groups do not. When taken together, these findings provide evidence that a model of spatial constriction of visual processing can mimic not only the perceptual performance of simultanagnosic patients with hierarchical letters (e.g., Dalrymple et al, 2010), but also their scanning behaviour while viewing complex natural scenes. Several aspects of our study point specifically to a restricted area of visuospatial processing as being key to these results. First, both simultanagnosic patients showed reduced fixations on the eye region, but our control patient, ES, who did not have simultanagnosia and therefore a presumably normal visuospatial processing area, showed normal distributions of fixations. Second, this behaviour was mimicked by a literal restriction to the viewing window of healthy subjects in our gaze-contingent paradigm. Finally, our mouse-contingent group, who viewed the same scenes through the same window as our gaze-contingent group, showed the same reduction in fixations on the eye region, even though they moved the window with a mouse rather than with their eyes. This indicates that it was the window itself that was crucial to creating this abnormal behaviour, not the method of controlling the window. One might wonder if SL and KC’s low fixation count on the eye region can be explained by poor ocular motor control, given the ocular motor apraxia that often forms part of Bálint syndrome. Inaccurate targeting, which may conceivably also occur in healthy subjects attempting to control an unfamiliar gaze-contingent window, could result in fixations intended for the eyes falling outside of that region. However, if this were the case, one would then expect an abnormally high proportion of fixations on the adjacent head region. Instead, our results show that all groups, including healthy participants, fixated the head equally. Furthermore, subjects in the mouse-contingent window condition were free to move their fixation position within the mouse-controlled aperture, thus eliminating the concern of inaccurate saccadic targeting. As we have demonstrated, the mouse-contingent group produced identical  124  scanning behaviour to the gaze-contingent group. Another aspect of the patient and gaze- and mouse- contingent group scanning behaviour that could underlie the reduced fixations on the eye region of the scenes is the abnormally small saccades. Interestingly, our control patient ES also made short saccades relative to controls, but did not show reduced fixations on the eye region. This indicates that small saccades are not related to the selection of the various regions by the patients and contingent groups. Specifically, short saccades do not cause reduced fixations to the eye regions. However, the fact that the gazeand mouse-contingent groups show abnormally short saccades like patients SL and KC, further validates this paradigm as a useful way of modeling simultanagnosic scanning patterns in healthy individuals. An alternate possibility for the reduced fixations on the eyes, and the one that we favour, is that the eye region is not especially informative to the patients and the gaze- and mouse-contingent groups. Birmingham et al. (2007) argue that people look at eyes in social scenes because they provide rich social information regarding the meaning of a scene, especially regarding how people in the scene are allocating their attention to other people and objects within a scene. This interpretation dovetails well with the present findings. Patients with simultanagnosia, and healthy participants using a gaze- or mouse-contingent window, do not have access to visual information beyond a confined perceptual window and are therefore unlikely to be able to infer where people in the scenes are attending. Similarly, items within a scene are unlikely to be perceived as “being looked at” or “not being looked at” by one or more individuals in the scene. Indeed, the present data suggest that this became less relevant for the observers. Of course, what did not become irrelevant to any of the participants is a general interest in people. The patients, as well as the gaze- and mouse-contingent groups, continued to allocate a significant amount of their viewing to the heads and bodies of the people in the scenes. What is unique and striking is that they appear to be uninterested in where the people in the scene are allocating their visual attention.  125  This interpretation converges with the well-established finding that simultanagnosic patients suffer from a global processing deficit. Patients describe scenes in a piece-meal fashion and are unable to see the global aspect of hierarchical letters. While it was originally thought that this global processing deficit was due to “local capture”, that is, an inability to disengage from local elements in order to perceive the global whole (Karnath, et al., 2000), eye movement data from patients performing this task have since disconfirmed this notion (Clavagnier, et al., 2006). Rather than being “stuck” on a few local elements, simultanagnosics produce many eye movements – far more than what is seen from healthy control subjects performing the same task (see Figure 5.4). While control subjects need only make a few fixations to extract the global meaning of a stimulus, simultanagnosic patients scan the stimulus in detail and may even trace the global shape in an attempt to piece together its global meaning (Clavagnier, et al., 2006; Dalrymple, et al., 2009). This behaviour is well accounted for by the Reverse Hierarchy Theory (Hochstein & Ahissar, 2002), which suggests that normal vision occurs on a continuum from “vision at a glance”, which provides a global gist of a visual scene to “vision with scrutiny”, which processes the details of the scene. The restricted spatial area of visual processing in simultanagnosia would in theory disrupt “vision at a glance”, forcing patients to rely on vision for scrutiny to derive meaning from a scene. This theory would therefore predict the pattern of exploration seen with hierarchical letters, as well as the pattern of exploration seen in the present study: rather than allocating a large proportion of fixations to one region, such as the eyes, and deriving the meaning of a scene through “gist”, simultanagnosic patients, and our gaze- and mouse-contingent groups, distribute their fixations more evenly across regions, using details to derive global meaning. Increased reliance on “vision with scrutiny” would also predict the abnormally small saccade amplitudes that were produced by these groups, as they move serially from detail to detail within the scene. Providing additional support for this possible reliance on vision for scrutiny, Nyffeler et al. (2005) recorded the eye movements of a simultanagnosic patient  126  while she read the time on a schematic analogue clock and found that the patient looked at the numbers of the clock in succession, rather than looking at the hands of the clock and the numbers they point to. Without ‘vision at a glance’ to give the gist of the object and to guide ‘vision for scrutiny’ to the important details of the clock (i.e. the hands), the patient was forced to make successive fixations to “uninformative” parts of the clock before locating the hands to tell the time. This result is also somewhat analogous to the current findings: while Nyffeler’s patient did not look at the hands of the clock and the numbers they were pointing to, our patients did not look at the eyes in the scenes and what they were looking at. Our findings of abnormally low fixations on eyes suggest potential deleterious effects of simultanagnosia on how facial information is processed. Previously we showed that patients with simultanagnosia may experience global capture with faces, seeing the face as a whole at the expense of the features (Dalrymple, et al., 2007). This suggests abnormal processing of details, including the eyes, and is consistent with results of the present study, which show that simultanagnosics make normal proportions of fixations to the heads of people in scenes, yet reduced fixations to the eyes. There are several implications to this finding, including the possibility that patients are impaired at normal social responses, such as gaze cuing. This possibility is of great interest as an avenue of future research in order to more fully understand the extent of the simultanagnosic deficits. Despite the strong relationship between the scanning patterns of our patients and model groups, one might question the validity of using an artificially restricted visual window as a model of simultanagnosic behaviour. However, the fact that a restricted viewing paradigm led to simultanagnosia-like scanning patterns of social scenes in healthy subjects is likely more than a coincidental convergence of patient and model behaviours. For one, the restricted window paradigm implemented here is a theoretically motivated model of the simultanagnosic deficit, based on descriptions of patient behaviours, empirical tests of what patients can and cannot see (i.e. demonstrating a restricted area of useful visual processing), and based on the reports of patients regarding their own experience (e.g. patient KC described his  127  world as being very narrow and expanding in size as he recovered from his simultanagnosia). Secondly, the high proportion of fixations to the eyes in scenes by healthy subjects under normal viewing conditions is a highly replicable finding that occurs across a variety of tasks (e.g. describing or remembering a scene: Birmingham, et al., 2007, 2008a, 2008b; Smilek, et al., 2006); there is no a priori reason to predict that simultanagnosic patients would show reduced fixations to the eyes of people in social scenes. Likewise, there is no a priori reason to predict that participants in a restricted viewing condition would show the same abnormal behaviour, beyond the link we hypothesized to exist between the reduced window of attention in simultanagnosia and the restricted window of vision in our paradigm. Finally, this same reduced window manipulation has led to simultanagnosic behaviour with other simpler stimuli (Dalrymple, et al., 2010). Specifically, healthy subjects asked to name the global level of hierarchical letters viewed through a gaze-contingent aperture showed accuracy patterns for global level report similar to patient SL, who performed the same task under natural viewing conditions. For the above reasons it appears reasonable to conclude that the restricted viewing window in our task is a valid model of the simultagnosia-like scanning patterns. It also encourages the speculation that visual restriction may in fact be a key underlying mechanisms that leads to the abnormal scanning of social scenes in simultanagnosia. Although artificially restricted window-viewing leads to complex behaviours similar to those of patients with a restricted window of attention due to brain damage, we do not claim that this model explains all simultanagnosic behaviours, or all the properties of the attentional window itself. For example, there is evidence that patients can be cued to locations outside their useful visual window (Egly, Robertson, Rafal, & Grabowecky, 1995), and others have shown that the restricted window of attention in simultanagnosia can be expanded through priming (Shalev, Humphreys, & Mevorach, 2004). It is unclear how healthy individuals could be cued to a location outside the rigid viewing window used in our study, or how our paradigm could be used to model the expansive properties of the attentional window in simultanagnosia. However, the present data show that the artificial window is a 128  good first step in investigating the link between restricted visual input due to damage to the visual system and an artificial restriction of visual input on individuals with normal brains. In fact, one strength of the current model is its parsimony: a very simple visual manipulation can lead to a complex behavioural pattern akin to one from a complex neurological disorder. Adjusting the current methodology to model other aspects of the restricted window of attention in Bálint syndrome is a challenge for future research that can only further inform the nature of this disorder. In addition to identifying a valid model of simultanagnosic behaviour, providing a useful tool for future research on a disorder for which patients are often scarce, the results of the present experiments have two other important implications. First, our results reveal how a spatially constricted window of visual processing affects the acquisition of information from the world. The fact that healthy subjects under natural viewing conditions robustly show disproportionately high fixation rates to the eyes in social scenes across several tasks (Birmingham, et al., 2007, 2008b; Smilek, et al., 2006) underscores the unusual nature of SL and KCʼs scanning patterns. Secondly, our results support the idea that people fixate the eyes in social scenes because they are informative to the overall meaning of the scene, in large part because they tell observers where people are directing their attention. When eyes are viewed without the surrounding visual context, they may lose this informative value. Thus, a constricted spatial area of processing, whether from neurological or artificially imposed limitations, has important consequences for how information is acquired from our visual world.  129  References Balint, R. (1909). Seelenlähmung des "Schauens", optische Ataxia, räumliche Störung der Aufmerksamkeit. Monatsschrift für Psychiatrie und Neurologie, 25, 51-81. Birmingham, E., Bischof, W., & Kingstone, A. (2007). Why do we look at people's eyes? Journal of Eye Movement Research, 1(1), 1-6. Birmingham, E., Bischof, W., & Kingstone, A. (2008a). Gaze selection in complex social scenes. Visual Cognition, 16(2/3), 341-355. Birmingham, E., Bischof, W., & Kingstone, A. (2008b). Social attention and real world scenes: The roles of action, competition and social content. Quarterly Journal of Experimental Psychology, 61(7), 986-998. Clavagnier, S., Fruhmann Berger, M., Klockgether, T., Moskau, S., & Karnath, H.-O. (2006). Restricted ocular exploration does not seem to explain simultanagnosia. Neuropsychologia, 44, 2330-2336. Dalrymple, K., Bischof, W., Cameron, D., Barton, J., & Kingstone, A. (2009). Global perception in simultanagnosia is not as simple as a game of connect-the-dots. Vision Research, 49, 1901-1908. Dalrymple, K., Bischof, W., Cameron, D., Barton, J., & Kingstone, A. (2010). Simulating simultanagnosia: spatially constricted vision mimics local capture and the global processing deficit. Experimental Brain Research, 202, 445455. Dalrymple, K., Kingstone, A., & Barton, J. (2007). Seeing trees OR seeing forests in simultanagnosia: Attentional capture can be local or global Neuropsychologia, 45, 871-875. Egly, R., Robertson, L., Rafal, R., & Grabowecky, M. (1995, November). Implicit processing of unreportable objects in Balint's syndrome. Paper presented at the Psychonomic Society, Los Angeles. Goodglass, H., & Kaplan, E. (1983). The assessment of aphasia and related disorders (2nd ed.). Philadephia: Lea and Febiger. Hochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804. Holmes, G., & Horrax, G. (1919). Disturbances of spatial orientation and visual attention, with loss of stereoscopic vision. Archives of Neurology and Psychiatry, 1, 385-407. Huberle, E., & Karnath, H.-O. (2006). Global shape recognition is modulated by the spatial distance of local elements - Evidence from simultanagnosia. Neuropsychologia, 44(6), 905-911. 130  Karnath, H.-O., Ferber, S., Rorden, C., & Driver, J. (2000). The fate of global information in dorsal simultanagnosia. Neurocase, 6, 295-306. Kim, M., & Robertson, L. (2001). Implicit representations of space after bilateral parietal lobe damage. Journal of Cognitive Neuroscience, 13(8), 1080-1087. Leibovitch, F., Black, S., Caldwell, C., Ebert, P., Ehrlich, L., & Szalai, J. (1998). Brain-behaviour correlations in hemispatial neglect using CT and SPECT: The Sunnybrook Strong Study. Neurology, 50, 901-908. Loschky, L., McConkie, G., Yang, J., & Miller, M. (2005). The limits of visual resolution in natural scene viewing. Visual Cognition, 12(6), 1057-1092. Malcolm, G., & Barton, J. J. S. (2007). "Sequence Agnosia" in Balint's Syndrome: Defects in Visuotemporal processing after bilateral parietal damage. Journal of Cognitive Neuroscience, 19(1), 102-108. McConkie, G., & Rayner, K. (1975). The span of the effective stimulus during a fixation in reading. Perception and Psychophysics, 17, 578-586. Moreaud, O. (2003). Balint Syndrome. Archives of Neurology, 60, 1329-1331. Nyffeler, T., Pflugshaupt, T., Hofer, H., Bass, U., Gutbrod, K., von Wartburg, R., et al. (2005). Oculomotor behaviour in simultanagnosia: A longitudinal case study. Neuropsychologia, 43, 1591-1597. Pomplun, M., Reingold, E., & Shen, J. (2001). Investigating the visual span in comparative search: the effects of task difficulty and divided attention. Cognition, 81, B57-B67. Rafal, R. (2003). Balint's syndrome: A disorder of visual cognition. In M. D'Esposito (Ed.), Neurological foundations of cognitive neuroscience (pp. 27-40). Cambridge, MA: MIT Press. Rizzo, M., & Vecera, S. (2002). Psychoanatomical substrates of Balint's syndrome. Journal of Neurology, Neurosurgery and Psychiatry, 72, 162-178. Shalev, L., Humphreys, G., & Mevorach, C. (2004). Global processing of compound letters in a patient with Balint's syndrome. Cognitive Neuropsychology, 22(6), 737-751. Smilek, D., Birmingham, E., Cameron, D., Bischof, W., & Kingstone, A. (2006). Cognitive Ethology and explory attention in real-world scenes. Brain Research, 1080(101-119). Tyler, R. (1968). Abnormalities of perception with defective eye movements (Balint's Syndrome). Cortex, 4(2), 154-171.  131  Chapter 61 Opening a window on attention: Documenting and simulating recovery from simultanagnosia  1  A version of this chapter has been accepted for publication. Dalrymple, KA, Birmingham, E, Bischof, WF, Barton, JJS, & Kingstone A. (2010). Opening a window on attention: Documenting and simulating recovery from simultanagnosia. Cortex.  132  Simultanagnosia is an inability to see more than one object at a time resulting from bilateral lesions to the parieto-occipital junction (Bálint, 1909; Holmes & Horrax, 1919; Riddoch, Chechlacz, Mevorach, Marvritsaki, Allen, & Humphreys, 2010; Rizzo & Vecera, 2002). It can be so severe that patients appear “functionally blind” (Kim and Robertson, 2001), with little or no understanding of the fragmented world they perceive. Although appearing functionally blind, simultanagnosia has been identified as a disorder of visual attention, rather than blindness (Holmes and Horrax, 1919): “The essential feature was his inability to direct his attention to, and take cognizance of, two or more objects that threw their images on the seeing portion of his retinae. As this occurred no matter on what parts of his retinae the images fell, it must be attributed to a special disturbance or limitation of attention…” (Holmes and Horrax, 1919, p.390). Since
then,
a
notion of simultanagnosia as resulting from a restricted window of attention has become increasingly apparent in the literature. Bay (1953) suggested that his patient’s simultanagnosia could be accounted for by “shaft vision”, which prevented the patient from seeing the whole picture. He described it as a “peripheral contraction”, not unlike “viewing [a] picture through a diaphragm” (p. 545, 546). Similarly, Thaiss and de Bleser (1992) suggested that their patient, TK, may suffer from a rigid reduction of the spatial extent of the visual spotlight. Previously, we conducted an experiment to test this idea of a restricted window of attention by “simulating” simultanagnosia by applying a literal restriction of vision to healthy participants while they identified hierarchical letter stimuli (Dalrymple, et al., 2010). When viewing hierarchical letters (large global letters made up of several repetitions of smaller local letters, e.g., a large letter K made up of small letter D’s), patients with simultanagnosia tend to report the local letters, and are unable to identify the larger, global letters (Clavagnier, et al., 2006; Dalrymple, et al., 2007; Huberle and Karnath, 2006; Karnath, et al., 2000; Shalev, et al., 2004). Interestingly, healthy participants viewing hierarchical letters through a gazecontingent window (a small window of vision that allows them to see only a small portion of the stimulus at once) showed similar accuracy patterns for identifying hierarchical letters to what is typically seen with patients with simultanagnosia. That is, they showed good accuracy for identifying the small, local letters, but were 133  impaired at identifying the large, global letters. We interpreted this as evidence that a narrowed window of vision leads to perceptual phenomena that are similar to those seen in simultanagnosia, suggesting that a literal restriction of vision may be a good model of the restriction of attention in simultanagnosia. In support of this conclusion, we found similar effects with our restricted window paradigm when we applied it to more complex stimuli (Dalrymple, et al., under review). We examined how simultanagnosic patients scan social scenes, with an interest in determining how their scanning behaviour relates to the concept of a restricted window of attention. We tested this by applying the restricted window paradigm to healthy subjects while they scanned social scenes. It is well established that healthy participants scanning social scenes under unrestricted viewing conditions tend to allocate a large proportion of fixations to the eyes of the people in the scenes (Birmingham, et al., 2007, 2008a, b; Smilek, et al., 2006). This is a highly replicable, robust finding, which seems to occur in a variety of tasks (e.g., describe the scene, look at the scene, remember the scene, etc.). In our experiment, we found that patients with simultanagnosia show reduced fixations to the eye region of the social scenes compared to healthy controls. Remarkably, healthy participants viewing the scenes through the restricted window also showed reduced fixations to the eyes of the people in the scenes. This behaviour occurred regardless of how participants controlled the movement of the window: whether using their eyes (gazecontingent window) or using a computer mouse (mouse-contingent window). This further solidified the idea that it is the restricted viewing window itself that is key to this behaviour. Again we concluded that a literal restriction of vision may be a good model of the restriction of attention in Bálint syndrome. Although our rigid window of vision successfully simulated both simple and more complex Bálint behaviours in these experiments, a careful inspection of the literature suggests that the restricted window of attention in simultanagnosia may not be rigid, but may instead be flexible in size. Like Thaiss and de Bleser’s (1992) description of patient TK, Tyler (1968) referred to the visual deficit in his patient as “shaft vision” (p. 166), yet Tyler’s description clearly implies some flexibility. When  134  he measured his patient’s effective visual fields he concluded that they were quite variable: consistent perception occurred within 2° of fixation, and could occur at up to 20°, though this larger window of effective vision quickly fatigued (within 10-30s). Tyler’s description, therefore, suggests a type of shaft vision, with a flexible spatial extent that may widen or shrink. It appears as though the size of the window is outside of the patient’s control: the window is restricted at rest and expanding the window is a demanding process that is easily fatigued. Shalev, et al. (2004) found direct evidence for a flexible window of attention in simultanagnosia. Specifically, through testing with hierarchical letters they found that first presenting patients with a large solid letter that takes up the same spatial extent as the global aspect of a hierarchical letter improved the patient’s ability to name the global level of that hierarchical letter. These authors interpreted this finding to indicate that the “default” restricted window of attention was temporarily widened by the prime, allowing for explicit processing of the global hierarchical letter. The idea of an expanding attentional window leads to some interesting questions about patient recovery. Little is known about how perception in simultanagnosia changes with recovery. Nyffeler et al (2005) documented the scanning behaviour of a simultanagnosic patient at 8, 14, and 37 weeks after injury, while she explored simple line drawing (e.g. objects, schematic clock). This patient showed improved performance over time (e.g. better object naming, increased exploration of the stimuli), which Nyffeler et al concluded reflected an enlargement of a restricted attentional field. Is it possible that the “default” size of the restricted attentional window increases over the course of patient recovery? Finding the answer to this question was the primary aim of this study. This expansion of a patient’s window of attention would be reflected in a recovery of normal behaviours. Therefore, applied to the scanning of social scenes, we hypothesized that a patient who initially showed reduced proportions of fixations on the eyes of people in social scenes may show increased fixations on the eyes in the scenes after some recovery. This was tested in Experiment 1 of this study.  135  Return of more normal behaviours is a natural and expected consequence of recovery and may not necessarily reflect and expansion of the attentional window over time. However, if our hypothesis is supported in Experiment 1 and patients do allocate more fixations to the eyes in scenes after some recovery, we determine whether this behaviour is related to a change in the size of the window of attention by changing the size of the restricted window in our paradigm with healthy participants. If changes in patient scanning behaviour is related to an expansion of the window of attention over time, a similar change in behaviour should occur with healthy participants viewing scenes through windows of different sizes (small vs. large). We tested this window expansion hypothesis in Experiment 2 by asking two groups of healthy participants to describe the social scenes while exploring them with a mouse-contingent aperture2 of two different sizes. If the recovery of normal behaviour in simultanagnosia is related to an expansion of a narrowed window of attention, this should be reflected in similar changes in behaviour through manipulations of the size of the mouse-contingent display. Similarly, this would provide further evidence of a flexible, rather than rigid, window of attention in simultanagnosia. Importantly, this would further link the simultanagnosic deficits to an underlying mechanism (a restricted window of attention), and provide direct evidence for how that mechanism evolves over the course of patient recovery. It is important to emphasize that the visual deficits in simultanagnosia are known to be attentional in nature (Holmes and Horrax, 1919). These patients have fully functioning retinas (Holmes and Horrax, 1919), and can see objects of different sizes (Rafal, 2001). What is not known is how that disorder of attention manifests itself. The current experiments apply the concepts of the disordered visual attention in simultanagnosia that are arising in the literature. Concepts such as “shaft vision” (Bay, 1953; Tyler, 1968) and “viewing [a] picture through a diaphragm” (Bay, 1953, p. 545, 546) imply a spatial context to the restriction of attention. Our aim is to test 2  We used the mouse-contingent manipulation because in Dalrymple et al. (under review) participants complained of discomfort (e.g., nausea) when using the gazecontingent procedure, and because the results of the gaze- and mouse- contingent methods were virtually identical. 136  the validity of these concepts by testing whether they adequately capture the true nature of the attentional deficit. The chosen method for testing this is through modeling the attentional disorder in healthy individuals. There are obvious difficulties in experimentally creating a restricted attentional window in healthy individuals. As an alternative, and inspired by the literature, a simulation of the attentional disorder in simultanagnosia was created through a literal restriction of vision in healthy individuals. This literal restriction of vision aims to create a representation of the visual experience of the patients, but also limits what can be attended by the individuals. That is, by limiting the spatial extent of what can be seen, this model limits what can be attended to, i.e., attention is restricted to what is visible in the window. Therefore, in both patients and healthy individuals, we have a scenario of reduced visual information: in patients, this results from a neurological restriction of the attentional window; in healthy subjects this is created through an artificial restriction of the visual window. To reflect this important distinction, specific terminology will be used throughout this paper: when discussing patients, the limitation will be described as a restricted or limited window of visual attention; when discussing healthy individuals in the model groups, the limitation will be described as a restricted or limited window of vision.  Method Case report: Patient SL Time 1: June 2005 Patient SL is a right-handed woman, with 12 years of education. She had idiopathic cerebral vasculitis resulting in bilateral parietal and lateral occipital infarcts (Figure 6.1). She had been treated with cyclophosphamide and prednisone for her vasculitis, but had completed these 4 months prior to her testing. She was on carbamazepine for a single seizure suffered several months prior. She presented with left hemi-neglect, as assessed with the Sunnybrook Neglect Assessment Battery (Leibovitch, et al., 1998), left inferior quadrantanopia, and Bálint syndrome,  137  with ocular motor apraxia, optic ataxia, and simultanagnosia, though her acuity was 20/25 in both eyes. Her optic ataxia was evident in that she often mis-reached for objects, and failed to orient her grasp correctly to the axes of objects such as pencils. This was evident despite normal motor and sensory function on her neurological examination, which also showed accurate reaching to her own body parts. Her simultanagnosia was evidenced through tests with four complex displays of visual scenes. For example, she could report elements of the Boston Cookie Theft picture (Goodglass and Kaplan, 1983), but was unable to make sense of the whole scene. Neuropsychological evaluation showed normal attention, language, and verbal memory functions. Her reading was in the borderline impaired range and she tended to guess words based on the first or last letters. She was successful at recognizing simple line drawings of objects and could correctly identify colours and simple shapes. After her discharge home, she continued to note difficulties. Her reading was slow but she could read menus and signs. She had trouble with photographs in books or newspapers, in that she often missed elements in them. She bumped into objects on either side when walking, and because of her navigational problems traveled in a wheelchair for some months. She had some minor difficulties using kitchen utensils with her left hand mainly, but this improved quickly.  Figure 6.1. Axial FLAIR sequences of MRI scans for patient SL. R = Right, L = Left side of the brain. Arrowheads indicate the junction of Brodmann areas 19 and 37, while arrows indicate the junction between areas 39 and 40.  138  Magnetic resonance imaging showed bilateral lesions in the lateral occipital cortex, at the junction of Brodmann areas 19 and 37, extending more dorsally through inferior and superior parietal cortex laterally, in Brodmann areas 39 and 40, with some minimal involvement of medial area 31 superiorly (Figure 6.1). These lesions are typical of those seen in other patients with dorsal simultanagnosia (Riddoch et al. 2010). Although a ventral form of simultanagnosia purportedly can result from lesions of the left occipital cortex, such patients tend to present mainly with alexia and right hemianopia, without the navigational problems and optic ataxia seen in SL (Kinsbourne & Warrington, 1962, 1963). Also, modern neuroimaging implicates the left fusiform gyrus in patients with letter-by-letter reading and alexia (Leff, Spitsyna, Plant, & Wise, 2006), which was spared in SL. Thus, on the basis of neuroimaging and behavioural evidence, it is probable that SL had simultaganosia as a consequence of bilateral ‘dorsal’ lesions of parietal and lateral occipital cortex. At the time of initial testing, SL was 48 years old and no longer showed left hemi-neglect or quadrantanopia and had no defects in saccadic targeting and generation, as was confirmed by her rapid and accurate saccades during the calibration of the eye monitor. However she still showed optic ataxia when using the left hand to point to targets. This was a specific sensorimotor transformation for the contralateral hand, and therefore not due to a general difficulty with perceptual localization (which would affect both hands). She also showed evidence of topographical difficulties, being unable to navigate through her environment without substantial assistance. Time 2: November 2008 SL was 52 years old at the time of testing. She had no new neurologic events, or seizures since her original trauma. Her acuity correction at far was 20/40-2 ou, 20/30-1 ou with pinhole. Her fixation and smooth pursuit showed distractibility, yet her saccades were quick and accurate, without delay in initiation to command and she showed no nystagmus. She was alert and attentive with normal language and speech. She named more elements on the Boston Cookie Theft picture than at Time 1 and described each of these elements fully. She could read short sentences well, 139  with only slight hesitation with longer words or when finding the next line. She made randomly distributed errors of omission and commission on an object cancellation task. She still met the diagnostic criteria for Bálint syndrome, and had a left inferior quadrantic scotoma, secondary to her cerebral vasculitis. SL had no new neurological incidents between the time of this scan and the time of testing. Stimuli and Apparatus: Scenes were chosen from those used in prior studies of visual attention in healthy subjects (Birmingham et al., 2007, 2008a, b). These full color images were originally taken with a digital camera in different rooms in the Psychology building at the University of British Columbia. Image size was 36.5 x 27.5 (cm) corresponding to 40.1° x 30.8° at the viewing distance of 50 cm, and image resolution was 800 x 600 pixels. Eight scenes were used in the present experiment. Each scene contained 3 persons in either “interactive” or “non-interactive” state. All scenes were comparable in terms of their basic layout: each room had a table, chairs, objects, and background items (e.g. see Figure 6.2a).  140  a.  b.  Figure 6.2. Example of scene stimuli (a) and regions of interest (b). “Interactive” scenes appear on the left, “Non-interactive” scenes on the right.  Eye movements were monitored using the EyeLink II eye tracking system for SL at Time 1 and the mouse-contingent group who viewed the scenes through the large aperture. The EyeLink 1000 eye tracking system was used for SL at Time 2 and the mouse-contingent group who viewed the scenes through a small aperture (SR Research Ltd., www.eyelinkinfo.com)3. The Eyelink II has a temporal resolution  3  The Eyelink 1000 was used for SL at Time 2 and the small window mousecontingent group because of equipment upgrades that took place in the 3.5 years after initial testing. Both Eyelink systems have the same 0.5° Gaze Accuracy, and the difference between the Eyelink II and Eyelink 1000 sampling rates does not affect the analysis of fixation frequencies, therefore the use of two different systems should not impact our results. 141  of 4 ms (sampling rate 250 Hz) and a spatial resolution of 0.5°. The EyeLink system records sample data indicating the location of gaze, in pixel coordinates. Before any analysis was carried out, these samples were parsed into fixation and saccade events (and blinks) using the EyeLink software. The event parser identifies epochs in the data file where a saccade is occurring by calculating the distance between gaze position in different samples and implementing motion, velocity and acceleration thresholds. The on-line saccade detector of the eye tracker was set to detect saccades with an amplitude of at least 0.5°, using an acceleration threshold of 9500°/s2 and a velocity threshold of 30°/s. A fixation is defined as any event that was not a saccade or a blink. One high-speed camera tracked the left eye, while a second camera tracked and compensated for head position by monitoring 4 infrared sensors placed on the corners of the display monitor. Cameras were mounted and held in place by a lightweight headband, which was placed and secured on the subjects. The Eyelink 1000 differs in that it has a temporal resolution of 1ms (sampling rate 1000Hz) and mounts the cameras on the desktop, rather than on a headband. The on-line saccade detector was set to detect saccades with an amplitude of at least 0.15°, using an acceleration threshold of 8000°/s2 and a velocity threshold of 30°/s. Two computers were used in the experimental setup and were connected to each other via Ethernet, allowing for real-time transfer of saccade and gaze position data. The Experimenter computer collected the data from the eye tracker and displayed an image of the participant’s eye and calibration information. The Display computer displayed the stimuli and recorded key-presses. Procedure: Subjects were seated 50 cm from the screen of the display computer with their chin supported by a chin rest. Eye movements were recorded monocularly from the left eye. The eye monitor was calibrated using a 9-dot array. Calibration was validated using the same procedure. After successful calibration and validation, subjects were asked to fixate a dot at centre-screen in order to correct for drift in gaze position. Once the dot was fixated, the experimenter initiated the onset of the scene image by key-press. 142  Scenes were presented in random order. Subjects were asked to verbally describe each scene while a digital voice recorder recorded their descriptions. SL had unlimited time to describe the scene, and indicated that her description was complete by informing the experimenter that she was ready for the next picture. At this point, the experimenter initiated the next trial by keypress. Healthy subjects viewed the scenes through a square mouse-contingent window of one of two sizes (“Small” 1°x1° or “Large” 2°x2°). They terminated their trials by keypress, initiating the next trial. All subjects viewed 8 social scenes.  Experiment 1: SL The purpose of this experiment was to determine if and how patient SL’s scanning of social scenes changed over the course of her recovery. We predicted that SL would scan social scenes more like healthy participants as she recovered from her deficits. Specifically, we predicted that she would allocate more fixations to the eyes of the people in the social scenes at Time 2 compared to Time 1. SL’s Time 1 data was taken from our previous report, Dalrymple et al, (under review), while her Time 2 data was collected for the present study.  Analysis: For each image, an outline was drawn around each region of interest (e.g. "eyes") and each region’s pixel coordinates and area were recorded. We defined the following regions: Eye, Head (excluding eyes), Body (including arms, torso and legs), foreground Objects (e.g., tables, chairs, objects on the table) and Background (e.g., walls, shelves, items on the walls). Figure 6.2b illustrates these regions for two scenes. To compensate for the different sizes of these regions, we computed area-normalized fixation proportions (Birmingham, et al., 2008a; Smilek, et al., 2006), by first dividing the number of fixations in each region by the area of the region, separately for each image and each participant, and then computing proportions based on these normalized data.  143  To determine where SL fixated, we analyzed the fixation proportion data (Figure 6.3a) with a series of paired two-tailed t-tests comparing the different regions of interest (Eye, Head, Body, Objects, and Background) to each other for SL at Time 1 and at Time 2, respectively. To compare SL’s fixation proportions at Time 1 to her fixation proportions at Time 2, we performed 2-tailed paired t-tests on the fixation proportions for each region. Time 1 data were paired with Time 2 data based on scene viewed. We also computed SL’s mean fixation durations, number of fixations per trial, and trial duration (Figure 6.4) and compared her data at Time 1 and Time 2 for each of these measures using a paired two-tailed t-test. All p values were compared to α=0.05. Finally, to determine how SL’s fixations are allocated over time, we plotted her cumulative proportions of fixations to the Eye region (our primary region of interest) at 5, 15, 30, 45, and 60 seconds (Figure 6.3b). We chose these intervals because most Control subjects do not exceed trial durations of 60 seconds. We also compared SL’s Time 1 versus Time 2 cumulative proportions of fixations on the Eye region at the earliest time intervals (5 seconds and 10 seconds) using paired twotailed t-tests (p value compared to α=0.05). This was done to determine when during the trial SL’s fixation allocation to the eyes at Time 2 diverged from that at Time 1.  Results Results from Experiment 1 and Experiment 2 are presented in Figures 6.3 and 6.4. For visual comparison, these figures also include data from healthy control subjects from Dalrymple et al. (under review), who viewed scenes in unrestricted viewing conditions. SL Time 1: SL’s highest proportion of fixations was on the Head region (0.34). This proportion was significantly larger than those for Object (0.11), t(7)=-4.64, p=0.002, and Background (0.08), t(7)=6.14,p<0.001, regions. SL also had a greater proportion  144  of fixations on the Body region (0.32) than the Object, t(7)=-5.35, p=0.001, and Background region, t(7)=5.27, p=0.010. No other Regions differed from each other. SL Time 2: SL’s highest proportion of fixations was on the Eye region (0.43). This proportion was significantly larger than those for Body (0.12), t(7)=6.34, p<0.001, Background (0.07), t(7)=7.69, p<0.001, and Object (0.06), t(7)=-10.38,p<0.001, regions. SL also had a greater proportion of fixations on the Head region (0.32) than the Body, t(7)=-10.61,p<0.001 Background, t(7)=-12.19,p<0.001, and Object, t(7)=8.99,p<0.001, regions, though her proportion of fixations to the Head region did not differ significantly from her proportion of fixations to the Eye region. The proportions of fixations to the Body, differed from those of the Background, t(7)=4.53,p=0.003, and Object, t(7)=-3.68,p=0.007, regions, though these later regions did not differ from each other. SL Time 1 vs. Time 2: SL had a significantly higher proportion of fixations on the Eye region at Time 2 compared to Time 1, t(7)=-3.79,p=0.007, and a significantly lower proportion of fixations to the Body region, t(7)=4.37,p=0.003, at Time 2. Her fixation proportions to the other regions remained unchanged: Head, t(7)=0.65,p=0.534; Object, t(7)=2.31,p=0.054, Background, t(7)=0.37,p=0.722. SL’s fixations were significantly shorter, t(7)=3.27,p=0.014, she made more fixations t(7)=-6.29,p<0.001, and took longer to describe the scenes, t(7)=-4.98,p=0.002, at Time 2 compared to Time 1. Interval analysis: Having established that SL fixates the Eye region more at Time 2 than at Time 1, we compared her Time 2 versus Time 1 fixations on the Eye region at the earliest viewing intervals to determine at what point within a trial, on average, her fixation proportions on the eyes increased as a result of recovery. SL’s increase in fixations to the eyes at Time 2 relative to Time 1 was apparent as early as 5 seconds into the trials, as indicated by a trend to significance in the comparison, t(7)=-2.30,p =0.055. By 10 seconds into the trials, SL was making significantly more fixations on the eyes at Time 2 compared Time 1, t(7)=-3.41,p=0.011.  145  a.  b.  Figure 6.3. a) Fixation proportions for patient SL at Time 1 and Time 2, and for participants who viewed scenes through Small and Large mouse-contingent windows. Fixation proportions are normalized for the size of the region. b) Cumulative proportions of fixations on the Eye region over the first 60s of viewing for SL at Time 1, and Time 2, and for participants who viewed scenes through Small and Large mouse-contingent windows. Also included in both a) and b) are control data from Dalrymple et al (under review) for visual comparison. Error bars represent standard error. 146  a.  b.  c.  Figure 6.4. a) Average fixation durations (in ms) b) Mean number of fixations per trial, and c) mean trial duration (in s), for patient SL at Time 1 and Time 2 and for participants who viewed scenes through Small and Large mouse-contingent windows. Also included are control data from Dalrymple et al (under review) for visual comparison. Error bars represent standard error.  147  Summary SL fixated the eyes in the social scenes more at Time 2 than at Time 1. These extra fixations were accompanied by a corresponding decrease in fixations to the bodies in the scenes, while fixations to the other regions remained constant. Another notable effect was that SL’s fixations decreased in duration, but increased in number from Time 1 to Time 2. Her trial durations also increased from Time 1 to Time 2. One might be concerned that SL’s increased fixations to the eyes at Time 2 may have been due to the fact that she explored the images for longer, giving her more of an opportunity to find the eye regions. However, as can be seen in Figure 6.3b, and confirmed by comparison of SL’s cumulative fixations at Time 2 versus Time 1, SL’s increased fixations on the eyes at Time 2 relative to Time 1 was evident as early as 5 seconds into the trial. This strongly suggests that her increased fixations on the eyes at Time 2 is not related to her increased trial duration.  Experiment 2: Mouse-Contingent windows The purpose of Experiment 2 was to test the hypothesis that SL’s increased fixations on the eyes of the social scenes at Time 2 compared to Time 1 is related to an expansion of the size of her window of visual attention. To test this hypothesis, we compared our mouse-contingent group from our previous study (Dalrymple, et al., under review) to a new mouse-contingent group who viewed the scenes with a window of a different, smaller, size than our initial group. If SL’s changes in behaviour reflect an increase in the size of her attentional window, we should see similar differences in the behaviour of participants viewing the scenes through small versus larger sized windows. In other words, participants viewing scenes through a larger window should look at the eyes more than participants with a smaller window.  148  Method Participants Window Size: Small Participants (n=13, 3 male) were undergraduate students at the University of British Columbia who ranged in age from 19 to 29 years (mean=21years). All participants reported normal or corrected-to-normal vision and gave informed consent prior to participation in the experiments, which were performed in accordance with the ethical guidelines of the University of British Columbia. The data from one participant was excluded from the analyses due to shift of the eye monitor during the task. Window Size: Large Participants (n=17, 7 male) were undergraduate students at the University of British Columbia who ranged in age from 17 to 23 years (mean=19years). All participants reported normal or corrected-to-normal vision and gave informed consent prior to participation in the experiments, which were performed in accordance with the ethical guidelines of the University of British Columbia. Stimuli, apparatus and procedure: The stimuli and procedure for Experiment 2 were the same as Experiment 1, except that participants viewed the scenes through a mouse-contingent aperture. The computer generated a square aperture centered on the mouse coordinates. This ‘window’ revealed a portion of the stimulus, with the screen outside the window being white. The aperture moved as participants moved the mouse across the screen. SL’s attentional window was previously estimated to be approximately 1.25° based her threshold for identifying the global level of hierarchical letter stimuli (Dalrymple, et al., 2007), thus the window sizes chosen for the present experiment were on the lower and upper side of that estimate: the “Small” window was 1° x 1°  149  while the “Large” window was 2° x 2° in size4. Participants were able to freely explore the screen with the window and their eyes were tracked according to the methodology used in Experiment 1. Eye fixations were defined as in Experiment 1. Data from the Large window group was previously reported in Dalrymple et al, (under review), whereas the Small window group data was collected for the purpose of the present study. Participants first underwent a practice trial using the mouse-contingent window. They were instructed to start from a circle at centre screen labeled “Start” and follow a line from that circle until they reached a second circle labeled “End”. This was designed to familiarize them with, and teach them how to control, the mouse-contingent aperture. They were then instructed to freely search the screen for a hidden object. Once they located the hidden object and felt comfortable with the apparatus, the main experiment began. Like SL, participants were asked to verbally describe social scenes while a digital voice recorder recorded their descriptions. Participants initiated the trials on their own, which were limited to 3 minutes. Participants rarely used the full 3 minutes to perform the task. Analysis: One participant from the Small aperture group was excluded due to shift of the eye monitor during the task. To determine where the mouse-contingent groups fixated, and if and how they differed, we analyzed the fixation proportion data with a two-way Mixed Design ANOVA with between-subjects factors of Window Size (Small vs. Large) and within-subjects factor of Region (Eye, Head, Body, Objects, and Background). Interactions were followed up with 1-way ANOVAs to compare the groups for each Region. We also computed average fixation durations, number of fixations per trial, and trial durations (Figure 6.4), and compared the Small and Large 4  Note that the “Large” window size was used in Dalrymple et al. (under review), while the “Small” window size is new to this experiment. The “Large” window group in Dalrymple et al had larger proportions of fixations on the eyes than SL at Time 1. Thus, in the present experiment, we are “working backwards” to simulate her Time 1 behaviour with the new, smaller window, while seeing if the original “Large” window better approximates her Time 2 behaviour. 150  aperture groups on this measure using one-way repeated measures ANOVAs with between subjects factor of Window Size (Small vs. Large). All p values were compared to α=0.05. Finally, we conducted a two-way mixed ANOVA with between subjects factor of Group (Small vs. Large window) and within subjects factor of time Interval (5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, and 60 seconds) to determine if the Small and Large window groups differ in their allocation of fixations on the Eye region (our primary region of interest) over time. We plotted each group’s proportions of fixations to this region at 15, 30, 45, and 60 seconds, to illustrate how their fixations are allocated over time (see Figure 6.3b).  Results Mouse-Contingent windows: Small vs. Large Omnibus ANOVA: There was a main effect of Region F(4,27)=13.78,p< 0.001, and a significant Window Size x Region interaction, F(4,108)=151.87,p< 0.001, but no main effect of Window Size F(1,27)=0.23,p=0.635. One-way ANOVAs and t-tests: There was a main effect of Window Size for each Region, indicating that the Small aperture group differed from the Large window group in terms of proportions of fixations on each of the Regions. The Large window group had a higher proportion of fixations on the Eye region compared to the Small window group, F(1,27)=132.71,p<0.001, and the Small window group had a higher proportion of fixations to all other regions compared to the Large window group: Head, F(1,27)=224.59,p<0.001; Body, F(1,27)=166.10,p<0.001; Objects, F(1,27)=203.90,p<0.001; Background, F(1,27)=95.94,p<0.001. The average duration of fixations for the Large window group was significantly shorter than that of the Small window group, F(1,27)=6.46,p=0.017, but these groups did not differ in terms of mean number of fixations per trial, F(1,27) =0.25,p=0.622, or mean trial duration, F(1,27)=0.02,p=0.879.  151  Mouse-Contingent window: Small. There was a main effect of Region, F(4,44)=100.24,p< 0.001. The Small window group’s highest proportion of fixations landed on the Body region (0.34), followed by Objects (0.26), Background (0.22), Head (0.11), and Eye (0.07). They had a significantly higher proportion of fixations to the Body than any other region. Their proportions of fixations to the Object and Background regions did not differ, though these proportions were significantly higher than the fixation proportions to the Head and Eye regions, which did not differ from each other. Mouse-Contingent window: Large. There was a main effect of Region, F(4,64)=88.41,p< 0.001. The Large window group’s highest proportion of fixations landed on the Eye region (0.36), followed by Head (0.31), Body (0.15), Objects (0.10), and Background (0.09). They had a significantly higher proportion of fixations to the Eye and Head region compared to all other regions, and a higher proportion of fixations to the Body region compared to the Background region. No other regions differed from each other. Interval analysis: There was a main effect of Interval F(11,27)=104.67,p <0.001, indicating that both the Small and Large window groups showed an increase in their fixations on the eyes over time. There was no main effect of Group, F(1,27)=2.97,p=0.097, or Group x Interval interaction, F(11,27)=0.62,p=0.811, indicating that the groups did not differ in terms of proportions of fixations on the eyes and that both groups increased their fixations on the eyes at the same rate. Summary The Large window group allocated a significantly greater proportion of fixations to the eyes in the scenes than did the Small window group. They also made significantly shorter fixations than the Small window group. Although these groups did not differ significantly in terms of the number of fixations made per trial, the Large window group made slightly more fixations per trial than the Small window group. Combined with shorter fixations, this lead to no difference in the average time spent describing each scene. As can be seen in Figure 6.3b, both groups show consistent  152  increases in the proportions of fixations on the eyes over time. Although the Small window group allocated a greater proportion of fixations to the eyes than the Large window group in the first 60 seconds of the trial, this difference was not significant. While the Small window group then fixated elsewhere in the scenes, reducing their overall proportion of fixations on the eyes relative to other regions, the Large window group continued to allocate more fixations to this region after the 60 second mark, resulting in a much larger overall proportion of fixations on the eyes compared to the Small window group.  Discussion Previously we reported that simultanagnosic patient SL showed reduced fixations on the eyes in social scenes compared to healthy controls (Dalrymple., et al. under review). We hypothesized that, with recovery, SL would allocate more fixations to the eyes in social scenes, behaving more like normal controls. Furthermore, we hypothesized that this behavioural change would be related to an expansion of the restricted window of attention that characterizes simultanagnosia, and that this expansion could be simulated in healthy subjects through the use of restricted viewing windows of different sizes. Our first hypothesis was supported by the results of Experiment 1, which indicate that SL did allocate more fixations to the eye region of social scenes after 3.5 years of recovery than she did when she was first tested. Our second hypothesis was supported by the results of Experiment 2, which indicate that healthy participants viewing social scenes through a larger mouse-contingent viewing window allocate more fixations to the eyes in social scenes than participants viewing scenes through a smaller mouse-contingent window. Furthermore, participants viewing the scenes through a smaller window show similar proportions of fixations on the eyes as SL at Time 1, while participants viewing the scenes through a larger window show similar proportions of fixations on the eyes as SL at Time 2. In addition, SL’s fixation durations at Time 1 were more similar to the fixation durations 153  of participants viewing the scenes through a small window (i.e. both groups had relatively long fixation durations), while her fixation durations at Time 2 were more similar to the fixation durations of participants viewing the scenes through a larger window (i.e. both groups had relatively short fixation durations). All groups show consistent allocations of fixations to the different regions over time. Participants’ descriptions of the scenes, though subjective in nature, are also informative about the similarities between SL and the window groups. Upon examination of these descriptions, we observed that both window groups and SL at Time 1 and Time 2 appear to determine the main action in the scene (e.g., people reading, see Figure 6.2a, right), but do so by describing details. As the trial times for SL and the window groups suggest, they spend longer collecting sufficient details to feel satisfied with their description of each scene. Interestingly, at Time 1, SL miscounts the number of people in one scene, counting one individual twice, without realizing that she had done so. This suggests that SL has poor spatial memory for where objects (or in this case, individuals) appear in the scene. One change in SL’s behaviour that is not captured by the change in window size is that SL spent more time describing the scenes at Time 2 compared to Time 1. We speculate that this is not related to SL’s recovery, but to her motivation and stamina at Time 2. That is, at Time 2, the healthier SL may have had more stamina been particularly motivated to describe the scenes to the best of her ability (which to her may have been as thoroughly as possible). Perhaps she was proud of her recovery and wanted to demonstrate her improved abilities. Motivation would not be a changing factor with the window groups and therefore time spent describing the scene would not change from small to large window sizes. How does SL compare to other simultanagnosics reported in the literature? We (Dalrymple et al, 2007) previously reported that SL shows the classic global processing deficits documented in other simultanagnosic patients (e.g. Clavagnier, et al., 2006; Karnath, et al., 2000; Shalev, et al., 2004, 2007). SL had difficulty naming the global level of hierarchical letters, particularly when they were large and made up of widely spaced local elements. SL’s recovery was documented with these 154  hierarchical stimuli. We tested her shortly after injury and then after some recovery and found that, like with the social scenes in the present study, her behaviour approaches, but does not reach, normal. That is, SL’s ability to name the global level of the hierarchical letter stimuli improved from her initial testing reported in Dalrymple et al, (2007), but is not yet at normal (perfect) levels. It is difficult to compare the magnitude of SL’s improvement on the letters versus the social scenes, but the fact that she shows partial improvement on the well-established letters task supports the results of partial improvement in the current scenes study. The results from these experiments not only document the behavioural changes of a patient with simultanagnosia while engaging in a complex behaviour, but also link those behavioural changes to an underlying mechanism, namely a change in the size of the attentional window that is characteristically restricted in simultanagnosia. The notion of simultanagnosia being related to a restricted window of visual attention has been evident in the literature since the early 1950s (Bay, 1953). Some referred to it as “shaft vision” (Bay, 1953; Tyler, 1968), while others (e.g. Thaiss and de Bleser, 1992) described a rigid reduction of the spatial extent of the visual “spotlight”, or “searchlight vision” (Trobe, 2001). More recently, several investigations have been aimed at understanding the nature of the restricted window of attention in simultanagnosia. For example, using a gaze-contingent display, we found that participants viewing hierarchical letters through a narrowed window of vision showed similar global processing deficits with hierarchical letter stimuli to what is typically seen with patients with simultanagnosia (Dalrymple, et al., 2010). Participants viewing these letters through a gaze-contingent window showed similar accuracy patterns for identifying the global level of the letters compared to patient SL, specifically showing poor performance for large letters made up of widely spaced local elements and improving as letters became smaller and more densely packed. This was a first insight into the close relationship between a restriction of visual attention in simultanagnosia and a literal restriction of vision in healthy subjects.  155  We recently extended this link to more complex stimuli by investigating how patients with simultanagnosia scan social scenes, and how that relates to a literal window of vision (Dalrymple, et al., under review). Again we found a close relationship between patient behaviour and the behaviour of healthy participants viewing the scenes through a restricted viewing window: the patients and the restricted window groups showed reduced fixations on the eyes in social scenes compared to healthy control participants scanning the same scenes under natural viewing conditions. While the reason for the reduced fixations on the eyes remains unclear (though we offer a possible explanation below), the behavioural link between those with a restricted window of attention and those with an artificially imposed restricted window of vision is remarkable. We have reinforced this link in the present study by showing that the increase in fixations on the eyes in social scenes that occurs with recovery from simultanagnosia can be mimicked by increasing the size of a restricted window of vision with healthy participants. The present experiments contribute substantially to the findings reported in Dalrymple et al (under review) by documenting simultanagnosic behaviour over the course of recovery, and by relating changes in behaviour to the mechanisms hypothesized to underlie the disorder. To our knowledge, only one other longitudinal study of simultanagnosia has been reported in the literature. Nyffeler et al (2005) documented the scanning behaviour of a patient at 8, 14, and 37 weeks after injury, while she explored simple line drawing (e.g. objects, schematic clock). Consistent with the current findings, this patient showed improved performance over time (e.g. better object naming, increased exploration of the stimuli). Based on a qualitative assessment of the scan patterns executed by this patient, Nyffeler et al concluded that the recovery reflected an enlargement of a restricted attentional field. This conclusion was primarily based on the fact that the patient’s scan patterns became more exploratory over the course of recovery (i.e. covered more area). The current data agree with these conclusions, and they also speak to the idea of an expanding attentional field by simulating it with an expanding window of vision in healthy subjects. While increased scanning of simple stimuli in Nyffeler et al’s study could be explained by other mechanisms (e.g. an improved ability to disengage attention from 156  previously attended stimuli), Experiment 2 of the current study establishes a direct link between an expansion of the useful area of processing and a recovery of normal behaviours in simultanagnosia. Simultanagnosia and other disorders of visual attention, such as visual neglect and extinction, have been linked to lesions to parietal areas (Posner & Peterson, 1990, Riddoch et al, 2010), but little work has been done to relate changes in patient behaviour to possible anatomical changes that may underlie recovery from these disorders. Our findings with simultanagnosia point to an anatomical change that results in an increase in the size of a restricted window of attention. It is possible that the initial restriction of attention in simultanagnosia is related to decreased cortical excitation in functional areas due to loss of input from damage to neighboring areas. Rizzo and Hurtig (1987) speculated that the spontaneous disappearance of objects despite steady fixation in simultanagnosia might reflect cortical fatigue, and Pavlov suggested that the visual deficits in simultanagnosia might be related to a “low tonus of excitation” in the visual cortex (Pavlov, 1955, p.609). If the initial restriction of visual attention is indeed related to reduced excitation of preserved parietal areas, it is possible that, with time, regions neighboring the damaged areas may regain cortical excitation through alternate cortical pathways. Recovery of excitation in the preserved regions may in turn lead to a corresponding expansion of the window of attention. Support for this hypothesis comes from evidence that the administration of stimulants (i.e. caffeine) to patients with simultanagnosia temporarily improved their ability to see more than one object at a time (Luria, 1959). Patients also claimed to see things “in a brighter light”, (Luria, 1959, p.447). Perhaps these stimulants provided the extra excitation to preserved brain areas that otherwise returns slowly over time. Our data provide insight into why a reduced window of attention results in reduced fixations on eyes in social scenes, with resulting implications for social attention in general. Previous research has suggested that people look at the eyes of people in social scenes because the eyes are informative to the viewer (Birmingham, et al., 2007, 2008b). Specifically, eyes and eye gaze provide  157  information about the attentional state of the people in the scenes, e.g., to whom, or to what, people in the picture are directing their attention. This is supported by the finding that when asked to describe where people in the scene are directing their attention, subjects make significantly more fixations on the eyes than they do when asked to simply describe the scenes (Birmingham, et al., 2008a; Smilek, et al., 2006). In the present study, the reduced fixations on the eyes from patients and healthy subjects viewing scenes through a restricted window suggests that the eyes have lost some of their informative value when they are viewed outside the context of the whole scene. This suggests that without context to help infer attentional states, the viewer can no longer gather this important information from the eyes and must look elsewhere. The fact that fixations on the eyes increase with recovery and with a larger window of viewing suggests that with increased context, the informative value of the eyes increases. This is not an all or nothing change, but rather a graded increase in fixations to eyes that occurs with a graded increase in context. A restricted window of attention may not be the only explanation for the simultanagnosic deficits. Throughout this century of research on simultanagnosia, several suggestions have been made about the mechanisms underlying this complex disorder. Some have suggested decreased cortical excitation (Luria, 1959; Rizzo & Hurtig, 1987), explaining perceptual events such as the spontaneous disappearance of objects despite steady fixation (Rizzo & Hurtig, 1987). Similarly, simultanagnosia has been placed in the context of the Integrated Competition Hypothesis (Duncan, et al., 1997), with the suggestion that processing resources are depleted in simultanagnosia, resulting in all or nothing competition between objects (Jackson, Swainson et al., 2009). Others have suggested that an inability to disengage from attended stimuli prevents the perception of new stimuli (Farah, 1990), though recent tests of this idea have failed to support it (Clavagnier, et al., 2006; Dalrymple, et al., 2009). Finally, some have suggested that impaired object perception in simultanagnosia results from an inability to combine preserved space and object information (Coslett & Lie, 2008). Although all interesting and valid, it is important to note that these different theories are not necessarily mutually exclusive with each other, or with the concept of a restricted window of attention. For example, 158  it is possible that patients can experience the disappearance of objects from awareness (e.g. Rizzo & Hurtig, 1987) within a restricted window of attention. Other simultanagnosic behaviours, like the inability to see more than one of two overlapping figures (i.e. objects that take up the same spatial location) (Rafal, 2001), also point to additional perceptual limitation that could occur within a limited window itself. Investigating these possibilities is an important avenue for future research. Our findings have broad implications for theories of simultanagnosia. While it is generally agreed that simultanagnosia is characterized by a deficit of visual attention (Holmes & Horrax, 1919), it is sometimes unclear what is meant by “attention” in this context. For example, some authors suggest that patients have a disorder of object-based attention, as evidenced by impaired processing of objects (e.g. Rafal, 2001), while others suggest that patients suffer from a disorder of spacebased attention (e.g. Coslett, et al., 1995; Robertson, et al., 1997) which is evidenced by narrowed spatial extent of the useful field of processing with otherwise fully functioning retinas (Holmes & Horrax, 1919; Thaiss & de Bleser, 1992; Tyler, 1968). Our findings support a spatial attentional account of this disorder. Furthermore, our findings provide evidence of unique properties of the disordered spatial attention that is seen in Bálint syndrome: spatial attention is restricted, but the degree of this restriction can change over time. This change in turn brings with it important consequences for the resulting perceptual experience of the patient.  159  References Balint, R. (1909). Seelenlähmung des "Schauens", optische Ataxia, räumliche Störung der Aufmerksamkeit. Monatsschrift für Psychiatrie und Neurologie, 25, 51-81. Bay, E. (1953). Disturbances of visual perception and their examination. Brain, 76(4), 515-550. Birmingham, E., Bischof, W., & Kingstone, A. (2007). Why do we look at people's eyes? Journal of Eye Movement Research, 1(1), 1-6. Birmingham, E., Bischof, W., & Kingstone, A. (2008a). Gaze selection in complex social scenes. Visual Cognition, 16(2/3), 341-355. Birmingham, E., Bischof, W., & Kingstone, A. (2008b). Social attention and real world scenes: The roles of action, competition and social content. Quarterly Journal of Experimental Psychology, 61(7), 986-998. Clavagnier, S., Fruhmann Berger, M., Klockgether, T., Moskau, S., & Karnath, H.-O. (2006). Restricted ocular exploration does not seem to explain simultanagnosia. Neuropsychologia, 44, 2330-2336. Coslett H and Lie G. (2008). Simultanagnosia: When a rose is not red. Journal of Cognitive Neuroscience, 20(1), 36-48. Coslett, H., Stark, M., Rajaram, S., & Saffran, E. (1995). Narrowing the spotlight: A visual attentional disorder in presumed Alzheimer's disease. Neurocase, 1(4), 305-318. Dalrymple, K., Birmingham, E., Bischof, W., Barton, J., & Kingstone, A. (under review). Experiencing simultanagnosia through windowed viewing of social scenes. Dalrymple K, Bischof W, Cameron D, Barton J, and Kingstone A. (2009). Global perception in simultanagnosia is not as simple as a game of connect-the-dots. Vision Research, 49(14), 1901-1908. Dalrymple, K., Bischof, W., Cameron, D., Barton, J., & Kingstone, A. (2010). Simulating simultanagnosia: spatially constricted vision mimics local capture and the global processing deficit. Experimental Brain Research, 202, 445455. Dalrymple, K., Kingstone, A., & Barton, J. (2007). Seeing trees OR seeing forests in simultanagnosia: Attentional capture can be local or global Neuropsychologia, 45, 871-875. Duncan J, Humphreys G, and Ward R. (1997). Competitive brain activity in visual attention. Current Opinion in Neurobiology, 7(2), 255-261.  160  Farah M. (1990). Visual agnosia: Disorders of object recognition and what they tell us about normal vision. Cambridge: MIT Press. Goodglass, H., & Kaplan, E. (1983). The assessment of aphasia and related disorders (2nd ed.). Philadephia: Lea and Febiger. Holmes, G., & Horrax, G. (1919). Disturbances of spatial orientation and visual attention, with loss of stereoscopic vision. Archives of Neurology and Psychiatry, 1, 385-407. Huberle, E., & Karnath, H.-O. (2006). Global shape recognition is modulated by the spatial distance of local elements - Evidence from simultanagnosia. Neuropsychologia, 44(6), 905-911. Jackson G, Swainson R, Mort D, Husain M, and Jackson S. (2009). Attention, competition, and the parietal lobes: Insights from Bálint syndrome. Psychological Research, 73(2), 263-270. Karnath, H.-O., Ferber, S., Rorden, C., & Driver, J. (2000). The fate of global information in dorsal simultanagnosia. Neurocase, 6, 295-306. Kim, M., & Robertson, L. (2001). Implicit representations of space after bilateral parietal lobe damage. Journal of Cognitive Neuroscience, 13(8), 1080-1087. Kinsbourne, M & Warrington, EK. (1962). A disorder of simultaneous form perception. Brain, 85(3), 461-486. Kinsbourne, M & Warrington, EK. (1963). The localizating significance of limited simultaneous visual form perception. Brain, 86(4), 697-702. Leff AP, Spitsyna G, Plant GT, Wise RJ. (2006). Structural anatomy of pure and hemianopic alexia. Journal of Neurology, Neurosurgery, and Psychiatry. 77(9),1004-1007. Leibovitch, F., Black, S., Caldwell, C., Ebert, P., Ehrlich, L., & Szalai, J. (1998). Brain-behaviour correlations in hemispatial neglect using CT and SPECT: The Sunnybrook Strong Study. Neurology, 50, 901-908. Luria A. (1959). Disorders of "Simultaneous perception" In a case of bilateral occipito-parietal brain injury. Brain, 82(3),437-449. Nyffeler T, Pflugshaupt T, Hofer H, Bass U, Gutbrod K, von Wartburg R, Hess C, & Muri R. (2005). Oculomotor behaviour in simultanagnosia: A longitudinal case study. Neuropsychologia, 43(11), 1591-1597. Pavlov I. (1955). Selected works (s. Belsky, trans.). Moscow: Foreign Languages Publishing House. Posner M. & Peterson S. (1990). The attention system of the human brain. Annual Review of Neuroscience, 13, 25-42.  161  Rafal, R. (2001). Balint's syndrome. In M. Behrmann (Ed.), Handbook of Neuropsychology (2nd ed., Vol. 4, pp. 121-141): Elsevier ScienceB.V. Riddoch, JM, Chechlacz, M, Mevorach, C, Mavritsaki, E, Allen, H, & Humphreys, GW. (2010). The neural mechanisms of visual selection: the view from neuropsychology. Annal of the New York Academy of Sciences, 1191(1),156181. Rizzo M. & Hurtig R. (1987). Looking but not seeing: Attention, perception, and eye movements in simultanagnosia. Neurology, 37(10),1642-1648. Rizzo, M. & Vecera, S. (2002). Psychoanatomical substrates of Balint's syndrome. Journal of Neurology, Neurosurgery and Psychiatry, 72, 162-178. Robertson, L., Treisman, A., Friedman-Hill, S., & Grabowecky, M. (1997). The interaction of spatial and object pathways: evidence from Balint's Syndrome. Journal of Cognitive Neuroscience, 9(3), 295-317. Shalev, L., Humphreys, G., & Mevorach, C. (2004). Global processing of compound letters in a patient with Balint's syndrome. Cognitive Neuropsychology, 22(6), 737-751. Shalev L., Mevorach C., & Humphreys G. (2007). Local capture in Bálint 's syndrome: Effects of grouping and item familiarity. Cognitive Neuropsychology, 24(1), 115-127. Smilek, D., Birmingham, E., Cameron, D., Bischof, W., & Kingstone, A. (2006). Cognitive Ethology and explory attention in real-world scenes. Brain Research, 1080(101-119). Thaiss, L., & de Bleser, R. (1992). Visual agnosia: A case of reduced attentional 'spotlight'? Cortex, 28(4), 601-621. Trobe, J. (2001). The Neurology of Vision. Oxford: Oxford University Press. Tyler, R. (1968). Abnormalities of perception with defective eye movements (Balint's Syndrome). Cortex, 4(2), 154-171.  162  Chapter 7 General discussion  163  This dissertation documents visual perception in Bálint syndrome through the use of simple and complex stimuli. The ultimate goal was to determine how abnormal visual experiences in Bálint syndrome relate to the restriction of visual attention that is thought to characterize the disorder. Five main research questions were raised in Chapter 1 of this dissertation. Below I consider how the data from my dissertation speak to these questions.  1. Why are patients with Bálint syndrome impaired at piecing together local elements into a global whole? The present dissertation began with a question that was implicit in the literature, but rarely explicitly addressed. Although it became popular to test Bálint perception with simple hierarchical letters because of patients’ clear and consistent processing deficit for global level stimuli, very few researchers were asking the key question of why global level perception fails in Bálint syndrome. In Study 1 I investigated the underlying neural mechanisms of global level perception of hierarchical letters with a population of healthy subjects. These subjects showed a distinct increase in attentional processing for global letters that are large and made up of widely spaced local elements- precisely the stimuli that patients with Bálint syndrome most frequently fail to identify. Thus, from Study 1 I concluded that patients with Bálint syndrome may be impaired at global perception of large global shapes with widely spaced local elements because those shapes require the recruitment of additional attentional resources for active grouping. Due to restriction of visual attention that characterizes this disorder, these stimuli may exceed the limited attentional resources of patients with Bálint syndrome. Study 2 also provided insight into this issue. In this study I acknowledged that patients were impaired at ‘normal’ global level perception, but asked whether they may be able to derive the global shape through successive fixations to local elements (e.g. through a “connect-the-dots” strategy). Contrary to my hypothesis, I found that when patients successfully identify global stimuli, they do not do so through the piecing together of local items through a series of successive fixations. Eye movements were not predictive of success for identifying the letters. Any  164  successful global perception seems to be the result of residual ‘normal’ global processing, perhaps due to the expansion of their restricted window of attention. Study 3 was designed to investigate the link between global level processing deficits and the restricted window of attention that is proposed to underlie the Bálint deficits. Here I investigated whether a literal restriction of vision in healthy individuals would be sufficient to mimic the restricted window of attention in Bálint syndrome, leading to similar global level processing difficulties despite normally functioning brains. Having healthy individuals view hierarchical letter stimuli through a gazecontingent window successfully produced Bálint-like accuracy patterns for identifying both the global and local levels of these stimuli. This led to the conclusion that the restricted window of attention in Bálint syndrome, through a reduction of the useful visual field, may not only impair ‘normal’ global processing, but, combined with normal limits of visual processing, may prevent patients from piecing together local elements into a larger, global form.  2. Is the global processing deficit in Bálint syndrome directly related to the restriction of visual attention that is thought to underlie this disorder? Study 3 from this dissertation sheds light on the answer to this question. The main purpose of simulating the Bálint visual experience in healthy individuals was to determine whether a literal restriction of vision would lead to Bálint-like behaviours in healthy individuals. I reasoned that if Bálint-like behaviours (in this case accuracy patterns when identifying global level stimuli) could be induced in healthy individuals through an artificial narrowing of their visual field, then Bálint syndrome itself may be related to a narrowing of the visual field. This narrowing, or limited ‘window’ of attention, is often alluded to in the literature, but formal tests of this concept have been minimal. Through use of a gaze-contingent window with healthy participants who viewed the same hierarchical letters as a simultanagnosic patient (SL), I discovered that these individuals showed the same clear pattern of perceptual difficulties as SL experienced with these stimuli. Thus, it appears as though the global processing deficit in simultanagnosia is the direct consequence of the  165  restricted window of attention that characterizes the disorder, both in terms of normal global processing, and in terms of piecing together the global form through successive fixations of local elements.  3. How do patients with Bálint syndrome scan complex stimuli? Part 2 of this dissertation sought to move beyond typical testing with simple stimuli to determine how patients with Bálint syndrome scan complex stimuli. In Study 4 I characterized and quantified the eye movement patterns of two patients (SL and KC) with Bálint syndrome while they described social scenes. I discovered that patients show abnormally low proportions of fixations to the eyes of people in social scenes, and instead distribute their fixations more evenly across the scenes compared to control subjects. This behaviour was particularly noteworthy because of the robust finding from previous studies that healthy subjects tend to allocate a disproportionate amount of fixations to the eyes in social scenes, across a variety of tasks (Birmingham, Bischof, & Kingstone, 2007, 2008a, 2008b).  4. Can a restricted window of visual attention account for more complex Bálint behaviours? Study 5 from this dissertation applied the same gaze-contingent technique from Study 3 to determine whether a literal restriction of vision can similarly induce Bálint-like scanning patterns in healthy individuals while they view complex stimuli (i.e. the social scenes). The restricted viewing paradigm in healthy subjects seems to be a valid model of Bálint syndrome with these stimuli: like the two patients, SL and KC, healthy individuals viewing scenes through a gaze-contingent window showed a reduction in the proportion of fixations to the eye regions of the scenes. This lends credence to the hypothesis that the patients’ unusual fixation distribution may be the result of their restricted window of visual attention. Study 6 furthered this view by demonstrating that healthy individuals viewing scenes through a computer mouse-controlled restricted viewing window showed the same reduction in fixations to the eyes of people in the social scenes as healthy individuals from Study 5, who viewed the scenes through a gaze-contingent window.  166  The results from Study 6, therefore, support the idea that the restricted window itself, rather than the method of controlling the window, is key to simulating the Bálint behaviours. This further supports the link between the restricted window of attention in Bálint syndrome and the behavioural repertoire of these patients.  5. How does Bálint patient behaviour change with recovery and what can we learn from changes in behaviour over the course of recovery? To complete this dissertation, Study 7 addresses how Bálint patient behaviour changes with recovery and how that might reflect a change in the restriction of the attentional window. Patient SL’s fixation distribution to the social scenes changed significantly from initial testing to testing after several years of recovery, with a significant increase in fixations on the eye region over time. Her fixation distributions at Time 1 and Time 2 were closely mimicked by healthy individuals viewing scenes through mouse-contingent windows of different sizes. Specifically, subjects viewing scenes through a small mouse-contingent viewing window showed reduced fixations to the eyes of people in the social scenes that were comparable to SL’s initial fixation distribution whereas subjects viewing the scenes through a large mousecontingent viewing window showed reduced fixations to the eyes of people in the scenes that were comparable to SL’s fixation distribution after recovery. The fact that changes to size of the artificial viewing windows mimicked patient recovery raises the possibility that the restricted window of attention in Bálint syndrome may expand with recovery.  Implications of this dissertation Object-based versus space-based theories of attention The implications of this dissertation are manifold. For one, the results from the studies that make up this dissertation speak to the underlying attentional mechanisms that lead to the chaotic visual experience of patients with Bálint syndrome. While there has been a long-term consensus on the fact that Bálint syndrome is a disorder of visual attention rather than blindness (Holmes & Horrax, 1919), there has been little agreement or clarification on the exact nature of the disorder of attention. Bálint syndrome has been touted as providing a unique view  167  into how the visual system defines an object (Baylis, Driver, Baylis, & Rafal, 1994; Holmes & Horrax, 1919) and patients with Bálint syndrome have been described as the perfect example of object-based selection, since they select single objects at the expense of all others (Baylis, et al., 1994). Yet a more careful consideration of the nature of the visual world of a patient with Bálint syndrome raises some interesting questions about theories of attention, and Bálint syndrome itself. For example, while the exclusive recognition of single objects in Bálint syndrome could suggest impaired object-based selection because the perception of a single object precludes the perception of all others, it could also be viewed as preserved object-based selection, since patients successfully recognize single objects. An absence of spatial attention could in theory restrict the number of objects that can be perceived. After all, if one cannot attend to more than one location in space, how can one perceive more than one object? This dissertation allows some of the previously bizarre and difficult to understand Bálint behaviours to be elucidated through a consistent framework of viewing Bálint syndrome as a disorder of visual spatial attention, particularly one of a restriction to the spatial area of the window of attention. This framework is consistent with the language used in the literature: Bay described it as a “peripheral constriction” (p. 545), Thaiss and de Bleser (1992) suggested that their patient may suffer from a rigid reduction of the spatial extent of the visual spotlight, and Tyler (1968) referred to the visual deficit in his patient as “shaft vision” (p. 166). The framework is also supported by several findings throughout the dissertation showing a relationship between a literal restriction of vision and the Bálint visual experience. The use of simple vs. complex stimuli in perception research As mentioned earlier, the gold standard in science has typically been to perform well-controlled experiments that only vary in terms of a restricted set of independent variables (preferably one) (Kingstone, Smilek, & Eastwood, 2008). This often involves the use of very simple, “impoverished” stimuli that are easy to manipulate and to keep constant between conditions apart from the critical manipulation. Part 1 of the current dissertation involves the use of such  168  impoverished stimuli, in this case, hierarchical letters, as an attempt to extend previous research in the field in a carefully controlled manner. The results from this first part of this thesis are informative, but one might ask how meaningful these results are in terms of their contact with behaviours from a patient’s day-to-day life. How often does a patient encounter a hierarchical letter in daily life? Can we extrapolate perceptual inaccuracies with this stimulus to other stimuli that are composed of parts? The research to-date has clearly demonstrated that a patient’s perception of these stimuli is influenced by the stimulus parameters (e.g. Dalrymple, et al., 2007; Shalev, Mevorach, & Humphreys, 2007), and even that other hierarchical stimuli, like hierarchical faces, may lead to entirely opposite findings (i.e. that the global face is perceived at the expense of the local elements, Dalrymple, Kingstone & Barton, 2007) suggesting that caution must be exercised when attempting to apply these results to perception in the real world. Complex stimuli, though more difficult to control, may offer exciting new insights into real-world perception in Bálint syndrome. As mentioned, there is a dearth of literature involving quantitative analyses of perception of pictures and scenes in patients with Bálint syndrome. Only a handful of studies have investigated how complex objects and scenes are processed by the Bálint brain, and arguably, many of those studies are merely qualitative in nature (e.g. Nyffeler, et al., 2005; Tyler, 1968). Complex stimuli can provide patients with more information for use in synthesizing a perceptual experience. The combination of multiple overlapping elements, colours, and depths, can facilitate a patient’s ability to successfully deduce the identity of what is being seen. This can challenge researchers to develop stimuli and tasks that make contact with real-world perception, while uncovering sometimes subtle, yet informative, deficits of vision. Indeed what was learned from Part 2 of this dissertation, involving the perception of social scenes, is that top-down guidance of scene perception is affected by the Bálint deficits such that patients with Bálint syndrome show reduced fixations to the eyes in social scenes. Documenting such a complex behaviour and later modeling it with a restricted window paradigm provides rich information about the nature of the disorder and its affect on the processing of stimuli in the real world. This can arguably offer more concrete and applicable 169  avenues that may benefit patients through the creation of more naturalistic rehabilitative strategies. Social attention It is a well-established and robust finding that healthy individuals tend to spend a disproportionate amount of time looking at eyes of people in social scenes (e.g.Birmingham, et al., 2007, 2008a, 2008b). This finding is consistent across a variety of tasks, such as describing, memorizing, or recalling a scene. Part 2 of this dissertation suggests that this blatantly normal behaviour is compromised in Bálint syndrome: patients with a limited window of attention allocate an abnormally low proportion of fixations to eyes in social scenes, and instead distribute their eye movements more evenly across all elements of a scene. This implies that the eye region is not especially informative to patients with restricted visual attention and to people viewing scenes through a literal restriction of the visual field, such as participants in the gaze- and mouse-contingent groups used in Studies 5 and 6. Birmingham et al. (2007) argue that people look at eyes in social scenes because they provide rich social information regarding the meaning of a scene, especially regarding how people in the scene are allocating their attention to other people and objects within that scene. Patients with Bálint syndrome, and healthy participants using a gaze- or mouse-contingent window, do not have access to visual information beyond a confined perceptual window and are therefore unlikely to be able to infer where people in the scenes are attending. Similarly, items within a scene are unlikely to be perceived as “being looked at” or “not being looked at” by one or more individuals in the scene. This has important implications for the processing of social information by patients with these images, but also in the real world. Neuroanatomy The results from this dissertation have important implications about the anatomy underlying the Bálint deficits and the contributions of the parietal lobes to visual perception and attention. Attention has been said to be the “glue” that holds the visual world together, and it is clear that the parietal lobes are heavily involved in 170  the creation of a coherent visual experience. Patients with Bálint syndrome, who suffer from bilateral lesions to the parietal-occipital junction view the world in a piecemeal fashion, showing little understanding for what they see and how it fits together in their visual world. Patients may be completely disoriented when relying on their visual sense, but can recover spatial awareness performing similar tasks with eyes closed (Karnath, Ferber, Rorden, & Driver, 2000), suggesting a strong visual component to the attentional system that the parietal lobes command. The support of a space-based theory of Bálint syndrome garnered in this dissertation further adds to the ever growing body of literature on the role of the parietal lobes in the processing of space (see Goodale & Miller, 1992, for a review). Patient care and rehabilitation It is unclear how the findings of this dissertation can affect patient care and rehabilitation. The long-term investigations with patient SL in Study 7 provide hope for some natural recovery of function over time. However, SL, after nearly 4 years of healing, is still far from normal in terms of her perceptual abilities. The tasks used in the studies of this dissertation are not expected to be useful for rehabilitation purposes and it is uncertain that the findings, though critical for shedding light on the mechanisms of impaired perception in Bálint syndrome, could contribute to the transition from abnormal to normal visual experiences in this patient population.  Limitations of this dissertation Although the restricted window paradigm that was used extensively throughout this dissertation provided a reasonable model of Bálint behaviours with simple hierarchical stimuli and with quantifiable eye movement distributions from complex scenes, there are some limitations to this methodology. There is a growing body of evidence in the literature that the restricted window of attention in Bálint syndrome is flexible in size (i.e. expandable). In every day situations, patients can see only one object at a time, regardless of object size (Rafal, 2001). The restricted window has been estimated to be 2 degrees at rest, but up to 20 degrees in some instances Tyler (1968). Shalev et al (2004) discovered that size of the spatial area that is processed by these patients can even be expanded through priming. It is  171  possible that the window is expanded at the expense of visual acuity (Dalrymple & Kingstone, under review). The artificial window used in this dissertation was rigid, and fixed in size, not allowing for tests of some of the above-mentioned properties of the Bálint window of attention. While Study 7 investigated changes to the size of the visual window in order to draw inferences about how the attentional window in Bálint syndrome may expand over recovery, this was a between-subjects design, and did not test the expansion hypothesis in a gradual manner. Furthermore, since the size of the window in Bálint syndrome is thought to change with stimulus properties (e.g. going from ant to elephant), the rigid window in this dissertation did not allow for online changes to the size of the model window. Despite these limitations, the rigid window does a reasonable job of mimicking Bálint behaviours with both simple and more complex stimuli. In future, a more flexible window can be designed to test other properties of the attentional window in Bálint syndrome. For example, a small, clear window could be used to mimic the window at rest, and this window could be expandable at the expense of acuity for the elements that fall within the window. Subjects could expand the window online through key press, and their behaviour could be monitored with respect to how visual perception changes with changes to the window, and how closely this models Bálint perception. One draw back to this approach, however, is that it is unclear whether patients with Bálint syndrome have conscious control over the properties of their window of attention. This means that allowing conscious control in a simulation of Bálint syndrome may not be theoretically justified.  Behaviours that suggest that Bálint syndrome involves more than a limited window of attention alone Although this dissertation provides systematic evidence for a relationship between the disordered perception in Bálint syndrome and a narrowed window of vision, some results in the literature cannot be easily explained by a reduced window of attention. One example of such a behaviour is the high number of Illusory Conjunctions (ICs) experienced by patients with Bálint syndrome (Friedman-Hill, 172  Robertson, & Treisman, 1995). ICs occur when the features of one object are mistakenly bound to another object (Treisman & Gelade, 1980). For example, when viewing a red letter X and a black letter O, a patient may report seeing a black letter X or a red letter O. One might predict that a reduced window of attention would enhance the processing of single objects that fall within the window and promote the correct binding of the features of those objects, yet patients show an unusually large number of incorrect binding errors. ICs occur in healthy individuals with rapid presentation times (200ms) and under conditions of divided attention (Treisman & Gelade, 1980; Treisman & Schmidt, 1982), thus it is possible that patients experience reduced or slowed processing capacity within the restricted visual attentional window. One patient, RM, experienced ICs with limited presentation durations (up to 10s), supporting this possibility (Friedman-Hill, et al., 1995). Interestingly, RM experienced fewer IC’s during serial compared to simultaneous presentation of the stimuli, suggesting that processing information at a single spatial location is more proficient than processing information at two locations simultaneously. This could be consistent with a restricted window theory of Bálint syndrome if the window remains constricted for items presented in succession, but must be expanded to process two items simultaneously. Perhaps, like with the hierarchical faces that are processed at a global level at the expense of “attentional acuity” for the local elements (Dalrymple, Kingstone, & Barton, 2007), attentional processing is less efficient when spread over a larger spatial area, resulting in a higher rate of ICs during simultaneous presentation. Thus, while not a clear fit with a restricted window of attention, the increased rate of ICs in Bálint syndrome is not necessarily inconsistent with this view. Another effect that cannot be easily explained by a restricted window of attention is the spontaneous disappearance of objects that are being fixated (Rizzo & Hurtig, 1987). Patients report that objects may disappear from awareness despite confirmed steady fixation. Rizzo and Hurtig speculate that this reflects cortical fatigue, which prevents sustained processing of stimuli to a level of conscious awareness, consistent with Pavlov’s hypothesis from several years earlier that the  173  visual deficits in Bálint syndrome were related to “low tonus of excitation” in the visual cortex (Pavlov, 1955, p.609). This hypothesis is supported by Luria’s (1959) finding that the administration of caffeine (a stimulant) can treat simultanagnosic symptoms. Luria’s patient’s performance improved on a number of tasks 15 to 20 minutes after injections of 0.05 to 0.1 grams of caffeine. For example, with tachistoscopic presentation of two simultaneously figures, the patient’s performance improved from seeing both figures on 0 of 30 trials to seeing both on 12 of 30 trials. Furthermore, the patient claimed to see things “in a brighter light” (p.447). While Luria’s results could be explained by an expansion of the window of attention with caffeine administration, it is more difficult to explain how single objects that are fixated may disappear. One possibility is that with cortical fatigue, the window of attention closes entirely. Another explanation is that information processing within the window of attention can be fatigued.  A note on the link between the restricted “window” of attention and the “spotlight” of attention The terminology used in the literature to discuss the spatial restriction of attention in Bálint syndrome can arguably be described as inconsistent and vague. While the perceptual end result is described with such terms as “shaft vision” (Bay, 1953; Tyler, 1968), and “peripheral contraction” (Bay, 1953), these descriptions stop short of offering conjectures about possible mechanisms that may underlie this overt perceptual experience. In the present dissertation, these analogies were applied in a literal sense, using a literal restriction of vision to create a reasonable model of the Bálint behaviours. However, despite the similarities in the overt behaviour, it is doubtful that the mechanisms underlying these behaviours are the same. In keeping with the early deductions by Holmes and Horrax (1919), this dissertation is not meant to suggest that Bálint behaviours result from a literal constriction of vision, but rather an attentional restriction. Moving then from vision to visual attention, one tempting analogy is between the restriction of the visual window of attention in Bálint syndrome, and a restriction of the “spotlight of attention” reported in normative  174  studies (Posner, 1980). Thaiss and deBleser (1992) even make this comparison directly, describing a neurological case of a “reduced attentional “spotlight”” (p.601), but little has been done to empirically test how well this analogy holds. Careful consideration of the literature can offer some insight into this issue. Much like the spotlight theory of attention proposed by Posner and colleagues in the early 1980s (Posner, 1980; Posner & Cohen, 1984; Posner, Nissen, & Ogden, 1978; Posner, Snyder, & Davidson, 1980), the concept of a restricted window of attention in Bálint syndrome is a spatial, rather than object-based, account of attention. The current dissertation establishes a detailed line of reasoning for why a spatial account of the Bálint deficits is appropriate. Also analogous, is the fact that both the spotlight of attention and the restricted window of attention in Bálint syndrome seem to widen and shrink at the expense of acuity. For example, the normative spotlight of attention can precisely cover a small region of space, or can widen to cover a larger region of space, though with a reduced accuracy for detail (e.g.Humphreys, 1981). The window of attention in Bálint syndrome can be widened through priming (Shalev, et al., 2004), and this widening may cause a loss of resolution of local details (Dalrymple, et al., 2007). Finally, peripheral cueing can be used to direct the spotlight of attention in healthy individuals (e.g. Eriksen & Hoffman, 1973; Posner, et al., 1978). Similarly, despite a loss of conscious awareness for peripheral stimuli, patients with Bálint syndrome also respond to cues in the periphery and may reorient their restricted window of attention accordingly (Egly, Robertson, Rafal, & Grabowecky, 1995). It is clear that there is evidence suggesting that Bálint syndrome may result from a restricted “spotlight” of attention, but further research is needed to confirm this conjecture. This is critical for understanding the underlying mechanisms of these Bálint deficits, and must be accomplished before the spotlight terminology can be safely applied in the Bálint literature. Possible avenues for future investigations into this issue are discussed below and outline important next steps into elucidating the true nature of Bálint syndrome.  175  Future directions The current dissertation provides insight into the relationship between impaired Bálint perception and a rigid spatial restriction of visual information. An obvious first step for future research would be to explore other properties of the window of attention in Bálint syndrome and attempt to incorporate those properties into a new simulation model of the disorder. It was previously mentioned that one challenge would be to create an expandable window to mimic the changes in size that seem to occur under certain conditions with Bálint patients. Similarly, it would be of interest to design a window paradigm that would allow normal peripheral cueing as is experienced by the patients (Egly, et al., 1995). The use of the opaque periphery in the current paradigm precludes the possibility of testing for similar peripheral cueing with healthy individuals, and therefore a translucent surrounding area may be preferable. One challenge would be to determine the degree of transparency needed to create a realistic analogy of the Bálint experience. Another important aspect of Bálint syndrome that could be addressed in future research is to make contact with each of the characteristic symptoms of the disorder. For example, patients with Bálint syndrome can only see one object at a time (simultanagnosia), but also have difficulty using visual information to guide accurate reaching towards objects (optic ataxia). It is possible that healthy subjects in a three dimensional version of the gaze-contingent window paradigm would also have difficulties reaching for objects they see. Seeing one object at a time impairs the ability to judge where objects are in space, relative to each other, and even relative to the environment (Rafal, 2003). Therefore it is reasonable to predict that limiting visual input to a small window of space would lead to behaviours akin to optic ataxia. This could have implications for how the restricted attentional resources in Bálint syndrome may influence the entire spectrum of Bálint behaviours. Finally, an important avenue for future research is to isolate the critical brain areas involved in creating the Bálint deficits. In this dissertation I provide data on three patients with bilateral parietal injury, two of who have Bálint syndrome, and one of whom does not. Furthermore, of the patients with Bálint syndrome, one  176  patient (SL) is more severely impaired than the other (KC). Through imaging techniques and comparative brain analyses, it may be possible to determine what anatomical substrates contribute to the severity of Bálint syndrome. This could in turn lead to a greater understanding of the contributions of the parietal lobes to visual perception and attention.  177  References Bay, E. (1953). Disturbances of visual perception and their examination. Brain, 76(4), 515-550. Baylis, G., Driver, J., Baylis, L., & Rafal, R. (1994). Reading of letters and words in a patient with Bálint's syndrome. Neuropsychologia, 32(10), 1273-1286. Birmingham, E., Bischof, W., & Kingstone, A. (2007). Why do we look at people's eyes? Journal of Eye Movement Research, 1(1), 1-6. Birmingham, E., Bischof, W., & Kingstone, A. (2008a). Gaze selection in complex social scenes. Visual Cognition, 16(2/3), 341-355. Birmingham, E., Bischof, W., & Kingstone, A. (2008b). Social attention and real world scenes: The roles of action, competition and social content. Quarterly Journal of Experimental Psychology, 61(7), 986-998. Dalrymple, K., & Kingstone, A. (under review). A world unglued: Piecing together the contributions of object- vs. space-based attention in Bálint syndrome. Dalrymple, K., Kingstone, A., & Barton, J. (2007). Seeing trees OR seeing forests in simultanagnosia: Attentional capture can be local or global Neuropsychologia, 45, 871-875. Egly, R., Robertson, L., Rafal, R., & Grabowecky, M. (1995, November). Implicit processing of unreportable objects in Bálint's syndrome. Paper presented at the Psychonomic Society, Los Angeles. Eriksen, C., & Hoffman, J. (1973). The extent of processing of noise elements during selective encoding of visual displays. Perception and Psychophysics, 14, 155160. Friedman-Hill, S., Robertson, L., & Treisman, A. (1995). Parietal contributions to visual feature binding: Evidence from a patient with bilateral lesions. Science, 269(5225), 853-855. Goodale, M. A., & Miller, M. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15(1), 20-25. Holmes, G., & Horrax, G. (1919). Disturbances of spatial orientation and visual attention, with loss of stereoscopic vision. Archives of Neurology and Psychiatry, 1, 385-407.  178  Humphreys, G. (1981). On varying the span of visual attention: Evidence for two modes of spatial attention. Quarterly Journal of Experimental Psychology, 33A, 17-31. Karnath, H.-O., Ferber, S., Rorden, C., & Driver, J. (2000). The fate of global information in dorsal simultanagnosia. Neurocase, 6, 295-306. Kingstone, A., Smilek, D., & Eastwood, J. (2008). Cognitive Ethology: A new approach for studying human cognition. British Journal of Psychology, 99, 317-345. Luria, A. (1959). Disorders of "simultaneous perception" in a case of bilateral occipito-parietal brain injury. Brain, 82, 437-449. Michel, F., & Henaff, M.-A. (2004). Seeing without the occipito-parietal cortex: Simultanagnosia as a shrinkage of attentional visual field. Behavioural Neurology, 15, 3-13. Nyffeler, T., Pflugshaupt, T., Hofer, H., Bass, U., Gutbrod, K., von Wartburg, R., et al. (2005). Oculomotor behaviour in simultanagnosia: A longitudinal case study. Neuropsychologia, 43, 1591-1597. Pavlov, I. (1955). Selected works (S. Belsky, Trans.). Moscow: Foreign Languages Publishing House. Posner, M. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32(1), 3-25. Posner, M., & Cohen, Y. (1984). Components of visual orienting. In H. Bouma & D. Bouwhuis (Eds.), Attention and performance (Vol. X). Hillsdale, NJ: Erlbaum. Posner, M., Nissen, M., & Ogden, W. (1978). Attended and unattended processing modes: The role of set for spatial location. In H. L. P. a. I. J. Saltzman (Ed.), Modes of Perceiving and Processing Information. Hillsdale, NJ: Lawrence Erlbaum Associates. Posner, M., Snyder, C., & Davidson, B. (1980). Attention and the detection of signals. Journal of Experimental Psychology: General, 109, 160-174. Rafal, R. (2001). Bálint’s syndrome. In M. Behrmann (Ed.), Handbook of Neuropsychology (2nd ed., Vol. 4, pp. 121-141): Elsevier ScienceB.V. Rafal, R. (2003). Bálint's syndrome: A disorder of visual cognition. In M. D'Esposito (Ed.), Neurological foundations of cognitive neuroscience (pp. 27-40). Cambridge, MA: MIT Press.  179  Rizzo, M., & Hurtig, R. (1987). Looking but not seeing: Attention, perception, and eye movements in simultanagnosia. Neurology, 37(10), 1642-1648. Shalev, L., Humphreys, G., & Mevorach, C. (2004). Global processing of compound letters in a patient with Bálint’s syndrome. Cognitive Neuropsychology, 22(6), 737-751. Shalev, L., Mevorach, C., & Humphreys, G. (2007). Local capture in Bálint’s syndrome: Effects of grouping and item familiarity. Cognitive Neuropsychology, 24(1), 115-127. Thaiss, L., & de Bleser, R. (1992). Visual agnosia: A case of reduced attentional 'spotlight'? Cortex, 28(4), 601-621. Treisman, A., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97-136. Treisman, A., & Schmidt, H. (1982). Illusory conjunctions in the perception of objects. Cognitive Psychology, 14, 107-141. Tyler, R. (1968). Abnormalities of perception with defective eye movements (Bálint's Syndrome). Cortex, 4(2), 154-171.  180  Appendices  181  Appendix A Case reports Patient SL Patient SL is a patient with Bálint syndrome, who is reported in Studies 2, 3, 4, 5, and 7. She was tested at two time periods, Time 1, 8 months after her strokes, and Time 2, after 3.5 years of recovery. Her case report will be presented here for these two time periods. Time 1: June 2005 Patient SL is a right-handed woman, with 12 years of education. She had idiopathic cerebral vasculitis resulting in bilateral parietal and lateral occipital infarcts (Figure A.1.a). She had been treated with cyclophosphamide and prednisone for her vasculitis, but had completed these 4 months prior to her testing. At the time of testing she was on carbamazepine for a single seizure suffered several months prior. She presented with left hemi-neglect, as assessed with the Sunnybrook Neglect Assessment Battery (Leibovitch, et al., 1998), left inferior quadrantanopia, and Bálint syndrome, with ocular motor apraxia, optic ataxia, and simultanagnosia, though her acuity was 20/25 in both eyes. Her optic ataxia was evident in that she often mis-reached for objects, and failed to orient her grasp correctly to the axes of objects such as pencils. Her simultanagnosia was evidenced through tests with four complex displays of visual scenes. For example, she could report elements of the Boston Cookie Theft picture (Goodglass & Kaplan, 1983), but was unable to make sense of the whole scene. Neuropsychological evaluation showed normal attention, language, and verbal memory functions. Her reading was in the borderline impaired range and she tended to guess words based on the first or last letters. She was successful at recognizing simple line drawings of objects and could correctly identify colours and simple shapes. At the time of testing, SL was 48 years old and no longer showed left hemineglect or quadrantanopia and had no defects in saccadic targeting and generation, as was confirmed by her rapid and accurate saccades during the calibration of the 182  eye monitor. However she still showed optic ataxia when using the left hand to point to targets. This was a specific sensorimotor transformation for the contralateral hand, and therefore not a due to a general difficulty with perceptual localization (which would affect both hands).  a.  b.  Figure A.1. MRI scans of patient SL at (a) Time 1 and (b) Time 2  Time 2: November 2008 SL was 52 years old at the time of testing. She had no new neurologic events, or seizures since her original trauma (Figure A.1.b). Her acuity correction at far was 20/40-2 ou, 20/30-1 ou with pinhole. Her fixation and smooth pursuit showed distractibility, yet her saccades were quick and accurate, without delay in initiation to command and she showed no nystagmus. She was alert and attentive with normal language and speech. She could name multiple elements on the Boston Cookie Theft picture and described them fully. She could read short sentences well, with only slight hesitation with longer words or when finding the next line. She made randomly distributed errors of omission and commission on an object cancellation task. She still met the diagnostic criteria for Bálint syndrome, and had a left inferior quadrantic scotoma, secondary to her cerebral vasculitis.  183  Patient KC Patient KC is an additional patient, who came to our attention in March 2009. He suffers from simultanagnosia, and is reported in Study 4. KC is a 55 year-old man with posterior reversible leucoencephalopathy in the setting of Crohn’s disease being treated with omisartan as part of an experimental trial. He was seen 2 months prior to testing for fluctuating visual symptoms of several weeks’ duration. He stated that he “could see but not perceive”. He could see things and recognize them, but had trouble locating and searching for household objects, and could not reach for items accurately. He saw an “echo” or multiple ghosts of objects when he stared at them. His reading was slow and at his worst he had trouble recognizing faces and difficulty with locating objects in depth. All of these problems improved rapidly after omisartan was stopped. He was first examined four weeks after onset. His acuity with correction at far was 20/30 od and 20/40 os, which improved by pinhole to 20/25-1 os. Confrontation showed full visual fields. Fixation was steady; pursuit and VOR cancellation were normal. He showed normal initiation of saccades and saccadic accuracy. There was no nystagmus. His reaching was accurate, and he showed correct grasp orientation to objects. Reading was slow, but accurate and without a word-length effect. Line cancellation and object cancellation showed a few errors, but these were not lateralized. His recognition of line drawings was normal. With the Boston Cookie Theft picture, he was able to name all objects. Thus, while his initial symptoms were suggestive of bi-parietal dysfunction, at the time of his examination many of these deficits appeared to have resolved, in keeping with the diagnosis of a reversible leucoencephalopathy, with only some mildly slowed reading and minor difficulty with visual search being found. During the time of his experimental testing 2 months later, his verbal report of scenes provided further corroboration that he was able to perceive multiple elements of complex displays. His MRI showed bilateral parietooccipital and right posterior occipital white matter FLAIR hyperintensities, as well as a small left occipital cortical infarct (Figure A.2).  184  Figure A.2 MRI scans of patient KC Patient ES Patient ES serves as a control patient in Studies 2 and 4. Like SL and KC, ES suffered bilateral posterior occipitoparietal damage (Figure A.3). Unlike the other two patients, however, ES never had signs of simultanagnosia or symptoms suggestive of any component of Bálint syndrome. She is a 47 year-old woman with systemic lupus erythematosis, tested several months after presenting with flashing lights, transient visual loss and headache. Her visual examination was normal, but MR imaging revealed bilateral lesions consistent with either vasculitis or posterior leucoencephalopathy. Subsequently she had a seizure, and was treated with phenytoin for 9 months. At her most recent visit she was taking prednisone, chloroquine, and mycophenolate mofetil. Her visual acuity without correction at far was 20/20 in both eyes. Confrontation showed full visual fields. Fixation, pursuit and saccades were normal. There was no oculomotor apraxia, optic ataxia, or simultanagnosia as shown by normal report on the Boston Cookie Theft picture. ES matched SL particularly well in age, gender, the chronic phase at testing, and probable pathology, since she also has an underlying condition that is associated with vasculitis.  185  Figure A.3 MRI scans of patient ES  186  References Goodglass, H., & Kaplan, E. (1983). The assessment of aphasia and related disorders (2nd ed.). Philadephia: Lea and Febiger. Leibovitch, F., Black, S., Caldwell, C., Ebert, P., Ehrlich, L., & Szalai, J. (1998). Brain-behaviour correlations in hemispatial neglect using CT and SPECT: The Sunnybrook Strong Study. Neurology, 50, 901-908.  187  Appendix B UBC Behavioural Research Ethics Board Certificate of Approval  188  Appendix C UBC Behavioural Research Ethics Board Certificate of Approval  189  Appendix D UBC Clinical Research Ethics Board Certificate of Approval  190  191  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.24.1-0071294/manifest

Comment

Related Items