Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Khandelwal, Siddhesh Shyam
Master of Science - MSc
University of British Columbia
Visual attention mechanisms have proven to be integrally important constituent components of many modern deep neural architectures. They provide an efficient and effective way to utilize visual information selectively, which has shown to be especially valuable in multi-modal learning tasks. However, all prior attention frameworks lack the ability to explicitly model structural dependencies among attention variables, making it difficult to predict consistent attention masks. In this work we develop a novel structured spatial attention mechanism which is end-to-end trainable and can be integrated with any feed-forward convolutional neural network. This proposed AttentionRNN layer explicitly enforces structure over the spatial attention variables by sequentially predicting attention values in the spatial mask in a bi-directional raster-scan and inverse raster-scan order. As a result, each attention value depends not only on local image or contextual information, but also on the previously predicted attention values. Our experiments show consistent quantitative and qualitative improvements on a variety of recognition tasks and datasets, including image categorization, question answering and image generation.
Enforcing Structure in Visual AttentionbySiddhesh Shyam KhandelwalB.Tech., Indian Institute of Technology (IIT) Guwahati, 2016A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)October 2019c© Siddhesh Shyam Khandelwal, 2019The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Enforcing Structure in Visual Attentionsubmitted by Siddhesh Shyam Khandelwal in partial fulfillment of the require-ments for the degree of Master of Science in Computer Science.Examining Committee:Leonid Sigal, Computer ScienceSupervisorJames Little, Computer ScienceSupervisory Committee MemberiiAbstractVisual attention mechanisms have proven to be integrally important constituentcomponents of many modern deep neural architectures. They provide an efficientand effective way to utilize visual information selectively, which has shown tobe especially valuable in multi-modal learning tasks. However, all prior attentionframeworks lack the ability to explicitly model structural dependencies among at-tention variables, making it difficult to predict consistent attention masks. In thiswork we develop a novel structured spatial attention mechanism which is end-to-end trainable and can be integrated with any feed-forward convolutional neuralnetwork. This proposed AttentionRNN layer explicitly enforces structure over thespatial attention variables by sequentially predicting attention values in the spa-tial mask in a bi-directional raster-scan and inverse raster-scan order. As a result,each attention value depends not only on local image or contextual information,but also on the previously predicted attention values. Our experiments show con-sistent quantitative and qualitative improvements on a variety of recognition tasksand datasets, including image categorization, question answering and image gen-eration.iiiLay SummaryVisual attention mechanisms are increasingly being used in various computer vi-sion (CV) tasks due to their ability to focus on relevant image regions. They pro-duce a spatial mask for a given image, which is expected to have higher values overthe regions of interest, and lower values everywhere else.Most CV methods leverage the grid-like structure of images, wherein neigh-bouring pixels share some correlation. In this work we argue that, similar to animage, the attention mask also has an inherent structure over its values. Existingmethods often ignore this structure, hence generating âpatchyâ masks where neigh-bouring attention values have sharp discontinuities.We propose a novel attention mechanism that enforces structure over the atten-tion mask by sequentially predicting values, where each value inherits informationfrom the previously predicted ones. We show the generalizability of the proposedmechanism by augmenting it to existing models over different tasks.ivPrefaceAll the work presented in this document was done, in its entirety, at the ComputerVision lab in the Department of Computer Science at the University of BritishColumbia, Vancouver campus.The work presented is done entirely by the author, Siddhesh Shyam Khandel-wal, under the supervision of Prof. Leonid Sigal. The main content of this thesisis based on a version that was accepted as a conference paper at the InternationalConference on Computer Vision (ICCV), 2019.â¢ S. Khandelwal and L. Sigal. AttentionRNN: A Structured Spatial AttentionMechanism. In IEEE International Conference on Computer Vision (ICCV),2019Design, implementation and the experiments required for the paper were done bythe author. Prof. Sigal provided feedback during each step and also helped withrefining the conference publication.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . 52 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1 Visual Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Autoregressive Models for Image Generation . . . . . . . . . . . 73 Probabilistic View of Attention . . . . . . . . . . . . . . . . . . . . . 9vi4 AttentionRNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.1 Combination Function . . . . . . . . . . . . . . . . . . . . . . . 164.2 Block AttentionRNN . . . . . . . . . . . . . . . . . . . . . . . . 175 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.1 Visual Attribute Prediction . . . . . . . . . . . . . . . . . . . . . 185.1.1 Using Expressive Local Spatial Context δ . . . . . . . . . 245.1.2 Inverse Attribute Prediction . . . . . . . . . . . . . . . . 255.1.3 Scalability of ARNN . . . . . . . . . . . . . . . . . . . . 275.2 Image Classification . . . . . . . . . . . . . . . . . . . . . . . . . 285.3 Visual Question Answering . . . . . . . . . . . . . . . . . . . . . 295.4 Image Generation . . . . . . . . . . . . . . . . . . . . . . . . . . 306 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37A Supporting Materials . . . . . . . . . . . . . . . . . . . . . . . . . . 42A.1 Additional Visualizations . . . . . . . . . . . . . . . . . . . . . . 42A.1.1 Visual Attribute Prediction . . . . . . . . . . . . . . . . . 42A.1.2 Inverse Attribute Prediction . . . . . . . . . . . . . . . . 45A.1.3 Image Generation . . . . . . . . . . . . . . . . . . . . . . 48viiList of TablesTable 5.1 Architectures for the models used in Section 5.1. â implies thatthe previous and the next layer are directly connected. The inputis passed to the top-most layer. The computation proceeds fromtop to bottom. . . . . . . . . . . . . . . . . . . . . . . . . . . 20Table 5.2 Model architectures for the scalability study described in Sec-tion 5.1. â implies that the previous and the next layer are di-rectly connected. ARRN in defined in Table 5.1. . . . . . . . . 21Table 5.3 Color prediction accuracy. Results are in % on MREF, MDISTand MBG datasets. Our AttentionRNN-based model, CNN+ARNN,outperforms all the baselines. . . . . . . . . . . . . . . . . . . 21Table 5.4 Mask Correctness and Scale experiment on MBG. The âCorr.âcolumn lists the mask correctness metric proposed by [21]. TheâScaleâ column shows the color prediction accuracy in % fordifferent scales. . . . . . . . . . . . . . . . . . . . . . . . . . 24Table 5.5 Color prediction accuracy on MBG dataset. Results are in%. Using ARNN is conjunction to local attention mechanismssignificantly improves performance. . . . . . . . . . . . . . . . 25Table 5.6 Block AttentionRNN. Ablation results on MBGb dataset. At-tentionRNN (ARNN) and Block AttentionRNN (BRNN) withblock sizes of 2 and 3 are compared. . . . . . . . . . . . . . . 27Table 5.7 Performance on Image Classification. The Top-1 and Top-5error % are shown for all the models. The ARNN based modeloutperforms all other baselines. . . . . . . . . . . . . . . . . . 29viiiTable 5.8 Performance on VQA. In % accuracy. . . . . . . . . . . . . . 29Table 5.9 Performance on Image Generation. ResNet18 [8] Classifica-tion Errors (%) for each attribute transformation. ARNN achievesbetter performance on two tasks. . . . . . . . . . . . . . . . . 30ixList of FiguresFigure 1.1 AttentionRNN. Illustration of the proposed structured atten-tion network as a module for down stream task. . . . . . . . . 2Figure 1.2 Different types of attention mechanisms. Compared are (a)global and (b) local attention mechanisms explored in priorworks and proposed structured AttentionRNN architecture in(c). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Figure 4.1 Skewing operation. This makes it easier to compute convo-lutions along the diagonal. The arrows indicate dependenciesbetween attention values. To obtain the image on the right,each row of the left image is offset by one position with re-spect to its previous row. . . . . . . . . . . . . . . . . . . . . 13Figure 4.2 Block AttentionRNN for γ = 2. The input is first down-sizedusing a γ Ã γ convolutional kernel. Attention is computed onthis smaller map. . . . . . . . . . . . . . . . . . . . . . . . . 16Figure 5.1 Synthetic Dataset Samples. Example images taken from thethree synthetic datasets proposed in [29]. . . . . . . . . . . . 19Figure 5.2 Qualitative analysis of the attention masks on the MBGdataset. Layer-wise attended feature maps sampled from ARNNâ¼.The samples span all the modes in the image. Additional visu-alizations are shown in Appendix. . . . . . . . . . . . . . . . 22xFigure 5.3 Qualitative analysis of the attention masks on the MBGinvdataset. Layer-wise attended feature maps generated by dif-ferent mechanisms visualized on images from MBGinv dataset.Additional visualizations are shown in the Appendix. . . . . . 23Figure 5.4 Difference between the attention layers used in UNetd=3and UNetd=3+ARNN. Both UNetd=3 and UNetd=3+ARNNuse the same base architecture (Table 5.1) barring the atten-tion layer. (a) The attention layer used after each pooling op-eration in UNetd=3. (b) The attention layer used after eachpooling operation in UNetd=3+ARNN. The details regardingthe incorporation of UNet [27] output into ARNN is explainedin Chapter 4. . . . . . . . . . . . . . . . . . . . . . . . . . . 26Figure 5.5 Difference between CBAM and CBAM+ARNN. (a) CBAM[38]module integrated with a ResNet[8] block. (b) CBAM+ARNNreplaces the spatial attention in CBAM with ARNN. It is ap-plied similar to (a) after each ResNet[8] block. Refer to Sec-tion 5.2 for additional details. . . . . . . . . . . . . . . . . . 28Figure 5.6 Difference between MCB+ATT and MCB+ARNN. (a) MCB+ATTmodel architecture proposed by [5]. It uses a δ = 1 local con-text to compute attention over the image features. (b) MCB+ARNNreplaces the attention mechanism in MCB+ATT with ARNN.It is applied in the same location as (a) with δ = 1 context.Refer to Section 5.3 for more details. . . . . . . . . . . . . . 31Figure 5.7 Qualitative Results on ModularGAN. Attention masks gen-erated by original ModularGAN [46] and ModularGAN aug-mented with ARNN are shown. Notice that the hair mask ismore uniform for MGAN+ARNN as it is able to encode struc-tural dependencies in the attention mask. Additional resultsshown in the Appendix. . . . . . . . . . . . . . . . . . . . . . 32xiFigure 5.8 Difference between MGAN and MGAN+ARNN. (a) Thetransformer module for the ModularGAN (MGAN) architec-ture proposed by [46]. It uses a 3Ã3 local context to computeattention over the intermediate features. (b) MGAN+ARNNreplaces the attention mechanism in MGAN with ARNN. It isapplied in the same location as (a) with 3Ã 3 local context.Note that the generator and encoder modules in MGAN andMGAN+ARNN are identical. Refer to Section 5.4 for moredetails. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Figure A.1 Qualitative Analysis of Attention Masks sampled from ARNNâ¼.Layer-wise attended feature maps sampled from ARNNâ¼ for afixed image and query. The masks are able to span the differentmodes in the image. For detailed explanation see Section 5.1. 43Figure A.2 Qualitative Analysis of Attention Masks sampled from ARNNâ¼.Layer-wise attended feature maps sampled from ARNNâ¼ for afixed image and query. The masks are able to span the differentmodes in the image. For detailed explanation see Section 5.1. 44Figure A.3 Qualitative Analysis of Attention Masks sampled from ARNNâ¼.Layer-wise attended feature maps sampled from ARNNâ¼ for afixed image and query. The masks are able to span the differentmodes in the image. For detailed explanation see Section 5.1. 45Figure A.4 Qualitative Analysis of Attention Masks on MBGinv. Layer-wise attended feature maps generated by different mechanismsvisualized on images from MBGinv dataset. ARNN is ableto capture the entire number structure, whereas the other twomethods only focus on a part of the target region or on somebackground region with the same color as the target number.For detailed explanation see Section 5.1. . . . . . . . . . . . . 46xiiFigure A.5 Qualitative Analysis of Attention Masks on MBGinv. Layer-wise attended feature maps generated by different mechanismsvisualized on images from MBGinv dataset. ARNN is ableto capture the entire number structure, whereas the other twomethods only focus on a part of the target region or on somebackground region with the same color as the target number.For detailed explanation see Section 5.1. . . . . . . . . . . . . 47Figure A.6 Qualitative Analysis of Attention Masks on MBGinv. Layer-wise attended feature maps generated by different mechanismsvisualized on images from MBGinv dataset. ARNN is ableto capture the entire number structure, whereas the other twomethods only focus on a part of the target region or on somebackground region with the same color as the target number.For detailed explanation see Section 5.1. . . . . . . . . . . . . 48Figure A.7 Qualitative Results for Image Generation. Attention masksgenerated by MGAN and MGAN+ARNN are shown. Noticethat the hair mask is more uniform for MGAN+ARNN as it isable to encode structural dependencies in the attention mask.For detailed explanation see Section 5.4. . . . . . . . . . . . . 49Figure A.8 Qualitative Results for Image Generation. Attention masksgenerated by MGAN and MGAN+ARNN are shown. Noticethat the hair mask is more uniform for MGAN+ARNN as it isable to encode structural dependencies in the attention mask.For detailed explanation see Section 5.4. . . . . . . . . . . . . 50xiiiGlossaryARNN AttentionRNNBILSTM Bi-Directional Long Short-Term MemoryBRNN Block AttentionRNNCBAM Convolutional Block Attention ModuleCNN Convolutional Neural NetworkGAN Generative Adversarial NetworksLSTM Long Short-Term MemoryMCB Multimodal Compact Bilinear PoolingMGAN Modular Generative Adversarial NetworksPAN Progressive Attention NetworksRNN Recurrent Neural NetworkVAE Variational Auto-EncodersVQA Visual Question AnsweringxivAcknowledgmentsThis research was funded in part by the Vector Institute for AI, Canada CIFARAI Chair, Natural Science and Engineering Research Council of Canada (NSERC)Canada Research Chairs and Discovery Grant Program.Firstly, I would like to thank my advisor Leonid Sigal for all the guidance,support and feedback throughout the course of my Masters program. Workingunder his supervision afforded me the opportunity to explore my interests in anenvironment conducive to high quality research, and also motivated me to furtherpursue a PhD. I would also like to thank Jim Little for agreeing to be my secondreader and taking the time to provide feedback on my thesis.A huge thank you also goes out to all the people at the Computer Vision Lab,especially Suhail, Mohit, Simar and Polina, for helping me through my researchstruggles and acting as a sounding board for my ideas. I would be remiss if I didnâtalso mention the other amazing people that made my journey memorable: Anna,Clement, Alistair, Fabian, Puneet, Hayley, Nico, Giovanni, Neil, and many morefriends that I made along the way. Thank you for carrying me through my ups anddowns, and providing me an outlet away from research. Finally, I am grateful formy friends and family back home for supporting me throughout my education.xvChapter 1IntroductionIn recent years, computer vision has made tremendous progress across many com-plex recognition tasks, including image classification [18, 47], image caption-ing [4, 13, 41, 43], image generation [32, 45, 46] and Visual Question Answer-ing (VQA) [2, 5, 14, 23, 29, 33, 40, 42]. Arguably, much of this success can beattributed to the use of visual attention mechanisms which, similar to the humanperception, identify the important regions of an image. Attention mechanisms typi-cally produce a spatial mask for the given image feature tensor. In an ideal scenario,the mask is expected to have higher activation values over the features correspond-ing to the regions of interest, and lower activation values everywhere else. For tasksthat are multi-modal in nature, like VQA, a query (e.g., a question) can additionallybe used as an input to generate the mask. In such cases, the attention activation isusually a function of similarity between the corresponding encoding of the imageregion and the question in a pre-defined or learned embedding space.Existing visual attention mechanisms can be broadly characterized into twocategories: global or local; see Figure 1.2a and 1.2b respectively for illustration.Global mechanisms predict all the attention variables jointly, typically based ona dense representation of the image feature map. Such mechanisms are proneto overfitting and are only computationally feasible for low-resolution image fea-tures. Therefore, typically, these are only applied at the last convolutional layer ofa Convolutional Neural Network (CNN) [23, 48]. The local mechanisms generateattention values for each spatial attention variable independently based on corre-1Figure 1.1: AttentionRNN. Illustration of the proposed structured attentionnetwork as a module for down stream task.sponding image region [5, 29, 30] (i.e., feature column) or with the help of localcontext [30, 38, 46]. As such, local attention mechanisms can be applied at arbi-trary resolution and can be used at various places within a CNN network (e.g., in[30] authors use them before each sub-sampling layer and in [38] as part of eachresidual block). However, all the aforementioned models lack explicit structure inthe generated attention masks. This is often exhibited by lack of coherence or sharpdiscontinuities in the generated attention activation values [30].Consider a VQA model attending to regions required to answer the question,âDo the two spheres next to each other have the same color?â. Intuitively, theattention mechanisms should focus on the two spheres in question. Furthermore,attention region corresponding to one sphere should inform the attention regionestimates for the other, both in terms of shape and size. However, most traditionalattention mechanisms have no ability to encode such dependencies. Recent modu-larized architectures [1, 10] are able to address some of these issues with attentivereasoning, but they are relevant only for a narrow class of VQA problems. Suchmodels are inapplicable to scenarios involving self-attention [38] or generative ar-chitectures, where granular shape-coherent attention masks are typically needed[46].2(a) Global Attention (b) Local Attention(c) Structured AttentionFigure 1.2: Different types of attention mechanisms. Compared are (a)global and (b) local attention mechanisms explored in prior works andproposed structured AttentionRNN architecture in (c).In this thesis, we argue that these challenges can be addressed by structuredspatial attention. Such attention models can potentially encode arbitrary constraintsbetween attention variables, be it top-down structured knowledge or local/globalconsistency and dependencies. To enforce this structure, we propose a novel at-tention mechanism which we refer to as AttentionRNN (see Figure 1.2c for illus-3tration). We draw inspiration from the Diagonal Bi-Directional Long Short-TermMemory (BILSTM) architecture proposed in [35]. As such, AttentionRNN gener-ates a spatial attention mask by traversing the image diagonally, starting from acorner at the top and going to the opposite corner at the bottom. When predictingthe attention value for a particular image feature location, structure is enforced bytaking into account: (i) local image context around the corresponding image fea-ture location and, more importantly, (ii) information about previously generatedattention values.One of the key benefits of our model is that it can be used agnostically inany existing feed-forward neural network at one or multiple convolutional fea-ture levels (see Figure 1.1). To support this claim, we evaluate our method ondifferent tasks and with different backbone architectures. For VQA, we considerthe Progressive Attention Networks (PAN) [30] and Multimodal Compact BilinearPooling (MCB) [5]. For image generation, we consider the Modular GenerativeAdversarial Networks (MGAN) [46]. For image categorization, we consider theConvolutional Block Attention Module (CBAM) [38]. When we replace the exist-ing attention mechanisms in these models with our proposed AttentionRNN, weobserve higher overall performance along with better spatial attention masks.1.1 ContributionThe contributions of this work can be summarized as follows:1. We propose a novel spatial attention mechanism which explicitly encodesstructure over the spatial attention variables by sequentially predicting val-ues. As a consequence, each attention value in a spatial mask depends notonly on local image or contextual information, but also on previously pre-dicted attention values.2. We illustrate that this general attention mechanism can work with any exist-ing model that relies on, or can benefit from, spatial attention; showing itseffectiveness on a variety of different tasks and datasets.3. Through experimental evaluation, we observe improved performance andbetter attention masks on VQA, image generation and image categorization4tasks.1.2 Thesis OrganizationThe thesis is organised as follows: Chapter 2 introduces the necessary backgroundand related work. We discuss the existing methods for attention mask genera-tion, and discuss their shortcomings. Chapter 3 provides a probabilistic view to theproblem of attention generation. Here we also introduce the relevant notations usedthroughout the document. Chapter 4 explains our proposed AttentionRNN methodin detail. In addition, we also introduce a variation to the proposed approach thatallows it to scale to larger image sizes. In Chapter 5 we discuss the experimentsperformed to validate the effectiveness of our proposed approach. Finally, in Chap-ter 6 we highlight the contributions and conclude the thesis.5Chapter 2Related Work2.1 Visual AttentionVisual attention mechanisms have been widely adopted in the computer visioncommunity owing to their ability to focus on important regions in an image. Eventhough there is a large variety of methods that deploy visual attention, they can becategorized based on key properties of the underlying attention mechanisms. Forease of understanding, we segregate related research using these properties.Placement of attention in a network. Visual attention mechanisms are typicallyapplied on features extracted by a CNN. Visual attention can either be applied: (1)at the end of a CNN network, or (2) iteratively at different layers within a CNNnetwork.Applying visual attention at the end of a CNN network is the most straight-forward way of incorporating visual attention in deep models. This has led to animprovement in model performance across a variety of computer vision tasks, in-cluding image captioning [4, 41, 43], image recognition [47], VQA [23, 40, 42, 48],and visual dialog [29].On the other hand, there have been several approaches that iteratively applyvisual attention, operating over multiple CNN feature layers [11, 30, 38]. Seo etal. [30] progressively apply attention after each pooling layer of a CNN networkto accurately attend target objects of various scales and shape. Woo et al. [38]6use a similar approach, but instead apply two different types of attention â one thatattends over feature channels and the other that attends over the spatial domain.Context used to compute attention. Attention mechanisms differ on how muchinformation they use to compute the attention mask. They can be global, that is,they use all the available image context to jointly predict the values in an attentionmask [23, 41]. As an example, [23] propose an attention mechanism for VQAwhere the attention mask is computed by projecting the image features into somelatent space and then computing its similarity with the question.Attention mechanisms can also be local, wherein attention for each variable isgenerated independently or using a corresponding local image region [5, 29, 30,38, 46]. For example, [30, 38, 46] use a kà k convolutional kernel to compute aparticular attention value, allowing them to capture local information around thecorresponding location.None of the aforementioned works enforce structure over the generated atten-tion masks. Structure over the values of an image, however, has been exploited inmany autoregressive models trained to generate images. The next section brieflydescribes the relevant work in this area.2.2 Autoregressive Models for Image GenerationGenerative image modelling is a key problem in computer vision. In recent years,there has been significant work in this area [6, 15, 25, 28, 35, 44, 46]. Althoughmost use stochastic latent variable models like Variational Auto-Encoders (VAE)[15, 25] or Generative Adversarial Networks (GAN) [6, 44, 46], autoregressivemodels [28, 34, 35] provide a more tractable approach to model the joint distribu-tion over the pixels. These models leverage the inherent structure over the image,which enables them to express the joint distribution as a product of conditionaldistributions â where the value of the next pixel is dependent on all the previouslygenerated pixels. Specifically, if X = {x1,x2, . . . ,xn} is the serialized representa-tion of an image, autoregressive models assume the following:p(X) =i=nâi=1p(xi | x1, . . . ,xiâ1) (2.1)7Van et al. [35] propose a PixelRNN network that uses Long Short-Term Mem-ory (LSTM) [9] to model this sequential dependency between pixels. They alsointroduce a variant, called PixelCNN, that uses CNNs instead of LSTMs to allow forfaster computation. They later extend PixelCNN to allow for it to be conditionedon some query [34]. Finally, [28] propose further simplifications to the PixelCNNarchitecture to improve performance. Our work draws inspiration from the Pix-elRNN architecture proposed in [35]. We extend PixelRNN to model structuraldependencies within attention masks.8Chapter 3Probabilistic View of AttentionGiven an input image feature X â RhÃmÃn, our goal is to predict a spatial attentionmask A â RmÃn, where h represents the number of channels, and m and n are thenumber of rows and the columns of X respectively. Let X= {x1,1, . . . ,xm,n}, wherexi, j âRh be a column feature corresponding to the spatial location (i, j). Similarly,let A = {a1,1, . . . ,am,n}, where ai, j â R be the attention value corresponding toxi, j. Formally, we want to model the conditional distribution p(A | X). In certainproblems, we may also want to condition on other auxiliary information in additionto X, such as a question in VQA. While in this work we formulate and modelattention probabilistically, most traditional attention models directly predict theattention values, which can be regarded as a point estimate (or expected value) ofA under our formulation.Global attention mechanisms [23, 48] predict A directly from X using a fullyconnected layer. Formally,p(A | X)â f (X ;θ) (3.1)where f is a fully connected layer parameterized by θ . Although this makes noassumptions on the factorization of p(A | X), it becomes intractable as the sizeof X increases. This is mainly due to the large number of parameters in the fullyconnected layer.Local attention mechanisms [29, 30, 38, 46], on the other hand, make strong9independence assumptions on the interactions between the attention variables ai, j.Particularly, they assume each attention variable ai, j to be independent of othervariables given some local spatial context δ (xi, j). More formally, for local atten-tion mechanisms,p(A | X)âi=m, j=nâi=1, j=1p(ai, j | δ (xi, j)) (3.2)Even though such a factorization improves tractability, the strong independenceassumption often leads to attention masks that lack coherence and contain sharpdiscontinuities.Contrary to local attention mechanisms, our proposed AttentionRNN tries tocapture some of the structural dependencies between the attention variables ai, j.We assumep(A | X) =i=m, j=nâi=1, j=1p(ai, j | a<i, j, X) (3.3)âi=m, j=nâi=1, j=1p(ai, j | a<i, j, δ (xi, j)) (3.4)where a<i, j = {a1,1, . . . ,aiâ1, j} (The blue and green region in Figure 4.1). Thatis, each attention variable ai, j is now dependent on : (i) the local spatial contextδ (xi, j), and (ii) all the previous attention variables a<i, j. Note that Equation 3.3 isjust a direct application of the chain rule. Similar to local attention mechanisms,and to reduce the computation overhead, we assume that a local spatial contextδ (xi, j) is a sufficient proxy for the image features X when computing ai, j. Equation3.4 describes the final factorization we assume.One of the key challenges in estimating A based on Equation 3.4 is to effi-ciently compute the term a<i, j. A straightforward solution is to use a recurrentneural network (e.g., LSTM) to summarize the previously predicted attention val-ues a<i, j into its hidden state. This is a common approach employed in manysequence prediction methods [3, 31, 36]. However, while sequences have a welldefined ordering, image features can be traversed in multiple ways due to theirspatial nature. Naively parsing the image along its rows using an LSTM, although10provides an estimate for a<i, j, fails to correctly encode the necessary informationrequired to predict ai, j. As an example, the LSTM will tend to forget informationfrom the neighbouring variable aiâ1, j as it was processed n time steps ago.To alleviate this issue, AttentionRNN instead parses the image in a diagonalfashion, starting from a corner at the top and going to the opposite corner in thebottom. It builds upon the Diagonal BILSTM layer proposed by [35] to efficientlyperform this traversal. The next section describes our proposed AttentionRNNmechanism in detail.11Chapter 4AttentionRNNOur proposed structured attention mechanism builds upon the Diagonal BILSTMlayer proposed by [35]. We employ two LSTMs, one going from the top-left cor-ner to the bottom-right corner (L l) and the other from the top-right corner to thebottom-left corner (L r).As mentioned in Equation 3.4, for each ai, j, our objective is to estimatep(ai, j | a<i, j,δ (xi, j)). We assume that this can be approximated via a combinationof two distributions.p(ai, j|a<i, j) = Îâ©p(ai, j|a<i,< j) , p(ai, j|a<i,> j)âª(4.1)where a<i,< j is the set of attention variables to the top and left (blue region inFigure 4.1) of ai, j, a<i,> j is the set of attention variables to the top and right of ai, j(green region in Figure 4.1), and Î is some combination function. For brevity, weomit explicitly writing δ (xi, j). Equation 4.1 is further simplified by assuming thatall distributions are Gaussian.p(ai, j|a<i,< j)âN(µ li, j,Ïli, j)p(ai, j|a<i,> j)âN(µri, j,Ïri, j)p(ai, j|a<i, j)âN (µi, j,Ïi, j)(4.2)12Figure 4.1: Skewing operation. This makes it easier to compute convolu-tions along the diagonal. The arrows indicate dependencies betweenattention values. To obtain the image on the right, each row of the leftimage is offset by one position with respect to its previous row.where,(µ li, j,Ïli, j) = fl (a<i,< j) ; (µri, j,Ïri, j) = fr (a<i,> j)(µi, j,Ïi, j) = Î(µ li, j,Ïli, j,µri, j,Ïri, j) (4.3)fl and fr are fully connected layers. Our choice for the combination function Î isexplained in Section 4.1. For each ai, j, L l is trained to estimate (µ li, j,Ï li, j), andL r is trained to estimate (µri, j,Ï ri, j). We now explain the computation forL l . L ris analogous and has the same formulation.L l needs to correctly approximate a<i,< j in order to obtain a good estimate of(µ li, j,Ï li, j). As we are parsing the image diagonally, from Figure 4.1 it can be seenthat the following recursive relation holds,a<i,< j = f (a<iâ1,< j , a<i,< jâ1) (4.4)That is, for each location (i, j),L l only needs to consider two attention variables-one above and the other to the left; [35] show that this is sufficient for it to be ableto obtain information from all the previous attention variables.To make computations along the diagonal easier, similar to [35], we first skewX into a new image feature XÌ. Figure 4.1 illustrates the skewing procedure. Eachrow of X is offset by one position with respect to the previous row. XÌ is now animage feature of size hÃmà (2nâ 1). Traversing X in a diagonal fashion from13top left to bottom right is now equivalent to traversing XÌ along its columns fromleft to right. As spatial locations (iâ 1, j) and (i, jâ 1) in X are now in the samecolumn in XÌ, we can implement the recursion described in Equation 4.4 efficientlyby performing computations on an entire column of XÌ at once.Let XÌ j denote the jth column of XÌ. Also, let hÌljâ1 and cÌljâ1 respectively denotethe hidden and memory state ofL l before processing XÌ j. Both hÌljâ1 and cÌljâ1 aretensors of size tÃm, where t is the number of latent features. The new hidden andmemory state is computed as follows.[o j, f j, i j,g j] = Ï(Kh~ hÌljâ1+Kx~ XÌcj)cÌlj = f j cÌljâ1+ i j g jhÌlj = o j tanh(clj)(4.5)Here ~ represents the convolution operation and represents element-wise mul-tiplication. Kh is a 2Ã 1 convolution kernel which effectively implements therecursive relation described in Equation 4.4, and Kx is a 1Ã1 convolution kernel.Both Kh and Ku produce a tensor of size 4tÃm. XÌcj is the jth column of the skewedlocal context XÌc, which is obtained as follows.XÌc = skew(Kc~X) (4.6)where Kc is a convolutional kernel that captures a δ -size context. For computa-tional simplicity and to ensure fair comparison to other attention mechanisms, thisconvolution operation is used to realize the local spatial context δ (xi, j). However,we would like to highlight our proposed mechanism imposes no constraints on howthe spatial context δ is modelled. One can easily use more complex networks N(such as U-Net [27]) to emulate δ (xi, j). More specifically, only Eq. 4.6 of the mainpaper needs to be modified to accommodate this change.XÌc = skew(N (X)) (4.7)where N (Â·) implies passing the input through the network N . Note that anynetwork N used to realize the δ can also be used to generate a valid attention14mask A by slightly modifying the output of N (for example applying a sigmoidnon-linearity). ThereforeN can be thought of as a local attention mechanism, asit uses local image information to compute A. We would like to emphasize thatour proposed AttentionRNN mechanism is complementary to any local attentionmechanismN , as one can easily incorporateN into the AttentionRNN formula-tion as described earlier (Eq 4.7). Using AttentionRNN in conjunction toN helpscapture explicit global constraints over the attention variables, which, albeit at thecost of speed, can provide significant performance gains.For tasks that are multi-modal in nature, a query Q can additionally be usedto condition the generation of ai, j. This allows the model to generate differentattention masks for the same image features depending on Q. For example, intasks like VQA, the relevant regions of an image will depend on the question asked.The nature of Q will also dictate the encoding procedure. As an example, if Q is anatural language question, it can be encoded using a LSTM layer. Q can be easilyincorporated into AttentionRNN by concatenating it with XÌc before passing it toEquation 4.5. Equation 4.8 shows this modification.[o j, f j, i j,g j] = Ï(Kh~ hÌljâ1+Kx~[XÌcj,Q])cÌlj = f j cÌljâ1+ i j g jhÌlj = o j tanh(clj)(4.8)Let hÌl = {hÌl1, . . . , hÌl2nâ1} be the set of all hidden states obtained from L l , andhl be the set obtained by applying the reverse skewing operation on hÌl . For eachai, j, a<i,< j is then simply the (i, j) spatial element of hl . a<i,> j can be obtainedby repeating the aforementioned process forL r, which traverses X from top-rightto bottom-left. Note that this is equivalent to running L r from top-left to bottom-right after mirroring X along the column dimension, and then mirroring the outputhidden states hr again. Similar to [35], hr is shifted down by one row to preventa<i,> j from incorporating future attention values.Once a<i,< j and a<i,> j are computed (as discussed above), we can obtain theGaussian distribution for the attention variableN (µi, j,Ïi, j) by following Equation4.3. The attention ai, j could then be obtained by either sampling a value fromN (µi, j,Ïi, j) or simply by taking the expectation and setting ai, j = µi, j. For most15Figure 4.2: Block AttentionRNN for γ = 2. The input is first down-sizedusing a Î³ÃÎ³ convolutional kernel. Attention is computed on this smallermap.problems, as we will see in the experiment section, taking the expectation is goingto be most efficient and effective. However, sampling maybe useful in cases whereattention is inherently multi-modal. Focusing on different modes using coherentmasks might be more beneficial in such situations. The proposed AttentionRNNmechanism is end-to-end differentiable and can be trained using gradient descent[26].4.1 Combination FunctionThe choice of the combination function Î implicitly imposes some constraints onthe interaction between the distributions N(µ li, j,Ï li, j)and N(µri, j,Ï ri, j). Forexample, assumption of independence would dictate a simple product for Î, withresulting operations to produce (µi, j,Ïi, j) being expressed in closed form. Specifi-cally,µi, j =(Ï ri, j)2µ li, j+(Ï li, j)2µri, j(Ï li, j)2+(Ï ri, j)2Ïi, j =(Ï li, jÏ ri, j)2(Ï li, j)2+(Ï ri, j)2(4.9)However, it is clear that independence is unlikely to hold due to image correla-tions. To allow for a more flexible interaction between variables and combination16function, we instead use a fully connected layer to learn the appropriate Î for aparticular task.µi, j,Ïi, j = fcomb(µ li, j,Ïli, j,µri, j,Ïri, j)(4.10)4.2 Block AttentionRNNDue to the poor performance of LSTMs over large sequences, the AttentionRNNlayer doesnât scale well to large image feature maps. We introduce a simple mod-ification to the method described in Section 4 to alleviate this problem, which werefer to as Block AttentionRNN (BRNN).BRNN reduces the size of the input feature map X before computing the atten-tion mask. This is done by splitting X into smaller blocks, each of size Î³Ã γ . Thisis equivalent to down-sampling the original image X to Xds as follows.Xds = Kds~X (4.11)where Kds is a convolution kernel of size γ Ã γ applied with stride γ . In essence,each value in Xds now corresponds to a Î³Ã γ region in X.Instead of predicting a different attention probability for each individual spatiallocation (i, j) in X, BRNN predicts a single probability for each Î³Ã γ region. Thisis done by first computing the attention mask Ads for the down-sampled imageXds using AttentionRNN (Section 4), and then Ads is scaled up using a transposedconvolutional layer to obtain the attention mask A for the original image feature X.Figure 4.2 illustrates the BRNN procedure.BRNN essentially computes a coarse attention mask for X. Intuitively, thiscoarse attention can be used in the first few layers of a deep CNN network to identifythe key region blocks in the image. The later layers can use this coarse informationto generate a more granular attention mask.17Chapter 5ExperimentsTo show the efficacy and generality of our approach, we conduct experiments overfour different tasks: visual attribute prediction, image classification, VQA and im-age generation. We highlight that our goal is not to necessarily obtain the absolutehighest raw performance (although we do in many of our experiments), but to showimprovements from integrating AttentionRNN into existing state-of-the-art modelsacross a variety of tasks and architectures. This section describes the experimentalsetting in each case and lists the obtained results.5.1 Visual Attribute PredictionDatasets. We experiment on the synthetic MREF, MDIST and MBG datasets pro-posed in [30]. Figure 5.1 shows example images from the datasets. The images inthe datasets are created from MNIST [19] by sampling five to nine distinct digitswith different colors (green, yellow, white, red, or blue) and varying scales (be-tween 0.5 and 3.0). The datasets have images of size 100 x 100 and only differ inhow the background is generated. MREF has a black background, MDIST has ablack background with some Gaussian noise, and MBG has real images sampledfrom the SUN Database [39] as background. The training, validation and test setscontain 30,000, 10,000 and 10,000 images respectively.Experimental Setup. The performance of AttentionRNN (ARNN) is comparedagainst two local attention mechanisms proposed in [30], which are referred as18(a) MREF (b) MDIST(c) MBGFigure 5.1: Synthetic Dataset Samples. Example images taken from thethree synthetic datasets proposed in [29].¬CTX and CTX. ARNN assumes ai, j = µi, j,δ = 3, where µi, j is defined in Equa-tion 4.10. To compute the attention for a particular spatial location (i, j), CTXuses a δ = 3 local context around (i, j), whereas ¬CTX only uses the informationfrom the location (i, j). We additionally define three variants of ARNN: i) ARNNâ¼where each ai, j is sampled from N (µi, j,Ïi, j), ii) ARNNind where the combina-tion function Î assumes the input distributions are independent, and iii) ARNNâ¼indwhere Î assumes independence and ai, j is sampled. The soft attention mechanism(SAN) proposed by [41] is used as an additional baseline. Given an image, themodels are trained to predict the color of the number specified by a query. Chanceperformance is 20%.Model Architectures. Similar to [30], the base CNN architecture is composed of19SAN ¬CTX CTX ARNNconv1 (3x3@32)pool1 (2x2)â ¬CTXsigmoid CTXsigmoid ARNNsigmoidconv2 (3x3@32)pool2 (2x2)â ¬CTXsigmoid CTXsigmoid ARNNsigmoidconv3 (3x3@32)pool3 (2x2)â ¬CTXsigmoid CTXsigmoid ARNNsigmoidconv4 (3x3@32)pool4 (2x2)SAN ¬CTXso f tmax CTXso f tmax ARNNso f tmaxTable 5.1: Architectures for the models used in Section 5.1. â implies that theprevious and the next layer are directly connected. The input is passed tothe top-most layer. The computation proceeds from top to bottom.four stacks of 3Ã 3 convolutions with 32 channels followed by 2Ã 2 max pool-ing layer. SAN computes attention only on the output of the last convolutionlayer, while ¬CTX, CTX and all variants of ARNN are applied after each pool-ing layer. Table 5.1 illustrates the model architectures for each network. {¬CTX,CTX, ARNN}sigmoid refers to using sigmoid non-linearity on the generated at-tention mask before applying it to the image features. Similarly, {¬CTX, CTX,ARNN}so f tmax refers to using softmax non-linearity on the generated attentionmask. We use the same hyper-parameters and training procedure for all models,which is identical to [30].Results. Table 5.3 shows the color prediction accuracy of various models onMREF, MDIST and MBG datasets. It can be seen that ARNN and all its vari-ants clearly outperform the other baseline methods. The difference in performanceis amplified for the more noisy MBG dataset, where ARNN is 6.8% better than the20NONE ARNN BRNNconv1 (3x3@32)pool1 (2x2)â ARNNsigmoid BRNNδsigmoidARNN (described in Table 5.1)Table 5.2: Model architectures for the scalability study described in Section5.1. â implies that the previous and the next layer are directly connected.ARRN in defined in Table 5.1.Attention MREF MDIST MBGRel.RuntimeSAN [41] 83.42 80.06 58.07 1x¬CTX [30] 95.69 89.92 69.33 1.08xCTX [30] 98.00 95.37 79.00 1.10xARNNâ¼ind 98.72 96.70 83.684.73xARNNind 98.58 96.29 84.27ARNNâ¼ 98.65 96.82 83.74ARNN 98.93 96.91 85.84Table 5.3: Color prediction accuracy. Results are in % on MREF, MDISTand MBG datasets. Our AttentionRNN-based model, CNN+ARNN, out-performs all the baselines.closest baseline. ARNNind performs poorly compared to ARNN, which furthersthe reasoning of using a neural network to model Î instead of assuming indepen-dence. Similar to [30], we also evaluate the models on their sensitivity to the size ofthe target. The test set is divided into five uniform scale intervals for which modelaccuracy is computed. Table 5.4 shows the results on the MBG dataset. ARNN isrobust to scale variations and performs consistently well on small and large targets.We also test the correctness of the mask generated using the metric proposed by[21], which computes the percentage of attention values in the region of interest.For models that apply attention after each pooling layer, the masks from differentlayers are combined by upsampling and taking a product over corresponding pixel21Figure 5.2: Qualitative analysis of the attention masks on the MBGdataset. Layer-wise attended feature maps sampled from ARNNâ¼. Thesamples span all the modes in the image. Additional visualizations areshown in Appendix.values. The results are shown for the MBG dataset in Table 5.4. ARNN is ableto more accurately attend to the correct regions, which is evident from the highcorrectness score.From Tables 5.3 and 5.4, it can be seen that ARNNâ¼ provides no significantadvantage over its deterministic counterpart. This can be attributed to the datasets22Figure 5.3: Qualitative analysis of the attention masks on the MBGinvdataset. Layer-wise attended feature maps generated by different mech-anisms visualized on images from MBGinv dataset. Additional visual-izations are shown in the Appendix.encouraging point estimates, as each query can only have one correct answer. Asa consequence, for each ai, j, Ïi, j was observed to underestimate variance. How-ever, in situations where a query can have multiple correct answers, ARNNâ¼ cangenerate diverse attention masks. To corroborate this claim, we test the pre-trainedARNNâ¼ on images that are similar to the MBG dataset but have the same digit in23Attention Corr.Scale0.5-1.0 1.0-1.5 1.5-2.0 2.0-2.5 2.5-3.0SAN [41] 0.15 53.05 74.85 72.18 59.52 54.91¬CTX [30] 0.28 68.20 76.37 73.30 61.60 57.28CTX [30] 0.31 77.39 87.13 84.96 75.59 63.72ARNNâ¼ind 0.36 82.23 89.41 86.46 84.52 81.35ARNNind 0.34 82.89 89.47 88.34 84.22 80.00ARNNâ¼ 0.39 82.23 89.41 86.46 84.52 81.35ARNN 0.42 84.45 91.40 86.84 88.39 82.37Table 5.4: Mask Correctness and Scale experiment on MBG. The âCorr.âcolumn lists the mask correctness metric proposed by [21]. The âScaleâcolumn shows the color prediction accuracy in % for different scales.multiple colors. Figure 5.2 shows the individual layer attended feature maps forthree different samples from ARNNâ¼ for a fixed image and query. For the queryâ9â, ARNNâ¼ is able to identify the three modes. Note that since Ïi, jâs were under-estimated due to aforementioned reasons, they were scaled up before generatingthe samples. Despite being underestimated Ïi, jâs still capture crucial information.The rightmost column of Table 5.3 compares the relative training run-time ofour proposed AttentionRNN method to other attention mechanisms, which is com-puted as a ratio to the fastest attention mechanism. In the case of AttentionRNN,the sequential prediction of the attention variables impacts the training and infer-ence speeds. As the image features are larger in the initial layers of the network(Table 5.1) compared to the later ones, they contribute to a large portion of thecomputation cost. One way to improve the run-time is to use Block AttentionRNN(Section 4.2) in the initial layers.5.1.1 Using Expressive Local Spatial Context δChapter 4 highlights the ability of our proposed mechanism to leverage the richerimage context captured by methods like U-Net [27] in its formulation by appro-priately defining δ (Eq. 4.7). To further emphasize the benefits of modelling ex-plicit structure using ARNN, Table 5.5 compares the performance of using ARNNin conjunction with U-Net [27] on the MBG dataset. Please refer to Section 5.1for details on the dataset. All models share the same base network architecture24ScaleTotal 0.5-1.0 1.0-1.5 1.5-2.0 2.0-2.5 2.5-3.0UNetd=2[27] 85.86 84.25 91.28 89.84 85.41 86.44UNetd=3 [27] 86.64 84.52 92.98 91.35 91.96 86.44UNetd=2+ARNN 91.73 90.47 95.78 93.98 92.26 93.89UNetd=3+ARNN 92.29 90.86 96.14 94.54 97.02 94.91Table 5.5: Color prediction accuracy on MBG dataset. Results are in %.Using ARNN is conjunction to local attention mechanisms significantlyimproves performance.(Table 5.1) barring the type of attention used. UNetd=i implies the use of a depthi U-Net architecture [27] to compute attention at each layer. Figure 5.4 highlightsthe differences between the attention layers used in UNetd=i and UNetd=i+ARNNfor d = 3. The differences for d = 2 are analogous. It can be seen that the use ofARNN to model structural dependencies significantly improves the performance(â¼ 5%).5.1.2 Inverse Attribute PredictionFigure 5.2 leads to an interesting observation regarding the nature of the task. Eventhough ARNNâ¼ is able to identify the correct number, it only needs to focus on atiny part of the target region to be able to accurately classify the color. To fur-ther demonstrate the ARNNâs ability to model longer dependencies, we test theperformance of ARNN, CTX and ¬CTX on the MBGinv dataset, which definesthe inverse attribute prediction problem - given a color identify the number corre-sponding to that color. The base CNN architecture is identical to the one used inthe previous experiment. ARNN, CTX and ¬CTX achieve an accuracy of 72.77%,66.37% and 40.15% and a correctness score [21] of 0.39, 0.24 and 0.20 respec-tively. Figure 5.3 shows layer-wise attended feature maps for the three models.ARNN is able to capture the entire number structure, whereas the other two meth-ods only focus on a part of the target region. Even though CTX uses some localcontext to compute the attention masks, it fails to identify the complete structure forthe number â0â. A plausible reason for this is that a 3Ã3 local context is too smallto capture the entire target region. As a consequence, the attention mask is com-25(a) Attention layer used in UNetd=3(b) Attention layer used in UNetd=3+ARNNFigure 5.4: Difference between the attention layers used in UNetd=3 andUNetd=3+ARNN. Both UNetd=3 and UNetd=3+ARNN use the samebase architecture (Table 5.1) barring the attention layer. (a) The atten-tion layer used after each pooling operation in UNetd=3. (b) The atten-tion layer used after each pooling operation in UNetd=3+ARNN. Thedetails regarding the incorporation of UNet [27] output into ARNN isexplained in Chapter 4.puted in patches. CTX maintains no information about the previously computedattention values, and therefore is unable to assign correlated attention scores for allthe different target region patches. ARNN, on the other hand, captures constraintsbetween attention variables, making it much more effective in this situation.26ScaleTotal 0.5-1.0 1.0-1.5 1.5-2.0 2.0-2.5 2.5-3.0NONE 91.43 85.63 92.57 94.96 94.77 93.59ARNN 91.09 84.89 92.25 94.24 94.70 94.52BRNNγ=3 91.67 85.97 93.46 94.81 94.35 93.68BRNNγ=2 92.68 88.10 94.23 95.32 94.80 94.01Table 5.6: Block AttentionRNN. Ablation results on MBGb dataset. Atten-tionRNN (ARNN) and Block AttentionRNN (BRNN) with block sizesof 2 and 3 are compared.5.1.3 Scalability of ARNNThe results shown in Table 5.3 correspond to models trained on 100Ã 100 inputimages, where the first attention layer is applied on an image feature of size 50Ã50.To analyze the performance of ARNN on comparatively larger image features, wecreate a new dataset of 224Ã 224 images which we refer to as MBGb. The datageneration process for MBGb is identical to MBG. We perform an ablation studyanalyzing the effect of using Block AttentionRNN (BRNN) (Section 4.2) insteadof ARNN on larger image features. For the base architecture, the ARNN modelfrom the previous experiment (Table 5.1 is augmented with an additional stack of3Ã3 convolution layer followed by a 2Ã2 max pooling layer. Table 5.2 illustratesthe differences between the models used in this experiment. Table 5.6 shows thecolor prediction accuracy on different scale intervals for the MBGb dataset. Asthe first attention layer is now applied on a feature map of size 112Ã112, ARNNperforms worse than the case when no attention (NONE) is applied due to the poortractability of LSTMs over large sequences. BRNNγ=2, on the other hand, is ableto perform better as it reduces the image feature size before applying attention.However, there is a considerable difference in the performance of BRNN whenγ = 2 and γ = 3. When γ = 3, BRNN applies a 3Ã 3 convolution with stride 3.This aggressive size reduction causes loss of information.27(a) CBAM module(b) CBAM+ARNN moduleFigure 5.5: Difference between CBAM and CBAM+ARNN. (a)CBAM[38] module integrated with a ResNet[8] block. (b)CBAM+ARNN replaces the spatial attention in CBAM with ARNN. Itis applied similar to (a) after each ResNet[8] block. Refer to Section5.2 for additional details.5.2 Image ClassificationDataset. We use the CIFAR-100 [17] to verify the performance of AttentionRNNon the task of image classification. The dataset consists of 60,000 32Ã32 imagesfrom 100 classes. The training/test set contain 50,000/10,000 images.Experimental Setup. We augment ARNN to the convolution block attention mod-ule (CBAM) proposed by [38]. For a given feature map, CBAM computes twodifferent types of attentions: 1) channel attention that exploits the inter-channeldependencies in a feature map, and 2) spatial attention that uses local context toidentify relationships in the spatial domain. Figure 5.5a shows the CBAM mod-ule integrated with a ResNet [8] block. We replace only the spatial attention inCBAM with ARNN. This modified module is referred to as CBAM+ARNN. Fig-ure 5.5b better illustrates this modification. ResNet18 [8] is used as the base model28Top-1Error (%)Top-5Error (%)Rel.RuntimeResNet18 [8] 25.56 6.87 1xResNet18 + CBAM [38] 25.07 6.57 1.43xResNet18 + CBAM + ARNN 24.18 6.42 4.81xTable 5.7: Performance on Image Classification. The Top-1 and Top-5 er-ror % are shown for all the models. The ARNN based model outperformsall other baselines.Yes/No Number Other TotalRel.RuntimeMCB [5] 76.06 35.32 43.87 54.84 1xMCB+ATT [5] 76.12 35.84 47.84 56.89 1.66xMCB+ARNN 77.13 36.75 48.23 57.58 2.46xTable 5.8: Performance on VQA. In % accuracy.for our experiments. ResNet18+CBAM is the model obtained by using CBAMin the Resnet18 model, as described in [38]. Resnet18+CBAM+ARNN is definedanalogously. We use a local context of 3Ã 3 to compute the spatial attention forboth CBAM and CBAM+ARNN. In addition, the hyper-parameters and trainingprocedure is used for both CBAM and CBAM+ARNN, which is identical to [38].Results. Top-1 and top-5 error is used to evaluate the performance of the models.The results are summarized in Table 5.7. CBAM+ARNN provides an improve-ment of 0.89% on top-1 error over the closest baseline. Note that this gain, thoughseems marginal, is larger than what CBAM obtains over ResNet18 with no atten-tion (0.49% on top-1 error).5.3 Visual Question AnsweringDataset. We evaluate the performance of ARNN on the task of VQA [2]. Theexperiments are done on the VQA 2.0 dataset [7], which contains images fromMSCOCO [20] and corresponding questions. As the test set is not publicly avail-able, we evaluate performance on the validation set.29Hair Gender SmileRel.RuntimeMGAN [46] 2.5 3.2 12.6 1xMGAN+ARNN 3.0 1.4 11.4 1.96xTable 5.9: Performance on Image Generation. ResNet18 [8] ClassificationErrors (%) for each attribute transformation. ARNN achieves better per-formance on two tasks.Experimental Setup. We use the Multimodal Compact Bilinear Pooling with At-tention (MCB+ATT) architecture proposed by [5] as a baseline for our experiment.To compute attention, MCB+ATT uses a local context of δ = 1 over the featuresobtained after using the compact bilinear pooling operation. Figure 5.6a illus-trates the architecture for MCB+ATT. We augment ARNN to the MCB architec-ture proposed by [5], which is referred to as MCB+ARNN. For fair comparison,MCB+ARNN also uses a δ = 1 local context to compute attention. Figure 5.6bbetter illustrates this modification. Note that even though MCB doesnât give state-of-the-art performance on this task, it is a competitive baseline that allows for easyARNN integration. We use the same hyper-parameters and training procedure forMCB, MCB+ATT and MCB+ARNN, which is identical to [5].Results. The models are evaluated using the accuracy measure defined in [2]. Theresults are summarized in Table 5.8. MCB+ARNN achieves a 0.69% improvementover the closest baseline 1. We believe this marginal improvement is because all themodels, for each spatial location (i, j), use no context from neighbouring locations(as δ = 1).5.4 Image GenerationDataset. We analyze the effect of using ARNN on the task of image generation.Experiments are performed on the CelebA dataset [22], which contains 202,599face images of celebrities, with 40 binary attributes. The data pre-processing is1Reported performance for the baselines are from our PyTorch [24] re-implementation of [5],and are directly comparable to MCB+ARNN. We note that [7] reports slightly better performance(byâ¼2%), which we attribute to differences in frameworks (PyTorch [24] vs Caffe [12]) and possibleuse of additional features (as noted in Table 4 of [5]).30(a) MCB+ATT(b) MCB+ARNNFigure 5.6: Difference between MCB+ATT and MCB+ARNN. (a)MCB+ATT model architecture proposed by [5]. It uses a δ = 1 localcontext to compute attention over the image features. (b) MCB+ARNNreplaces the attention mechanism in MCB+ATT with ARNN. It is ap-plied in the same location as (a) with δ = 1 context. Refer to Section5.3 for more details.identical to [46]. The models are evaluated on three attributes: hair color = {black,blond, brown}, gender = {male, female}, and smile = {smile, nosmile}.Experimental Setup. Please refer to Section 4.4 of the main paper for task defini-tions. We compare ARNN to a local attention mechanism used in the ModularGAN(MGAN) framework [46]. MGAN consists of three modules: 1) encoder modulethat encodes an input image into an intermediate feature representation, 2) genera-31Figure 5.7: Qualitative Results on ModularGAN. Attention masks gener-ated by original ModularGAN [46] and ModularGAN augmented withARNN are shown. Notice that the hair mask is more uniform forMGAN+ARNN as it is able to encode structural dependencies in theattention mask. Additional results shown in the Appendix.tor module that generates an image given an intermediate feature representation asinput, and 3) transformer module that transforms a given intermediate representa-tion to a new intermediate representation according to some input condition. Thetransformer module uses a 3Ã 3 local context to compute attention over the fea-ture representations. Figure 5.8a illustrates the transformer module proposed by[46]. We define MGAN+ARNN as the network obtained by replacing this localattention mechanism in the transformer module with ARNN. Note that the gener-ator and encoder modules are unchanged. MGAN+ARNN also uses a 3Ã 3 localcontext to compute attention. Figure 5.8b better illustrates this modification to the32(a) Transformer module for MGAN(b) Transformer module for MGAN+ARNNFigure 5.8: Difference between MGAN and MGAN+ARNN. (a) The trans-former module for the ModularGAN (MGAN) architecture proposed by[46]. It uses a 3Ã3 local context to compute attention over the interme-diate features. (b) MGAN+ARNN replaces the attention mechanism inMGAN with ARNN. It is applied in the same location as (a) with 3Ã3local context. Note that the generator and encoder modules in MGANand MGAN+ARNN are identical. Refer to Section 5.4 for more details.transformer module. We use the same hyper-parameters and training procedurefor both MGAN and MGAN+ARNN, which is identical to [46]. The models aretrained to transform an image given an attribute.Results. To evaluate the performance of the models, similar to [46], we train aResNet18 [8] model that classifies the hair color, facial expression and gender onthe CelebA dataset. The trained classifier achieves an accuracy of 93.9%, 99.0%33and 93.7% on hair color, gender and smile respectively. For each transformation,we pass the generated images through this classifier and compute the classificationerror (shown in Table 5.9). MGAN+ARNN outperforms the baseline on all cate-gories except hair color. To analyse this further, we look at the attention masksgenerated for the hair color transformation by both models. As shown in Fig-ure 5.7, we observe that the attention masks generated by MGAN lack coherenceover the target region due to discontinuities. MGAN+ARNN, though has a slightlyhigher classification error, generates uniform activation values over the target re-gion by encoding structural dependencies.34Chapter 6ConclusionIn this work, we developed a novel structured spatial attention mechanism whichis end-to-end trainable and can be integrated with any feed-forward convolutionalneural network. The proposed AttentionRNN layer explicitly enforces structureover the spatial attention variables by sequentially predicting attention values in thespatial mask. As a result, each attention value not only depends on local image orcontext information, but also on previously predicted attention values. Experimentsshow consistent quantitative and qualitative improvements on a large variety ofrecognition tasks, datasets and backbone architectures.The proposed framework, however, isnât without its limitations. Our choice ofmaintaining a recurrent state for attention computation, though allows for effec-tively modelling explicit structural dependencies, impacts the training and infer-ence speeds. Also, when factorizing the joint distribution p(A | X) into a productof easier-to-compute conditional distributions, we assume a particular ordering tothe set of attention variables. The assumed ordering, however, might not be ideal incertain situations where relative local dependencies exist between different objectsin the image.One possible direction for exploration is using Graph Neural Networks (GNNs)[16] to realise the joint distribution p(A |X) instead of a Recurrent Neural Network(RNN). For the problem of attention generation, GNNs can be used to iterativelyrefine the attention estimate for a particular variable by using information from itsneighbours. Contrary to the sequential prediction of attention variables, iterative35refinement would help speed up the attention mask computation as all attentionvariables are predicted at once.As the image is processed diagonally from a corner on the top to the oppo-site corner at the bottom, the current formulation necessitates the use of a RNNto capture long range dependencies. Non-Local Neural Networks [37], however,provide an alternate approach to model these dependencies. Unlike traditional ap-proaches that process the image one local neighbourhood at a time, Non-LocalNeural Networks compute the activation at a particular position as a weighted sumof the features at all positions in the input feature map. As long-range dependen-cies can now be computed directly via interactions between any two positions, itmight make the use of RNNs obsolete, leading to considerable speedups.36Bibliography[1] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks.In IEEE Conference on Computer Vision and Pattern Recognition, 2016. âpage 2[2] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, andD. Parikh. Vqa: Visual question answering. In IEEE InternationalConference on Computer Vision, pages 2425â2433, 2015. â pages 1, 29, 30[3] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointlylearning to align and translate. In International Conference on LearningRepresentations, 2015. â page 10[4] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T.-S. Chua.Sca-cnn: Spatial and channel-wise attention in convolutional networks forimage captioning. In IEEE Conference on Computer Vision and PatternRecognition, pages 6298â6306. IEEE, 2017. â pages 1, 6[5] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach.Multimodal compact bilinear pooling for visual question answering andvisual grounding. In Conference on Empirical Methods in Natural LanguageProcessing, pages 457â468. ACL, 2016. â pages xi, 1, 2, 4, 7, 29, 30, 31[6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. InAdvances in Neural Information Processing Systems, pages 2672â2680,2014. â page 7[7] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the vin vqa matter: Elevating the role of image understanding in visual questionanswering. In IEEE Conference on Computer Vision and PatternRecognition, pages 6325â6334, 2017. â pages 29, 3037[8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for imagerecognition. In IEEE Conference on Computer Vision and PatternRecognition, pages 770â778, 2016. â pages ix, xi, 28, 29, 30, 33[9] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neuralcomputation, 9(8):1735â1780, 1997. â page 8[10] R. Hu, J. Andreas, K. Saenko, and T. Darrell. Explainable neuralcomputation via stack neural module networks. In European Conference onComputer Vision, 2018. â page 2[11] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformernetworks. In Advances in Neural Information Processing Systems, pages2017â2025, 2015. â page 6[12] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fastfeature embedding. arXiv preprint arXiv:1408.5093, 2014. â page 30[13] J. Johnson, A. Karpathy, and L. Fei-Fei. Densecap: Fully convolutionallocalization networks for dense captioning. In IEEE Conference onComputer Vision and Pattern Recognition, 2016. â page 1[14] J. Johnson, B. Hariharan, L. van der Maaten, F.-F. Li, L. Zitnick, andR. Girshick. Clevr: A diagnostic dataset for compositional language andelementary visual reasoning. In IEEE Conference on Computer Vision andPattern Recognition, 2017. â page 1[15] D. P. Kingma and M. Welling. Auto-encoding variational bayes. stat, 1050:10, 2014. â page 7[16] T. N. Kipf and M. Welling. Semi-supervised classification with graphconvolutional networks. ICLR, 2017. â page 35[17] A. Krizhevsky. Learning multiple layers of features from tiny images.Technical report, Citeseer, 2009. â page 28[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification withdeep convolutional neural networks. In Advances in Neural InformationProcessing Systems, pages 1097â1105, 2012. â page 1[19] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learningapplied to document recognition. Proceedings of the IEEE, 86(11):2278â2324, 1998. â page 1838[20] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dolla´r,and C. L. Zitnick. Microsoft coco: Common objects in context. In Europeanconference on computer vision, pages 740â755, 2014. â page 29[21] C. Liu, J. Mao, F. Sha, and A. L. Yuille. Attention correctness in neuralimage captioning. In AAAI, pages 4176â4182, 2017. â pagesviii, 21, 24, 25[22] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in thewild. In IEEE International Conference on Computer Vision, 2015. â page30[23] J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-imageco-attention for visual question answering. In Advances In NeuralInformation Processing Systems, pages 289â297, 2016. â pages 1, 6, 7, 9[24] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation inPyTorch. In NIPS Autodiff Workshop, 2017. â page 30[25] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagationand approximate inference in deep generative models. InternationalConference on Machine Learning, 2014. â page 7[26] H. Robbins and S. Monro. A stochastic approximation method. The annalsof mathematical statistics, pages 400â407, 1951. â page 16[27] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks forbiomedical image segmentation. In International Conference on Medicalimage computing and computer-assisted intervention, pages 234â241.Springer, 2015. â pages xi, 14, 24, 25, 26[28] T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma. Pixelcnn++:Improving the pixelcnn with discretized logistic mixture likelihood andother modifications. ICLR, 2017. â pages 7, 8[29] P. H. Seo, A. Lehrmann, B. Han, and L. Sigal. Visual reference resolutionusing attention memory for visual dialog. In Advances in Neural InformationProcessing Systems, pages 3719â3729, 2017. â pages x, 1, 2, 6, 7, 9, 19[30] P. H. Seo, Z. Lin, S. Cohen, X. Shen, and B. Han. Progressive attentionnetworks for visual attribute prediction. British Machine Vision Conference,2018. â pages 2, 4, 6, 7, 9, 18, 19, 20, 21, 2439[31] S. Shankar and S. Sarawagi. Posterior attention models for sequence tosequence learning. In International Conference on LearningRepresentations, 2019. â page 10[32] Y. Tang, N. Srivastava, and R. R. Salakhutdinov. Learning generative modelswith visual attention. In Advances in Neural Information ProcessingSystems, 2014. â page 1[33] T. Tommas, A. Mallya, B. Plummer, S. Lazebnik, A. Berg, and T. Berg.Solving visual madlibs with multiple cues. In British Machine VisionConference, 2016. â page 1[34] A. van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al.Conditional image generation with pixelcnn decoders. In Advances inNeural Information Processing Systems, pages 4790â4798, 2016. â pages7, 8[35] A. Van Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neuralnetworks. In IEEE International Conference on Machine Learning, pages1747â1756, 2016. â pages 4, 7, 8, 11, 12, 13, 15[36] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, andK. Saenko. Sequence to sequence-video to text. In Proceedings of the IEEEinternational conference on computer vision, pages 4534â4542, 2015. âpage 10[37] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. InProceedings of the IEEE Conference on Computer Vision and PatternRecognition, pages 7794â7803, 2018. â page 36[38] S. Woo, J. Park, J.-Y. Lee, and I. So Kweon. Cbam: Convolutional blockattention module. In European Conference on Computer Vision, pages 3â19,2018. â pages xi, 2, 4, 6, 7, 9, 28, 29[39] J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva. Sun database:Exploring a large collection of scene categories. International Journal ofComputer Vision, 119(1):3â22, 2016. â page 18[40] H. Xu and K. Saenko. Ask, attend and answer: Exploring question-guidedspatial attention for visual question answering. In European Conference onComputer Vision, pages 451â466, 2016. â pages 1, 640[41] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel,and Y. Bengio. Show, attend and tell: Neural image caption generation withvisual attention. In International Conference on Machine Learning, pages2048â2057, 2015. â pages 1, 6, 7, 19, 21, 24[42] Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networksfor image question answering. In IEEE Conference on Computer Vision andPattern Recognition, pages 21â29, 2016. â pages 1, 6[43] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo. Image captioning withsemantic attention. In IEEE conference on computer vision and patternrecognition, pages 4651â4659, 2016. â pages 1, 6[44] H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas.Stackgan: Text to photo-realistic image synthesis with stacked generativeadversarial networks. IEEE International Conference on Computer Vision,2017. â page 7[45] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attentiongenerative adversarial networks. In arXiv preprint arXiv:1805.08318, 2018.â page 1[46] B. Zhao, B. Chang, Z. Jie, and L. Sigal. Modular generative adversarialnetworks. European Conference on Computer Vision, 2018. â pagesxi, xii, 1, 2, 4, 7, 9, 30, 31, 32, 33[47] H. Zheng, J. Fu, T. Mei, and J. Luo. Learning multi-attention convolutionalneural network for fine-grained image recognition. In IEEE InternationalConference on Computer Vision, volume 6, 2017. â pages 1, 6[48] Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7w: Groundedquestion answering in images. In IEEE Conference on Computer Vision andPattern Recognition, pages 4995â5004, 2016. â pages 1, 6, 941AppendixSupporting MaterialsA.1 Additional VisualizationsA.1.1 Visual Attribute PredictionPlease refer to Section 5.1 for task definition. Figures A.1 - A.3 show the individuallayer attended feature maps for three different samples from ARNNâ¼ for a fixedimage and query. It can be seen that ARNNâ¼ is able to identify the different modesin each of the images.42Figure A.1: Qualitative Analysis of Attention Masks sampled fromARNNâ¼. Layer-wise attended feature maps sampled from ARNNâ¼for a fixed image and query. The masks are able to span the differentmodes in the image. For detailed explanation see Section 5.1.43Figure A.2: Qualitative Analysis of Attention Masks sampled fromARNNâ¼. Layer-wise attended feature maps sampled from ARNNâ¼for a fixed image and query. The masks are able to span the differentmodes in the image. For detailed explanation see Section 5.1.44Figure A.3: Qualitative Analysis of Attention Masks sampled fromARNNâ¼. Layer-wise attended feature maps sampled from ARNNâ¼for a fixed image and query. The masks are able to span the differentmodes in the image. For detailed explanation see Section 5.1.A.1.2 Inverse Attribute PredictionPlease refer to Section 5.1 for task definition. Figures A.4 - A.6 show the individuallayer attended feature maps comparing the different attention mechanisms on theMBGinv dataset. It can be seen that ARNN captures the entire number structure,whereas the other two methods only focus on a part of the target region or onsome background region with the same color as the number, leading to incorrect45predictions.Figure A.4: Qualitative Analysis of Attention Masks on MBGinv. Layer-wise attended feature maps generated by different mechanisms visu-alized on images from MBGinv dataset. ARNN is able to capture theentire number structure, whereas the other two methods only focus ona part of the target region or on some background region with the samecolor as the target number. For detailed explanation see Section 5.1.46Figure A.5: Qualitative Analysis of Attention Masks on MBGinv. Layer-wise attended feature maps generated by different mechanisms visu-alized on images from MBGinv dataset. ARNN is able to capture theentire number structure, whereas the other two methods only focus ona part of the target region or on some background region with the samecolor as the target number. For detailed explanation see Section 5.1.47Figure A.6: Qualitative Analysis of Attention Masks on MBGinv. Layer-wise attended feature maps generated by different mechanisms visu-alized on images from MBGinv dataset. ARNN is able to capture theentire number structure, whereas the other two methods only focus ona part of the target region or on some background region with the samecolor as the target number. For detailed explanation see Section 5.1.A.1.3 Image GenerationPlease refer to Section 5.4 for task definition. Figures A.7 and A.8 show the at-tention masks generated by MGAN and MGAN+ARNN for the task of hair colortransformation. MGAN+ARNN encodes structural dependencies in the attention48values, which is evident from the more uniform and continuous attention masks.MGAN, on the other hand, has sharp discontinuities which, in some cases, leads toless accurate hair color transformations.Figure A.7: Qualitative Results for Image Generation. Attention masksgenerated by MGAN and MGAN+ARNN are shown. Notice that thehair mask is more uniform for MGAN+ARNN as it is able to encodestructural dependencies in the attention mask. For detailed explanationsee Section 5.4.49Figure A.8: Qualitative Results for Image Generation. Attention masksgenerated by MGAN and MGAN+ARNN are shown. Notice that thehair mask is more uniform for MGAN+ARNN as it is able to encodestructural dependencies in the attention mask. For detailed explanationsee Section 5.4.50
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Enforcing structure in visual attention
Text