Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Abdelaziz, Omar
Master of Science - MSc
University of British Columbia
Single object tracking is a crucial yet challenging task in computer vision. In this task, a model is given the appearance of an arbitrary object in a sequence of frames. The model is required to track the object in all sequence frames. Traditionally, discriminative correlation filters and Siamese convolutional networks have dominated the field but with limitations when tracking challenging objects such as occluded and long-term tracked objects. However, many techniques backed by recent advancements in machine learning theory, such as masked image modelling, generative adversarial networks, variational autoencoders, and diffusion models, are being increasingly leveraged in single object tracking to provide more customized representations that capture the unique characteristics of the target object. This thesis explores recent advancements in single object tracking by providing three novel contributions. The first contribution is that we comprehensively survey emerging trends in single object tracking, including sequence models, generative models, and self-supervised learning, proposing a novel categorization for single object tracking methods. We further compare existing approaches and identify promising directions for future research. The second contribution focuses on the critical role of bounding box regression within single object tracking. We argue that leveraging the receptive field of convolutional networks is essential for accurate object localization. We introduce two novel bounding box regression networks that achieve superior performance on benchmark datasets compared to existing methods. The third contribution addresses the persistent performance gap between training and test data in single object tracking, particularly for transformer-based trackers. It proposes the Deformable Masking Tracker (DMTrack), which injects deformable convolution within the Vision Transformer (ViT) architecture. DMTrack improves the robustness of attentional features, leading to a significant performance boost of up to 2% across seven tracking benchmarks. Overall, this thesis contributes to the field of single object tracking by offering a comprehensive analysis of novel techniques, demonstrating the importance of bounding box regression and proposing a novel solution to bridge the data performance gap. The findings in this thesis pave the way for further advancements in robust and generalizable single object tracking algorithms.
Beyond Traditional Single ObjectTrackingbyOmar AbdelazizB.Sc., Hons., Ain Shams University, 2021A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe College of Graduate Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)June 2024© Omar Abdelaziz 2024The following individuals certify that they have read, and recommend tothe College of Graduate Studies for acceptance, a thesis/dissertation entitled:Beyond Traditional Single Object Trackingsubmitted by Omar Abdelaziz in partial fulfillment of the requirements ofthe degree of the degree of Master of ScienceMohamed S. Shehata, The University of British Columbia, Okanagan CampusSupervisorMohamed Abdelpakey Mohamed, The University of British Columbia, Okanagan CampusCo-SupervisorPourang Irani, The University of British Columbia, Okanagan CampusSupervisory Committee MemberShan Du, The University of British Columbia, Okanagan CampusSupervisory Committee MemberMegan Smith, The University of British Columbia, Okanagan CampusUniversity ExamineriiAbstractSingle object tracking is a crucial yet challenging task in computer vision.In this task, a model is given the appearance of an arbitrary object in a se-quence of frames. The model is required to track the object in all sequenceframes. Traditionally, discriminative correlation filters and Siamese convolu-tional networks have dominated the field but with limitations when trackingchallenging objects such as occluded and long-term tracked objects. How-ever, many techniques backed by recent advancements in machine learningtheory, such as masked image modelling, generative adversarial networks,variational autoencoders, and diffusion models, are being increasingly lever-aged in single object tracking to provide more customized representationsthat capture the unique characteristics of the target object.This thesis explores recent advancements in single object tracking byproviding three novel contributions. The first contribution is that we com-prehensively survey emerging trends in single object tracking, including se-quence models, generative models, and self-supervised learning, proposing anovel categorization for single object tracking methods. We further compareexisting approaches and identify promising directions for future research.The second contribution focuses on the critical role of bounding box regres-sion within single object tracking. We argue that leveraging the receptivefield of convolutional networks is essential for accurate object localization.We introduce two novel bounding box regression networks that achieve supe-rior performance on benchmark datasets compared to existing methods. Thethird contribution addresses the persistent performance gap between train-ing and test data in single object tracking, particularly for transformer-basedtrackers. It proposes the Deformable Masking Tracker (DMTrack), whichinjects deformable convolution within the Vision Transformer (ViT) archi-tecture. DMTrack improves the robustness of attentional features, leadingto a significant performance boost of up to 2% across seven tracking bench-marks. Overall, this thesis contributes to the field of single object trackingby offering a comprehensive analysis of novel techniques, demonstrating theimportance of bounding box regression and proposing a novel solution tobridge the data performance gap. The findings in this thesis pave the wayiiifor further advancements in robust and generalizable single object trackingalgorithms.ivLay SummarySingle-object tracking is a hard problem by nature. Although current single-object trackers are sophisticated enough to capture complex interactionsbetween the target object and the scene, they struggle with scenes with in-terleaved challenges, such as object deformation, occlusion, and distractorobject presence. Moreover, current single object tracking techniques facea significant gap in performance between training and testing data. Thisis mainly due to overfitting representations given in the training data. Inthis work, we survey the most recent techniques in single object tracking,introducing a new taxonomy for trackers. We investigate the effect of thebounding box regression head on the performance of a vision transformer-based tracker and propose two bounding box regression heads that enhancethe performance of ViT trackers. Finally, inspired by the works presentedin the survey, we propose a new tracker, DMTrack, that narrows the gapbetween training and testing performance and achieves state-of-the-art per-formance on tracking benchmarks.vPrefaceChapters 3 through 5 of this thesis contain content from work submitted toscientific journals and conferences in which Omar Abdelaziz was the primaryauthor. Other co-authors for published or submitted work include MohamedH. Abdelpakey and Mohamed S. Shehata.Chapter 3 contains a version of content from ”Beyond Traditional Sin-gle Object Tracking: A Survey,” which is submitted to the InternationalJournal of Machine Learning and Cybernetics. Omar Abdelaziz was thelead investigator responsible for surveying and summarizing non-traditionalsingle-object tracking techniques. Mohamed H. Abdelpakey was responsiblefor the manuscript review. Mohamed S. Shehata supervised all research andtechnical contributions and reviewed the final manuscript before submission.Chapter 4 contains a version of content from ”A Novel Bounding BoxRegression Method for Single Object Tracking,” which is submitted to theInternational Conference on Intelligent Systems, Blockchain, and Commu-nication Technologies. Omar Abdelaziz led all research for this work, whichinvolved technique formulation, technique implementation, training, testing,and manuscript composition. Mohamed S. Shehata oversaw the supervisionof the research and reviewed the final manuscript before submission.Chapter 5 contains content from ”DMTrack: Learning Deformable MaskedVisual Representations for Single Object Tracking,” which is submitted tothe IEEE Transactions on Artificial Intelligence. Omar Abdelaziz was thelead investigator responsible for formulating, training, and testing the pro-posed technique. Mohamed S. Shehata supervised all research and manuscriptcomposition.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiGlossary of Notation . . . . . . . . . . . . . . . . . . . . . . . . . . xviAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Research Motivation . . . . . . . . . . . . . . . . . . . . . . . 21.3 Research Objective . . . . . . . . . . . . . . . . . . . . . . . 31.4 Contributions Summary . . . . . . . . . . . . . . . . . . . . . 31.5 Published and Submitted Works . . . . . . . . . . . . . . . . 41.6 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1 Single Object Tracking Surveys . . . . . . . . . . . . . . . . 52.2 Bounding Box Heads . . . . . . . . . . . . . . . . . . . . . . 62.2.1 Convolutional Design Choices . . . . . . . . . . . . . 62.2.1.1 Inception network . . . . . . . . . . . . . . . 62.2.1.2 Deformable Convolutions . . . . . . . . . . . 62.3 Deformable Masking . . . . . . . . . . . . . . . . . . . . . . . 6vii2.3.1 Tracking Paradigms . . . . . . . . . . . . . . . . . . . 62.3.1.1 Two-Stream Trackers . . . . . . . . . . . . . 72.3.1.2 One-Stream Trackers . . . . . . . . . . . . . 82.3.2 Masking in Visual Tracking . . . . . . . . . . . . . . . 92.3.3 Deformable Convolutions in Visual Tracking . . . . . 93 Beyond Traditional Single Object Tracking Survey . . . . 103.1 Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.1.1 Sequence Models . . . . . . . . . . . . . . . . . . . . . 103.1.1.1 Autoregressive models . . . . . . . . . . . . 103.1.1.2 Memory-based sequence models . . . . . . . 143.1.2 Probabilistic Generative Models . . . . . . . . . . . . 193.1.2.1 Masked Image Modelling . . . . . . . . . . . 193.1.2.2 Generative Adversarial Models . . . . . . . 203.1.2.3 Diffusion Models . . . . . . . . . . . . . . . 253.1.2.4 Variational Autoencoders . . . . . . . . . . 253.1.3 Learning Paradigms . . . . . . . . . . . . . . . . . . . 283.1.3.1 Self-Supervised Learning . . . . . . . . . . . 283.1.3.2 Unsupervised Learning . . . . . . . . . . . . 323.1.4 Meta Learning . . . . . . . . . . . . . . . . . . . . . . 343.1.5 Continual Learning . . . . . . . . . . . . . . . . . . . 403.1.6 Domain Adaptation . . . . . . . . . . . . . . . . . . . 413.2 Experimental Analysis . . . . . . . . . . . . . . . . . . . . . 433.2.1 Selected Datasets . . . . . . . . . . . . . . . . . . . . 433.2.2 Performance Metrics . . . . . . . . . . . . . . . . . . 493.2.2.1 Average Overlap (AO) . . . . . . . . . . . . 493.2.2.2 Success Rates (SR0.5 and SR0.75) . . . . . . 493.2.2.3 Area Under the Success Curve (AUC) . . . 493.2.2.4 Precision (P) . . . . . . . . . . . . . . . . . 493.2.2.5 Normalized Precision (PNorm) . . . . . . . . 493.2.2.6 Expected Average Overlap (EAO) . . . . . . 503.2.3 Cumulative Remarks . . . . . . . . . . . . . . . . . . 503.3 Discussion and Future Directions . . . . . . . . . . . . . . . . 504 Investigating Bounding Box Regression Method for One-Stream Single Object Tracking Transformers . . . . . . . . 544.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 544.1.1 Adopted Tracking Framework . . . . . . . . . . . . . 544.1.2 Inception Network . . . . . . . . . . . . . . . . . . . . 544.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 56viii4.2.1 Implementation Details . . . . . . . . . . . . . . . . . 564.2.2 State of the Art Comparison . . . . . . . . . . . . . . 574.2.2.1 GOT-10k . . . . . . . . . . . . . . . . . . . 574.2.2.2 UAV123 . . . . . . . . . . . . . . . . . . . . 584.2.2.3 OTB2015 . . . . . . . . . . . . . . . . . . . 594.2.3 Ablation Study . . . . . . . . . . . . . . . . . . . . . 594.2.4 Visualization . . . . . . . . . . . . . . . . . . . . . . . 594.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615 Learning Deformable Masked Visual Representations for Sin-gle Object Tracking . . . . . . . . . . . . . . . . . . . . . . . . 625.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 625.1.1 Problem Statement . . . . . . . . . . . . . . . . . . . 625.1.2 Learning Robust Tracking Features . . . . . . . . . . 655.1.2.1 Deformable Convolution Module . . . . . . 655.1.2.2 Masking Module . . . . . . . . . . . . . . . 655.1.2.3 Fusion . . . . . . . . . . . . . . . . . . . . . 665.1.3 Head and Loss . . . . . . . . . . . . . . . . . . . . . . 675.1.4 Tracking mode . . . . . . . . . . . . . . . . . . . . . . 675.2 Experimental Analysis . . . . . . . . . . . . . . . . . . . . . 685.2.1 Implementation Details . . . . . . . . . . . . . . . . . 685.2.1.1 Training Details . . . . . . . . . . . . . . . . 685.2.1.2 Tracking Details . . . . . . . . . . . . . . . . 715.2.2 Comparison with Current State-of-the-Art Trackers . 725.2.2.1 LaSOT . . . . . . . . . . . . . . . . . . . . . 725.2.2.2 LaSOText . . . . . . . . . . . . . . . . . . . 725.2.2.3 TrackingNet . . . . . . . . . . . . . . . . . . 735.2.2.4 GOT-10k . . . . . . . . . . . . . . . . . . . 735.2.2.5 UAV123 and OTB100 . . . . . . . . . . . . 735.2.3 Ablation Study and Analysis . . . . . . . . . . . . . . 745.2.3.1 Necessity of Deformable Convolution . . . . 745.2.3.2 Masking Ratio . . . . . . . . . . . . . . . . . 745.2.3.3 When to Mask . . . . . . . . . . . . . . . . 745.2.3.4 What to Mask . . . . . . . . . . . . . . . . . 745.2.3.5 High-Level Fusion . . . . . . . . . . . . . . . 755.2.3.6 Low-Level Fusion . . . . . . . . . . . . . . . 755.2.4 Visualization . . . . . . . . . . . . . . . . . . . . . . . 755.2.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . 755.3 Discussion and Future Directions . . . . . . . . . . . . . . . . 77ix6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80AppendixA Implementation and Reproducibility Details . . . . . . . . 108xList of Tables3.1 Summary of the reported performance measures. The datasetsinvolved are: GOT-10k [62], LaSOT [38], LaSOText [39], Track-ingNet [114], OTB50 [167], OTB100 [117], COCO [102], ILSVRC[128], ALOV++ [135], UAV123 [112], NUS-PRO [85], NFS[42], NYUv2 [132], YouTube-VOS [174], YouTube-BB [125],Kinetics300 [67], and DAVIS [123]. Please refer to section3.2.1 for a brief overview of the reported datasets. . . . . . . 453.2 Summary of the reported performance measures on the VOTchallenge dataset from 2016 to 2020 [74–78]. . . . . . . . . . . 473.3 A brief overview of the selected datasets used in reportingthe different performance metrics of the discussed trackers.Other mentioned training datasets in Tab. 3.1 and Tab. 3.1are briefly discussed in section 3.2.1. . . . . . . . . . . . . . . 484.1 Performance comparison against the state-of-the-art methods.We emphasize that our models are only trained on the trainingsplit of the GOT-10k dataset. Significant results are in bold 574.2 Investigation of the performance of the two proposed bound-ing box networks on two modern one-stream trackers. Sig-nificant results are in bold. All trackers are retrained fromMAE [52] pretraining weights on our machine using the sameconfigurations provided in their repositories for fair comparison. 584.3 Study of the performance of ODTrack with deformable convo-lution Inception module. ”deformable only” refers to the de-formable branch without the regular convolution while ”Convonly” refers to the original ODTrack [199]. ”Both” refers tousing the bypass branch. Significant results are in bold. . . 585.1 The average overlap of two SOTA trackers compared to ourson GOT-10k validation and test sequences at the end of train-ing on the training split of GOT-10k [62]. . . . . . . . . . . . 62xi5.2 Performance comparison with state-of-the-art tracking meth-ods. Our model outperforms the state-of-the-art ViT-basemethods by a large margin while competing with bigger ViT-Large and ViT-Giant. The two best results are red and green.The Asterisk ∗ indicates that all models tested on the GOT-10k test set are solely trained on the GOT-10k training set. 715.3 Demonstration of DMTrack efficiency according to MACs,number of parameters, and GOT-10k test set Average Overlap(AO). Our model exceeds computationally comparable mod-els in terms of AO by at least 2%. Moreover, it matches theperformance of a much bigger model, LORAT-G. . . . . . . . 725.4 Experimenting necessity of deformable convolutions. . . . . . 725.5 Experimenting selected counts of masked representations. . . 725.6 Experimenting the masking order at the best number of maskedrepresentations 10. . . . . . . . . . . . . . . . . . . . . . . . . 725.7 Experimenting the type of masked representations. . . . . . . 735.8 Experimenting high-level fusion variants. . . . . . . . . . . . . 735.9 Experimenting the low-level fusion variants. . . . . . . . . . . 735.10 State-of-the-art demonstration on UAV123 and OTB-100 . . 73xiiList of Figures3.1 Literature Survey Tree . . . . . . . . . . . . . . . . . . . . . . 113.2 The autoregressive family of trackers. The template size canbe different from one tracker to another. Some trackers usethe full resolution of the template while others use a smallersize that focuses on the object of interest. . . . . . . . . . . . 133.3 A general diagram of memory-based sequence models. . . . . 153.4 A general pretraining scheme using a masked autoencoder insingle object tracking. . . . . . . . . . . . . . . . . . . . . . . 193.5 A general diagram of GAN-based single object tracking meth-ods. The input to the generator is the features of the templateimage or a combination of the features of the template and thesearch. The generator output is either a map representing anenhanced set of template appearances or a robust feature rep-resentation depending on the model design. Either output isfed to a discriminator alongside ground truth maps that rep-resent parts of the search image or the template itself. Thediscriminator mainly outputs a classification decision reflect-ing a final score map or a scaler that determines the qualityof a generator output. . . . . . . . . . . . . . . . . . . . . . . 213.6 A general diagram of A VAE in single object tracking. Thedotted path is the condition used to direct the CVAE, whichis only mandatory in the case of a CVAE. In the diagram,N = N (0, 1) is a Gaussian noise vector. . . . . . . . . . . . . 263.7 A breakdown of the number of publications in single objecttracking that utilize transformers, CNNs, or any other ap-proach such as DCF. . . . . . . . . . . . . . . . . . . . . . . . 514.1 The overall architecture of the proposed ViT-based single ob-ject tracker bounding box head. . . . . . . . . . . . . . . . . . 55xiii4.2 The simple Inception subnetwork architecture adapted from[146]. The blue-filled circle indicates concatenation operation.Filter sizes are shown in the figure. Please refer to [146] formore details about the parameters of each convolution. . . . . 564.3 The deformable inception block. . . . . . . . . . . . . . . . . 564.4 Success cases of the proposed methods on the OTB2015 Bas-ketball sequence. It can be noted that the pure Inception net-work is most accurate especially when comparing the first col-umn of frames. Although bounding boxes converge at frame#50, the initially predicted bounding boxes at frame #2 indi-cate the fast adaptation of the Inception bounding box network. 605.1 End-to-end single object tracker based on the deformable mask-ing module. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.2 Two variants of the deformable masking module. (A) is de-noted as the bypass variant where outputs of the deformablemasking module at each layer are fused. (B) is the sequentialvariant where the outputs from the last layer’s deformablemasking module are fused with the output of the encoder andthen fed into the next layer transformer encoder. The num-ber of the deformable masking modules is L−1 for any of thevariants. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685.3 Deformable convolution network. In this work, the offset con-volution is a stack of 3×3 convolution and layer normalization[2]. The modulation convolution is a stack of 3 × 3 convolu-tion, softmax activation and layer normalization. We adoptlayer normalization in this submodule according to conver-gence time and test metrics results. . . . . . . . . . . . . . . . 695.4 Possible masking areas of a transformer feature map at a cer-tain layer. (A) masking is carried on the embedding dimensionwhere several representations are removed from all combinedsearch and template tokens. (B) masking is carried on the to-ken dimension where several tokens are removed. (C) maskingis performed on both dimensions. . . . . . . . . . . . . . . . 69xiv5.5 Three ways of information flow in the fusion block. (A) is asimple addition of all feature maps input to the fusion module.(B) is learnable single-stage weighted concatenation, wherethe inputs to the fusion module are first concatenated andthen fed into a 1 × 1 convolution to output a feature mapthe same size as any of the inputs. (C) is learnable multi-stage weighted concatenation, in which each two input featuremaps are concatenated and fed into a 1×1 convolution whichoutputs a new feature map that is then concatenated with thenext input to be fed into the next 1×1 convolution. The totalnumber of 1× 1 convolutions is L− 1 where L is the numberof layers of the transformer. . . . . . . . . . . . . . . . . . . 705.6 Visual results of DMTrack on three different datasets. . . . . 765.7 Visual results of DMTrack on two webcam sequences. . . . . 76xvGlossary of NotationAI . . . . . . . . . . . . Artificial IntelligenceViT . . . . . . . . . . Vision TransformerCNN . . . . . . . . . Convolutional Neural NetworkDCN . . . . . . . . . Deformable Convolutional NetworksNLP . . . . . . . . . . Natural Language ProcessingSSL . . . . . . . . . . Self-Supervised LearningSGD . . . . . . . . . Stochastic Gradient DescentADAM . . . . . . . Adaptive MomentumBB . . . . . . . . . . . Bounding BoxIN1K . . . . . . . . . The ImageNet-1k datasetFSL . . . . . . . . . . Few-Shot LearningVAE . . . . . . . . . . Variational AutoencoderMAE . . . . . . . . . Masked Auto EncoderGAN . . . . . . . . . Generative Adversarial NetworkILSVRC . . . . . . ImageNet Large Scale Visual Recognition CompetitionVOT . . . . . . . . . Visual Object Tracking ChallengeOTB . . . . . . . . . Object Tracking BenchmarkGOT-10k . . . . . Generic Object Tracking-1k DatasetLaSOT . . . . . . . Large-Scale Single Object Tracking DatasetxviAcknowledgementsFirst and foremost, I would like to thank my supervisor, Dr. Mohamed She-hata, for his support, encouragement, and patience throughout my studies.I would also like to extend my sincerest thanks to Islam Osman and Mo-hamed H. Mohamed. Your valuable feedback, support and guidance madeit possible for this work to fruit.xviiDedicationI dedicate this work to my Mother, Father, Siblings and friends.xviiiChapter 1Introduction1.1 OverviewResearchers in the field of computer vision have long strived to mimic human-level capabilities in understanding and interpreting visual information. Earlyattempts at object tracking, such as the Kalman Filter [66], relied on math-ematical models and hand-crafted features to estimate object trajectorieswithin video sequences. These methods achieved modest success, but theirperformance often deteriorated when faced with challenging scenarios in-volving occlusion, background clutter, and rapid motion.Over the years, various approaches emerged to refine object tracking, in-cluding Particle Filters [9] and Mean-Shift [25]. These techniques employedmore sophisticated motion modelling and feature representation, leading tosome improvements. However, their reliance on handcrafted features lim-ited their ability to adapt to the inherently challenging nature of real-worldvideos.The recent surge in deep learning has revolutionized object tracking,leading to the development of Deep Single Object Tracking techniques. Con-volutional Neural Networks (CNNs) [83], with their powerful ability to learncomplex feature representations directly from data, have proven adept atextracting robust and discriminative features from video sequences. Thishas enabled approaches to outperform traditional methods by a significantmargin, achieving higher accuracy and robustness in challenging trackingscenarios. However, the locality of CNNs induces an inductive bias thatdegrades the performance of a CNN-tracker when the model is required tolearn complex relations between the object and its surroundings.This was the case till the Vision Transformer (ViT) was introduced [36]to leverage the attention mechanism to limit assumptions about data and in-crease the learning capacity. When employed in tracking, ViTs were proveneffective in tackling many challenging scenes that traditional and CNN-basedmethods struggled with. The current research landscape leverages a one-stream ViT tracking where the ViT takes in a template image representingthe object to be tracked and the search image and outputs an attentional11.2. RESEARCH MOTIVATIONfeature map from which the location of the object can be determined. De-spite the significant improvement of this paradigm compared to previousmethods, there are still two substantial problems. First, the model does notfully leverage the information provided in the template image. Second, theperformance gap between the training and test data is still vast.This thesis delves into the recent advancements in single object trackingbased on the recent theories and technologies in Artificial Intelligence (AI).We provide a comprehensive overview of the emerging technologies in thefield, analyzing the established deep learning-based tracking techniques thatdominate the current landscape and exploring the limitations of these ap-proaches. We delve into the specific challenges that continue to hinder singleobject tracking performance, such as handling occlusions where the targetobject is temporarily hidden by another object, dealing with rapid motionblur, and adapting to significant appearance changes that the object mightundergo. We can identify opportunities to push the boundaries of singleobject tracking performance by critically examining these limitations. Fur-thermore, we critically assess non-traditional approaches that deviate fromthe standard deep learning framework. Highlighting the potential and ar-eas for further exploration in these non-traditional approaches is crucial forfostering innovation and achieving breakthroughs in single object tracking.Following our review of the current landscape, we propose a novel bound-ing box regression model that leverages the strengths of deep learning foraccurate object localization. This model addresses the local receptive fieldlimitation identified in the prevalent convolutional bounding box head.Furthermore, we introduce a new visual tracker that harnesses the de-formable convolutional networks associated with uniform random maskingthat enhances the quality of the attentional representations of a one-streamViT tracker. This is done considering the problems identified in the surveyto achieve robust and efficient tracking performance. The novel strategyensures the tracker can maintain consistent performance even in challengingsituations.1.2 Research MotivationThis research was motivated by the emerging techniques in deep learningresearch. Current single-object tracking research is lagging when implement-ing new theoretical techniques. Moreover, the gap in performance betweentraining and test data needs to be narrowed by learning high-quality repre-sentations. Ultimately, this gap should be minimal, as is the case with other21.3. RESEARCH OBJECTIVEfields such as image classification. As such, Our intention is to contribute tothis body of work and inspire further research pathways.1.3 Research ObjectiveThe overarching goal behind this research involves examining the currentstate-of-the-art for few-shot learning with regard to out-of-domain perfor-mance and formulating new techniques for improvement in this field. Inorder to achieve this goal, we focus on the following objectives:1. Survey the emerging techniques such as sequence models, masked im-age modelling, generative adversarial networks, diffusion models, vari-ational autoencoders, self-supervised learning, unsupervised learning,meta-learning, and continual learning, and domain adaptation in singleobject tracking, analyze each method and propose future directions.2. Investigate the bounding box regression head often associated withViT-trackers and propose new bounding box networks to address theshortcomings of existing methods.3. Enhance the quality of learned representations using one of the emerg-ing techniques examined in the survey.4. Decrease the performance gap between the training and testing singleobject tracking data.1.4 Contributions SummaryTo summarize, the main contributions of this thesis are as follows:1. This thesis surveys the novel non-traditional deep learning techniquesin single object tracking. To the best of the author’s knowledge, thisresearch is the first to provide a taxonomy based on emerging deep-learning techniques and their employment in single object tracking.2. This thesis proposes Inception and Deformable Bounding Box Net-works, which can be integrated into any one-stream ViT-tracker forimproved object localization.3. This thesis provided a Deformable Masking Tracker (DMTrack) thatleverages deformable masked representations to narrow the gap be-tween the training and testing accuracy. DMTrack achieves state-of-the-art performance on multiple single-object tracking benchmarks.31.5. PUBLISHED AND SUBMITTED WORKS1.5 Published and Submitted WorksAs part of working towards the Master’s degree, multiple research papersare published or in submission. These papers can be classified into relatedand unrelated to my thesis, which is listed as follows:1. Walsh R, Osman I, Abdelaziz O, Shehata MS. Fully Self-SupervisedOut-of-Domain Few-Shot Learning with Masked Autoencoders. Jour-nal of Imaging. 2024; 10(1):23. (Unrelated, Published)2. Beyond Traditional Single Object Tracking: A survey, Omar Abde-laziz, Mohamed S. Shehata, Mohamed H. Mohamed, InternationalJournal of Machine Learning and Cybernetics, 2024 (Related, Sub-mitted)3. A Novel Bounding Box Regression Method for Single Object Track-ing, Omar Abdelaziz, Mohamed S. Shehata, International Conferenceon Intelligent Systems, Blockchain, and Communication Technologies,2024 (Related, Submitted)4. DMTrack: Learning Deformable Masked Visual Representations forSingle Object Tracking, Omar Abdelaziz, Mohamed S. Shehata, IEEETransactions on Artificial Intelligence, 2024 (Related, Submitted)1.6 Thesis OutlineThe remainder of this thesis is organized as follows: Chapter 2 providesa background on work related to the experiments conducted and proposedtechniques. Chapter 3 surveys the emerging deep learning techniques andprovides insights for future directions. Chapter 4 incorporates deformableand inception convolutions in two novel bounding box regression heads.Chapter 5 proposes DMTrack, a state-of-the-art tracker with high-qualityjoint tracking representations. Finally, Chapter 6 concludes this thesis andprovides avenues for future work in this area.4Chapter 2Related Work2.1 Single Object Tracking SurveysIn the existing literature, several survey studies on Visual Object Tracking(VOT) have been conducted over the past two decades. For example, Yilmazet al. [183] stand out as pioneers in comprehensively analyzing the track-ing pipeline. They systematically categorized tracking methods into featurecorrespondence, primitive geometric models, and contour methods.Smeulders et al. [135] provided a new dataset, the ALOV++ and con-ducted an experimental survey that covered 19 distinct online trackers, span-ning the period from 1999 to 2012. Their study included multiple evaluationmetrics applied to single object trackers. Zhang et al. [192] contributeda survey focused on recent methodologies in deep learning and correlation-filter paradigms. Javed et al. [65] concentrate on recent developments in thetwo most popular traditional and deep learning-based methods paradigms:discriminative correlation filters and the Siamese-based paradigms. In theirsurvey, 90 trackers were presented, and 9 trackers out of them were experi-mentally tested.Given the current state of Vision Transformers (ViTs) [36] and its ubiq-uitousness in computer vision, Kugarajeevan et al. [79] extensively surveyedthe new paradigms emerging with the adoption of ViT in single object track-ing. They conducted attribute-based comparisons of different transformer-based and non-transformer-based trackers. Each tracking challenge, e.g.,camera motion and shape deformation, is examined independently in thesecomparisons. Moreover, all transformer-based and non-transformer-basedmethods are compared in another set of experiments to demonstrate theeffectiveness of the transformer-based paradigms.Contrary to other works, this work sheds light on the emerging theoret-ical deep learning paradigms and their application in single-object tracking.The main focus of this survey is to delve into recent single-object trackingmethods that are built upon a solid foundation of statistical theories andhave led to the introduction of new paradigms such as generative modelsand meta-learning.52.2. BOUNDING BOX HEADS2.2 Bounding Box Heads2.2.1 Convolutional Design Choices2.2.1.1 Inception networkThe inception network was introduced in [133] to overcome the problem ofthe narrow receptive field of previous CNNs. The main idea behind theoriginal paper and its variants [145] is that for a convolutional network togo deeper, its receptive field has to be wide enough to process the inputon many deep stages without losing information from the early stages. Forthat purpose, the Inception module carries on several convolutional passes inparallel, each representing a different convolutional filter. Different outputsof the passes are concatenated and passed to a 1 × 1 convolutional filter toreduce the channel dimension in a learnable manner. In this work, we applythe simplest inception module to the output of a ViT encoder from threeknown trackers.2.2.1.2 Deformable ConvolutionsDeformable convolution [27] is an extension of the regular convolution wherethe sampling positions of the convolutional filter are learnable. In this regard,the deformable convolution operation involves two runs of convolutional fil-ters. The first run determines the offsets that the filter of the second runwill use to sample values of the input feature map, which will apply theconvolution operation. In another version of the deformable convolutions[207], the amplitudes of the input feature maps are modulated by a learn-able mask, ensuring an extra degree of freedom. We implement a variant ofthe proposed Inception bounding box network with deformable convolutionsto exploit the powerful learning paradigm encapsulated by the learnablesampling positions and input amplitudes.2.3 Deformable Masking2.3.1 Tracking ParadigmsRecently, deep learning-based trackers have led to new performance fronts insingle-object tracking. Based on feature extraction and relation modelling,deep learning trackers can be categorized into two main categories: one-stream and two-stream trackers.62.3. DEFORMABLE MASKING2.3.1.1 Two-Stream TrackersHistorically, two-stream deep trackers had been prevalent with the intro-duction of Siamese convolutional networks for tracking in SINT [148] andSiamFC [5]. These methods treated visual object tracking as an instancesearch problem where two representations of the first appearance of theobject to track (template) and the search image are mapped to the sameembedding space using two identical branches of CNN. This stage is thefeature extraction stage. In the subsequent stage, the relation between themapped representations is modelled as a cross-correlation filter that outputsa classification score map from which the object’s location can be determinedin the search location. Many trackers based on Siamese networks were intro-duced afterwards [61, 86, 87, 134, 194] involving many enhancements. Forinstance, SiamDW [194] trained deeper networks by mitigating the spatialinconsistency resulting from the padding operation. SiamRPN [87] lever-aged the region proposal networks introduced in object detection literatureto accurately determine the object’s location. SiamMask [61] integrated anenhanced objective of mask prediction alongside the bounding box.An extension to the two-stream tracking was introduced in [21] by re-placing the relation modelling stage with a transformer cross-attention. Tr-SIAM [157] integrated the transformer architecture by considering the trans-former’s encoder as a feature embedding network and its decoder as a rela-tion modelling network. Another extension to the two-stream paradigm isSTARK [176], which deploys the vision transformer into it, having the en-coder as a feature embedding model, taking its output from a CNN featureextractor. Then, the feature embeddings are fed into a transformed decoderthat decodes the whole sequence into a query token that models the framehistory. MixFormer [26] uses a mix of transformer layers and CNN layers toachieve the optimal model size by discarding the template-to-search cross-attention by concatenating the tokens and applying self-attention to theconcatenated output. AiATrack [44] proposed the Attention-in-Attentionmodule to focus on the template object and eliminate the erroneous rela-tions between the template representations and the representations of sim-ilar distracting objects. SwinTrack [100] used a Shifted Windows (Swin)transformer [104] to encode the features and used a slim transformer layerto model the relation between template and search embeddings. Althoughtransformers had pushed the performance of the single object tracking for-ward, the two-stream tracking paradigm did not fully exploit the generalityof the transformers.72.3. DEFORMABLE MASKING2.3.1.2 One-Stream TrackersInstead of relying on separate stages to extract visual features and modelthe relation between those features, one-stream tracking allows feature ex-traction that is aware of the tracking task and a relation modelling that canadapt to the extracted features. This paradigm was introduced in OSTrack[182] and [15]. Both works follow a simple paradigm of dividing the templateand search images into patches and feeding concatenated patches into thetransformer. OSTrack [182] enhanced the efficiency of the one-stream modelby introducing candidate elimination to prune dissimilar search tokens, en-hancing the FPS and the model’s attention to the critical token. Many worksrapidly exploited the new paradigm. GRM [45] generalized the one-streampipeline to include adaptive search token selection. ROM [12] trained a one-stream tracker to exploit object representations from different frames in anautoregressive manner by passing selected tokens from the previous frameembedding layer to the current one. VideoTrack [170] introduced video-leveltraining of one-stram trackers by dividing multiple frames into patches andembedding them in one step. SeqTrack [20] and ARTrack [163] introducedsequence-to-sequence modelling in tracking by treating single object track-ing as a sequence prediction problem. The main idea is to exploit previousbounding boxes in the current step during training and inference.Our adapted baseline ODTrack [199] is a recent tracker that utilizes amemory of templates and searches to achieve video-level tracking. The mem-ory is represented at each layer by a query token containing informationabout the object from different template frames. The token is propagatedin the transformer layers by attaching it to the attentional feature maps ateach layer, thus generalizing the one-stream paradigm to the sequence level.Despite the success ODTrack has shown, we argue that incorporating manytemplates and searches in a one-stream framework might lead the modelto learn easy representations that do not align with the test data. This ismainly because a one-stream tracker can flexibly modify the representationsto fit the straightforward relation (representation) that can be deduced givenmultiple frames with many redundant features. To alleviate this effect, weintroduce the deformable masking module that masks some of these repre-sentations randomly during training, thus preventing the transformer fromdeducing easy representations because they will be masked in the followingtraining step.82.3. DEFORMABLE MASKING2.3.2 Masking in Visual TrackingMany computer vision tasks have exploited masking input images and in-ternal representations. Masked Image Modelling (MIM) was introduced inMasked Auto Encoder (MAE) [52] as a self-supervised learning task for im-age classification. In this task, the model is prompted to reconstruct maskedparts of the input image. In visual tracking, an MAE-like objective was in-troduced in DropTrack [164] and MAT[196]. In DropTrack [164], the modelis prompted to reconstruct the search frames with a modified backbone thatdrops out some of the representations in addition to the masking performedon the image level. MAT [196] reconstructs the template image and thepart corresponding to the template in the search image using an encoderand two shared-weight decoders. The tracking model utilizes the encoderonly by performing a cross-correlation operation between the template andsearch embeddings produced solely by the encoder. In our work, we maskout complete tokens to reduce overfitting. We mask tokens correspondingto joint interaction between template and search embeddings. This maskingscheme aims to produce high-quality, valuable representations across differ-ent datasets.2.3.3 Deformable Convolutions in Visual TrackingSince their introduction [207], deformable convolutions have been heavilyused in many computer vision tasks and visual tracking is no exception.SiamDCN [198] integrated deformable convolutions in the Siamese track-ing, showing that a dynamic receptive field is crucial for visual trackingto catch different deformations that affect the object. More recently, [118]introduced a deformable attention module in CNN to attend to differenttemplate cues dynamically. Additionally, [64] incorporated deformable con-volutions in three branches of CNN, ViT, and DCF sub-trackers and showedthat each tracker sees the object from a different perspective. In this work,we utilize deformable convolutions to reconstruct the transformer’s mask-ing representations, allowing long-range dependencies without requiring aneasy solution. Moreover, deformable convolutions barely add to the system’scomplexity compared to an additional transformer layer.9Chapter 3Beyond Traditional SingleObject Tracking Survey3.1 TaxonomyIn this survey, the taxonomy mainly focuses on non-traditional visual ob-ject tracking techniques. Figure 3.1 shows the taxonomy of the surveyedmethods.3.1.1 Sequence Models3.1.1.1 Autoregressive modelsThis family views single object tracking as a sequence prediction problem,just as language models do. Given an input pair of template and searchimages, the goal of these models is to predict a four-token sequence repre-senting the bounding box parameters to predict the location of the templateappearance in the search image as shown in Fig. 3.2. Each of the predictedsequence tokens is causally dependent on each previous token. The trainingobjective is to maximize the likelihood of the current token given the previ-ous tokens and the search and template features, which is usually achievedby a cross-entropy loss function as follows:ℓ = maxT∑t=1logP (Y t|Y t−N :t−1, Xt, Z) (3.1)where T is the number of frames. N is the number of previous frames onwhich the generation of the current frame is dependent. N is also calledin literature [180] the order of autoregression. Xt is the search image. Zis the exemplar frame, which can be the initial frame or a combination ofpast frames. Formulating the problem this way requires a vocabulary ofnumbers that is shared across all sequences and certainly a discrete repre-sentation of numbers so that it can be contained in a single dictionary. Theimplementation and architectural aspects of each model are discussed below.103.1. TAXONOMYNon-traditionalTracking MethodsSequence ModelsAutoregressive ModelsARTrack [163]SeqTrack [20]AR-HMM[119]Memory-Based ModelsAD-LSTM[188]STMTrack [41]MemDTC[177]MemTrack[177]Siamese R-RPN [197]GCT [43]BA + TID +DDM [161]RFL [178]Generative Prob-abilistic ModelsMasked Image ModellingMAT [196]DropMAE[164]Generative Adversarial ModelsTGGAN [49]VITAL [139]GAN-Based[50]GARAT [181]RCNet [195]SGAN +FGAN + RD[51]AFSL [184]Diffusion Models Diff-SiamRPN++[191]Variational AutoencodersSINT++ [159]VAE-MCMC[81]SiamVAE [173]Depth-aware[63]Figure 3.1: Literature Survey Tree113.1. TAXONOMYNon-traditionalTracking MethodsLearning ParadigmsSelf-SupervisedD-EM [169]QCT [206]Joint-task [94]ETC [205]Crop-Transform-Paste [93]CycleSiam[186]UniTrack [162]UnsupervisedPUL [166]UDT [156]ContrastiveCycle [203]Meta-LearningMeta-Tracker[120]GradNet [91]MLT [22]DIMP [7]FCOS-MAML+ Retina-MAML [155]LMTU [28]CLNet [35]MGA-Net [165]FastAndRobust[96]RTS [121]Continual LearningTracking viaContinualLearning [190]ConTACT [23]Domain AdaptationCODA [88]MDNet [116]Figure 3.1: Literature Survey Tree123.1. TAXONOMYAutoregressiveModelSearchTemplatext yt ht wt [T ]xt−1 yt−1 ht−1 wt−1 [T ]Figure 3.2: The autoregressive family of trackers. The template size can bedifferent from one tracker to another. Some trackers use the full resolutionof the template while others use a smaller size that focuses on the object ofinterest.[AR-HMM] Autoregressive Hidden Markov Model: The foundations ofthe autoregressive family of models were established in this paper [119]. Thismethod aimed to solve the visual tracking problem by exploiting temporaldependencies between sequential target appearances. This method assumesthe following about the autoregressive tracking problem: First, the objecthas many appearance models in the same time frame. Their method aimsto calculate the most probable appearance model, which can be employedafterwards to identify the object in the image. Second, the dependencybetween the frames is considered only between the consecutive appearancemodels, hence, the current state of the object is only dependent on the cur-rent appearance model and the current appearance model is dependent onthe previous appearance models. Furthermore, the dependencies are mod-elled using a hidden Markov chain. With this formulation, the goal is to trainsome classifier that internally generates appearance models that lead to asuccessful classification of image patches as a current state of the trackedobject. This classification is done hierarchically, and a final classificationscore for each patch is given according to a combination of shared weightsacross all appearance models and specific ones. Finally, an argmax samplingmethod is used to determine the next state based on the previous state andlearned appearance models.[SeqTrack] Sequence-to-Sequence Tracker: In this paper [20], the model133.1. TAXONOMYis prompted with a pair of template and search images and a box token. Eachimage I ∈ R3xHxW , is different from many works in the literature [5, 182]where the template is smaller in size than the search image. Using the fullresolution makes the model more robust as the training is done with morebackground information. While training, the input box sequence is mod-elled as [start, x, y, h, w]. The target sequence is modelled as [x, y, h, w, end]where x, y are the coordinates of the bounding box center and h,w is thewidth and height. Each continuous variable is discretized uniformly to havean integer value between [1, nbins] where nbins is the number of bins used todiscretize the continuous parameter. The model architecture in this methodis composed of an encoder-decoder architecture. The encoder is a regularvision transformer (ViT) [36] where the patches of the search and templateare fed into. Half of the output patches are considered for the next stage,which is a causal decoder [151]. The decoder jointly takes the encoded searchand template patches and the previous bounding box token to predict thecurrent one. During the inference phase, the prior knowledge is integrated toupdate the appearance model, penalizing drastic changes between two con-secutive frames using a Hanning window. Moreover, the template is updatedonline after Tu frames are processed according to the average probability ofall tokens of the current bounding box sequence.[ARTrack] Autoregressive Visual Tracker: In this work [163], the au-toregressive model is different from SeqTrack [20] where the current sequencedepends on N (order of autoregression) previous sequences. This model usesdifferent sets of inputs and outputs. First, the template size is smaller thanthe search. Second, the bounding box sequence representation is in the formof [xmin, ymin, xmax, ymax] where xmin, ymin are the coordinates of the top-left corner of the box and xmax, ymax are the coordinates of the bottom-rightcorner. The coordinate system in this work is unified to have all coordinatesof the N frames referred to in one coordinate system with one global zero.Interestingly, this method does not use start or end tokens because the se-quence size is always set to a fixed length of four. Another input is thecommand token, a learnable token that distinguishes between bounding boxsequences whenN past sequences are used. The model architecture is mainlyan encoder-decoder architecture. The loss function uses a combination ofcross entropy and L1 losses.3.1.1.2 Memory-based sequence modelsAs depicted in Fig. 3.3, memory-based sequence models are designed tomodel the temporal correspondence between consecutive frames (i.e., video143.1. TAXONOMYSearchTemplateTrackingModelMemoryModelMemoryUpdaterxt yt ht wtFigure 3.3: A general diagram of memory-based sequence models.frames). The appearance model in these models is formed and updated usinga Recurrent Neural Network (RNN).[RFL] Recurrent Filter Learning for Visual Tracking: This paper [178]integrated ConvLSTM [131] for the first time in object tracking network toexploit the spatial information captured by a CNN [83] and update it slowlyusing an LSTM as a memory unit. The network consists of an Exempler-CNN (E-CNN), which extracts the exemplar (template) features, a Search-CNN (S-CNN), which extracts the search features and a ConvLSTMmodule.Similar to SiamFC [5], the goal is to get the cross-correlation between theexemplar and search features and use the correlation map to predict the ob-ject location. However, the main difference is the application of ConvLSTMon the exemplar features to recall the early exemplar states while benefittingfrom the current state. The loss function is the cross entropy similar to [5].[GCT] Graph Convolutional Tracking: This work [43] was the first to in-troduce Graph Convolutional Networks (GCNs) [71] in visual tracking. Thepaper proposed using a GCN to generate spatial-temporal graph embeddingsof mapped historical templates Zt−T :t−1 referred to as ST-GCN. Further-more, the graph comprises nodes representing target parts at different timeframes and edges in a fully connected manner to represent the dense depen-dencies between the template parts in space and throughout time. Parallelto this process, a context-aware network leverages context information fromthe current search image for target localization. The context embeddings arethen concatenated with the spatial-temporal graph embeddings and fed intoa graph learning network to form an adaptive graph. The adaptive graph isfed afterwards to a context-aware GCN that also takes the spatial-temporal153.1. TAXONOMYgraph embeddings to output an adaptive feature map. These feature mapscontain information from spatial-temporal relations between template statesand the context of the current search image. Finally, the adaptive featuremap is cross-correlated with the search image embeddings following [5] toproduce a score map, which is then utilized to locate the object in the searchimage.[Siamese R-RPN] Cooperative Use of Recurrent Neural Network andSiamese Region Proposal Network for Robust Visual Tracking: In this work[197], the researchers built upon SiamRPN [87] which involves tracking viatwo Siamese networks for classification and regression. The idea of using Re-gion Proposal Networks (RPNs) [126] has been used extensively in variousfields in computer vision [54, 72, 142]. This is owing to its effectiveness inattending to important parts in the images where the gradient is updated fol-lowing the proposal selection process. This method is intended to fix some ofSiamRPN issues by introducing three new components: Channel and spatialattention, memory anchor selection and RPN integrated loss. First, chan-nel and spatial attention is a series of convolutional layers that enhance theRPN capability to identify relevant features. Second, the memory anchorselection subnetwork is a two-layer LSTM network that takes feature mapsof both the previous frame and the current frame concatenated to outputand enhanced anchor based on the trajectory of the object. Henceforth,the network significantly reduces the number of region proposals and sub-sequently increases the performance while reducing the run time. Finally,the loss function is a combination of classification, regression, and anchormatching losses, hence ensuring that the anchor selection process is guidedin a supervised manner.[STMTrack] Template-free Visual Tracking with Space-time MemoryNetworks: In this work [41], the memory is a bank of previous frames con-trary to other designs with a bank of previous representations. The modelhas three main components: A feature extraction network, a space-timememory reader network, and a head network. As inputs, the model takesa query frame q and a set of memory frames m of length T alongside aper-frame classification map with a positive label where the object existsand a negative label elsewhere. Beginning with the feature extraction, twoinstances of GoogLeNet [147] extract features of query frame and memoryframes separately. Regarding the memory frames, each frame is combinedwith its classification map and is fed through a shared instance of the featureextraction network across all frames. In contrast, the query frame is fed intoa different instance of the feature extraction network. Moreover, the fea-tures obtained from both query and memory frames are fed into a memory163.1. TAXONOMYreader network. The reader network functions as a one-layer cross-attentionmechanism to calculate the non-local attention of a query frame w.r.t. thememory frames. Considering the attention mechanism, the key K is thememory features, the query Q is the query features, and the value V is notused. Finally, the output, represented by an attention map, is fed into thehead network to predict the bounding box coordinates in the query frame.The head network architecture is explained in [175]. During inference, thenetwork selects N frames to be used as memory frames, comprising the firstframe in the video, the frame just before the current frame, and selectedN − 2 in-between frames that balance several aspects. These aspects aredomain adaptation, underfitting effect and time cost.[MemDTC] Visual Tracking via Dynamic Memory Networks: In thispaper [177], The researchers introduce a dynamic memory network equippedwith reading and writing mechanisms tailored for visual tracking. A Con-volutional Neural Network (CNN) is initially employed to extract featuresfrom the provided search image. These image features are then fed into anattention-based Long Short-Term Memory (LSTM) network, overseeing theprocesses of memory reading and writing. A residual template is extractedfrom the positive memory and combined with the initial template acquiredfrom the first frame, resulting in the positive template. Subsequently, anegative template is retrieved from the negative memory to nullify specificportions of the positive template via a channel-wise gate. The gate considersthe similarity between the positive and negative memory features, leading tothe creation of the final template. The final template undergoes convolutionwith the features of the search image to generate a response map, from whichthe prediction of the target bounding box is derived. The newly identifiedtarget’s template is cropped using the predicted bounding box, and featuresare then extracted and written into the positive memory to facilitate modelupdating. Concurrently, the negative template that contains the distractorsis extracted from the search feature map based on the response map. Re-sponses exceeding a predefined threshold score and situated at predefineddistances far from the target’s center are identified as negative (distractor)templates, contributing to the writing process in the negative memory.[AD-LSTM]Attention-Driven Memory Network for Online Visual Track-ing: In this method [188], the authors proposed AD-LSTM inspired by bothlong-term and short-term human visual memory in addition to a memoryupdater via online training. To this end, the tracker consists of four compo-nents: Feature extractor, attention-driven LSTM, convolutional short-termmemory and feed-forward memory updater. First, the input search imageis processed by a VGG16 network as a feature extractor [133], resulting173.1. TAXONOMYin a feature map as an output. Furthermore, the feature map is fed intothe Attention-Driven [60] Long-Short Term Memory (AD-LSTM), which isa variation of ConvLSTM [131] aiming to memorize the complex appear-ance models of the tracked object. The different part of AD-LSTM fromConvLSTM is where spatial and channel attention to input feature mapsare considered. While it is calculated independently for each channel mapto obtain information about the target location and interactions with sur-roundings, channel attention is calculated across all channels to obtain in-formation about the target’s appearance and shape in different spaces. Eventhough both spatial attention and channel attention are intended for differ-ent tasks, they are implemented using mainly average and max pooling andfeed-forward multi-layer perceptions (not the attention mechanism used inViTs). Parallel to long-term memory processing, short-term memory is in-troduced to tackle the subtle change in the appearance of the tracked object.Both memory outputs are combined to form a response map to determine theexact target object location. After that, for upcoming frames after the firstone, the tracking result is fed to a memory updater to determine whetherthe current tracking result can be used to update the long-term memory.The memory updater is a convolutional network that outputs a real numberscore according to which the current frame can be considered based on acertain threshold in the update process. Finally, a natural choice for theobjective function is to minimize the mean of squared differences betweenthe predicted response map and the ground truth, which can be considereda classification problem. The learning objective becomes trivial since thedominant labels within the response map are negative. A better objectiveis to minimize a gradient harmonized loss function [189], which gives moreweight to valuable (hard) samples in comparison to easy samples (negatives).The idea is intended to alleviate the effect of easy negatives by determiningwhich samples are easy according to the L2 loss gradient at each sample andthen counting the samples at each gradient value. Finally, inversely propor-tional weights to the number of samples assigned at each gradient value arecalculated. These weights determine the importance of each sample.[BA+TID+DDM] Robust Tracking via Fully Exploring BackgroundPrior Knowledge: In this method [161], the goal is to memorize distractorsthat appear in the background by explicitly modelling and mining them.First, the template and search regions are embedded using ResNet50 [55].For both regions, X,Z ∈ R3xHxW . After that, the object and backgroundfeatures are extracted from the template features to exploit the BackgroundArea (BA). Second, a Discriminative Distractors Mining (DDM) modulecalculates the similarity between the target object and different background183.1. TAXONOMYareas. The similarity (inner product) selects the top K distractors to feedthem with the target and search embeddings into a transformer-based en-coder. Through the DDM module, prior knowledge of what parts confusethe model (i.e. distractors) can be explicitly modelled, and other redundantparts are discarded. Following the DDM, a multi-head self-attention Trans-former is used to interactively mix the concatenated object, background(distractors), and search features. The self-attention mechanism forces theexploitation of useful distractor information from the template image andcontext information from the search image to locate the target and have highconfidence in prediction by distractor awareness. Consequently, a Target-distractor Independent Decoder (TID) is presented as a memory of previousappearances of the distractors using multi-head cross attention. The memoryis accessed by combining target and search features and another combinationof the distractors and search features. Lastly, a classification and regressionhead is employed on top of the fused target-distractor-context to predict thetarget’s confidence and location as in [182].3.1.2 Probabilistic Generative Models3.1.2.1 Masked Image ModellingSearchTemplateMasking ReconstructionTemplateSearch EncoderDecoderFigure 3.4: A general pretraining scheme using a masked autoencoder insingle object tracking.After a remarkable success in Natural Language Processing (NLP) [68,73, 84], the masked information modelling paradigm made its way into thevision domain as masked image modelling [52]. It also attained astonishingachievements in many vision subfields such as classification [52], semanticsegmentation [158] and image generation [14]. In a single object tracking set-ting, the masked autoencoder is generally used to mask two time-dependent193.1. TAXONOMYframes of the same sequence and predict parts or whole of them as depictedin Fig. 3.4.[MAT] Representation Learning for Visual Object Tracking by MaskedAppearance Transfer: This method [196] is an extension of [52] in the visualtracking field. Both the template and search images masked patches are fedinto an encoder. Two lightweight decoders that share weights are employedto reconstruct the search image and template appearance within the searchregion. This is done based on the assumption that reconstructing both im-ages pushes the encoder to cluster the embeddings of the template closer toits corresponding search embeddings. Based on this method, the tracker usesthe pre-trained encoder to embed the search and template images jointly.Following that, a cross-correlation operation between the search’s and thetemplate’s jointly produced embeddings is performed to obtain a final rep-resentation. Based on that representation, a simple bounding box networkthat adapts [82] is used to infer the corners of the bounding box.[DropMAE] Masked Autoencoders with Spatial-Attention Dropout forTracking Tasks: This work [164] shows that the direct adaptation of MAEto the single object tracking task overlooks the temporal cues in the framesin favour of the spatial cues. The proposed solution is that, given two inputframes, an adaptive spatial dropout module is applied to each head of thedecoder multi-head self-attention module. This module drops out large-value attention elements in the spatial dimension with a certain probabilityif these elements have good matches in the temporal dimension. The spatialdropout learns better when two identity embeddings that distinguish theinput frames are added. To finally integrate the model in a tracking setting,OSTrack [182] is used as a base method with the DropMAE pre-trainedencoder and the identity embeddings added.3.1.2.2 Generative Adversarial ModelsGenerative Adversarial Networks (GANs) [47] have greatly impacted com-puter vision society since their introduction. GANs are still widely usedin many subfields such as image generation [50, 103, 160], object detection[89, 115] and semantic segmentation [140, 187] indicating their effectiveness.Conditional GAN typically involves a minimax game mechanism where thegenerator objective is to minimize the log-likelihood while the discrimina-tor objective is to maximize the probability of assigning the correct label asfollows:ℓ = Ex,y∼pdata(x,y)[log(D(x, y))] +Ex∼pdata(x),z∼pz(z)[log(1−D(x,G(x, z)))](3.2)203.1. TAXONOMYSearchRobustVisual FeaturesORRobustVisual FeaturesORNoise VectorGroundTruthMapsGeneratedMapsGANDiscriminatorGANGeneratorGeneratedMapsTemplateFeatureExtractorFeatureExtractorClassificationDecisionFigure 3.5: A general diagram of GAN-based single object tracking meth-ods. The input to the generator is the features of the template image or acombination of the features of the template and the search. The generatoroutput is either a map representing an enhanced set of template appearancesor a robust feature representation depending on the model design. Eitheroutput is fed to a discriminator alongside ground truth maps that representparts of the search image or the template itself. The discriminator mainlyoutputs a classification decision reflecting a final score map or a scaler thatdetermines the quality of a generator output.213.1. TAXONOMYwhere G is the generator that learns to map data x from distribution pdata(x)and z from noise distribution pz(z) to the distribution over y, and D repre-sents the discriminator. A general scheme of GANs in single object trackingis shown in Fig. 3.5.[TGGAN] Generating Reliable Online Adaptive Templates for VisualTracking: This method [49] aims to enhance Siamese tracking by generat-ing new target object views that encourage the Siamese network to have aholistic understanding of different appearances the object might have. Froma GAN perspective, the generator, guided by the ground truth template,generates many templates that simulate the changes in object appearance.At the same time, the discriminator has to verify the quality of the gen-erated templates. Overall, the network at training undergoes two phases.First, A Siamese-based CNN is well-trained on a tracking dataset to guidethe generation process afterwards. Second, the generator and the discrim-inator are trained to meet the GAN training objective and a Task-Guided(TG) objective realized by the Siamese network, hence the name TGGAN.During the GAN training phase, the Siamese network remains frozen. TheSiamese-based CNN operates similarly to a standard Siamese-based trackingnetwork: Given a set of exemplars and a search image, it outputs a similarityscore (inner product) for locations in the search region. During the generatortraining, the discriminator trains to classify whether the generated imagesare authentic or fake. The total loss function combines the GAN loss anda margin contrastive loss [24] for the Siamese branch output. During track-ing, the Siamese network receives the enhanced set of generated exemplarsalongside the current search image and outputs a similarity map.[VITAL] Visual Tracking via Adversarial Learning: This work [139] pro-poses an adversarial training of an existing tracking-by-detection frameworkfollowing the literature [87, 117]. For any tracking-by-detection technique,a detector is applied first, and then a classifier is used to discriminate be-tween the positive target and negative background. This method introducesa generator as an intermediate step while the classifier is viewed as the GANdiscriminator. Flowing in the network during training, a search image getsits features extracted. The features are then fed into the generator. The gen-erator produces a mask, initialized by random noise, that drops out somesearch features to degrade the discriminator the most. If the generator cando so, then the discriminator has learned the discriminative features of thetarget and implicitly how object distractors may appear. It is crucial tonotice that the loss function adopted in this method cures the imbalancebetween the positive and negative samples because most of the classificationmap is background (easy negative). The curation is done via a modulation223.1. TAXONOMYfactor that increases the weight of the positive samples. During trackingtime, the generator is removed, and the discriminator acts like a classifier ina regular tracking-by-detection mechanism.[RCNET] Adversarial Deep Tracking: This work [195] views end-to-endtracking problem as an adversarial training problem. The network comprisesa Regression CNN (R-Net), which acts as a generator, and a ClassificationCNN (C-Net), which acts as a discriminator. As opposed to TGGAN [49],the generator and discriminator are employed in the training and the track-ing. Concretely, the R-Net generator outputs a response map directly insteadof generating samples to be used by another stage of tracking. R-Net expectsa template image Z and a search image X where X,Z ∈ R3xHxW generatesa three-scale response map. The C-Net expects a search image, a templateimage, and a response map, which then generates a classification score thatdetermines the response map’s quality. The template is updated every nframe. The C-Net is fine-tuned using the ground truth from the first frameand can select the best response of the three response maps generated byR-Net. The objective is to optimize the GAN objective in equation 3.2 inaddition to another ℓ2 loss between the ground truth response map and theR-CNN generated response map.[OnlineGAN] Online Object Tracking based on Generative AdversarialNetworks: This paper [185] adopts the same idea of VITAL [139] where agenerator is employed to drop features from the discriminator input to makeit robust against distracting features. The main difference is that the outputfeatures of the generator are two scaled feature maps instead of one. Hence,the generator generates two adaptive dropout masks that operate on twoinput feature maps obtained from the first three layers of the VGG featureextractor. The discriminator concatenates the two feature maps to outputone classification score. At the end of training, the two masks should learnhow to cancel out distracting features. Subsequently, the feature maps can betransformed into score maps that accurately determine the object’s location.This is done via regression and classification CNNs.[SGAN+FGAN+RD] Robust Visual Tracking based on Adversar-ial Unlabeled Instance Generation with Label Smoothing Loss Regulariza-tion: This method [51] combines ideas from TGGAN [49] and VITAL [139].The network performs tracking-by-detection by exploiting three networks:Sample-level GAN (SGAN), Feature-level GAN (FGAN), and a reliableRe-Detection (RD) module. The overall goal is to use the discriminatorof the FGAN to determine the best object candidate among a set of se-lected candidates from the search image based on the tracking-by-detectionparadigm. Therefore, during training time, the SGAN is trained to gener-233.1. TAXONOMYate diverse training samples that mimic various types of changes affectingthe appearance of the object, such as motion blur and shape deformation.Consequently, the FGAN uses the samples obtained from SGAN to trainits generator to provide dropout masks while training its discriminator toprovide a classification score corresponding to locations in the search image.Additionally, a label smoothing regularization loss is introduced to com-bine the unlabelled images generated by the SGAN with real images. TheSGAN-generated images are set to have a uniform distribution per class,hence avoiding overfitting. Finally, the re-detection algorithm applies a clas-sification score threshold to determine the reliability of the score map. Itproposes that new regions of the search image be flowed in the FGAN againuntil reliable conditions are met before the final output is obtained.[AFSL]Adversarial Feature Sampling Learning for Efficient Visual Track-ing: In a typical tracking-by-detection setting, many region proposals aresampled from the search image using an area around the target location inthe previous frame. This method [184], proposes sampling in the featurespace using Spatial Bilinear Resampling (SBR) [144] which mainly samplesregion proposals from the feature maps obtained from the feature extrac-tor. Although sampling from embedded feature maps is beneficial regardingFPS speed, it involves many overlapping features in the feature space. Theauthors propose sampling from two different feature spaces and fusing thesampled features. This way, the sampled features capture semantic relationsand target appearance variation. A GAN in this method mitigates the effectof overlap between samples by dropping out some samples by leveraging theadaptive dropout algorithm provided by VITAL [139]. The GAN genera-tor selects the mask with the largest loss to feed into the discriminator toenhance discriminative feature detection.[GARAT] Generative Adversarial Learning for Robust and AccurateTracking: This work [181] enhances the Siamese tracking pipeline by remov-ing distractors in the search image. The network comprises a Siamese CNN[5] and a GAN. During training, the GAN is trained first to enable its gener-ator to produce a set of distractors that mimic the appearance change of thetarget object. Distractor ground truth images are obtained from the trainingset by subtracting the ground truth target from the search image. DuringGAN training, the GAN discriminator receives the generated distractors andthe ground truth distractors to output a classification score indicating thequality of the generated distractors. Consequently, the Siamese network istrained on search images without the distractors by subtracting the searchimage from the distractors generated by the generator. During tracking, thegenerator, the only GAN part used in this phase, cancels the distractors243.1. TAXONOMYfrom the search image. Finally, a distractor-free image is fed to the Siamesenetwork.3.1.2.3 Diffusion ModelsSince their introduction [136], thermodynamics-inspired diffusion modelshave impacted the computer vision society. Based on this idea, denois-ing diffusion models were introduced [58] to mark the start of a new era inimage generation. Henceforth, denoising diffusion models have been ubiq-uitous in many computer vision applications such as deep image generation[34, 48, 59], semantic segmentation [122, 179], object detection [17] and re-cently multi-object tracking [108].[Diff-SiamRPN] Siamese Network for Object Tracking with DiffusionModel: This work [191] integrates the concept of Denoising Diffusion ImplicitModels (DDIM) [138] into SiamRPN [87]. Firstly, a DDIMmodel generates nimages prompted by the template image. These images are assumed to enrichthe object appearance modelling by integrating many scenarios in which thetarget object can be, such as different lighting conditions or multiple poses.Finally, the SiamRPN network is trained using the generated images astemplates by fusing the n templates in one template map. Notably, theDDIM is only trained on the general-purpose ImageNet-1K dataset [32] andis prompted with template images from single object tracking datasets suchas GOT-10K [62] and LASOT [38].3.1.2.4 Variational AutoencodersA Variational AutoEncoder (VAE) [70] is a special type of an autoencoder inwhich the model is prompted to reconstruct a given input x using a learnedlatent variable z, while the latent variable is restricted to have a normaldistribution of certain parameters µ and σ. The rationale behind such atraining paradigm is that it is often easier to manipulate the latent vari-able. In a typical setting, the learned manifold in the latent space generatesnew examples that diversify the training dataset. VAE maximizes the datamarginal distribution by maximizing its variational lower bound. The lowerbound for a data point xi is given by:L(θ, ϕ;xi) =∫qϕ(z|x)(log pθ(xi|z) + log pθ(x)− log qϕ(z|x)) dz (3.3)where qϕ(z|x) refers to the encoder parameterized by ϕ, pθ(xi|z) refers tothe decoder parameterized by θ. The previous equation yields the training253.1. TAXONOMYSearchConditionMapµσµ+ σN + CGeneratedScoreMapTemplateFeatureExtractorFeatureExtractorFeatureExtractorVAEEncoderVAEDecoderFigure 3.6: A general diagram of A VAE in single object tracking. The dot-ted path is the condition used to direct the CVAE, which is only mandatoryin the case of a CVAE. In the diagram, N = N (0, 1) is a Gaussian noisevector.objective for VAE (see [70]):min ||x− fϕ(z)||2 +KL(qϕ(z|x)||pθ(z)) (3.4)where ||x−fϕ(z)||2 is the reconstruction error andKL is the Kullback–Leiblerdivergence [80] between the distribution of the encoded data qϕ(z|x) and themarginal distribution of the latent variable pθ(z) which is usually assumed tobe Gaussian with mean µ and variance σ. For a long time now, VAEs havebeen used in a wide variety of computer vision applications ranging fromgeneration [4, 11, 124] to classification [13, 19] and single object tracking isno exception. A representative diagram of VAE adaptation in single objecttracking is shown in Fig. 3.6.[SINT++] Robust Visual Tracking via Adversarial Positive InstanceGeneration: This work [159], builds upon Siamese INsTance Search (SINT)[148] which uses a Siamese network to search for the first frame templateappearance in the next frames of a given video. Overall, this method aimsto enhance the robustness of SINT by generating hard positive samples. Forthat sake, the network comprises two novel components: A Positive SampleGeneration Network (PSGN) and a Hard Positive Transformation Network(HPTN). To be succinct, PSGN, the only part implementing a VAE, isdiscussed here. PSGN is a VAE trained using the VOT dataset [74] to extracta compact object representation from the dataset. This compact manifold263.1. TAXONOMYz generates more diverse training examples to account for unseen scenarios.After generating diverse samples, these samples are fed into HPTN to betransformed into harder samples, and the final training samples are fed intoSINT for training.[VAE-MCMC] Robust Visual Tracking Based on Variational Autoen-coding Markov Chain Monte Carlo: This research [81] proposes trackingwith the aid of supporters. Supporters are assumed to be objects whosemotion is linked to the target object. They can help locate the target be-cause they are geometrically correlated with it over time. The objective ofthis method is to find the best target state X and supporters’ states S tomaximize the marginal probability. The MCMC is used to formulate thejoint marginal p(X,S) using proposal and acceptance [69]. VAE-MCMC de-composes the proposal required for p(X,S) into two proposal terms, each ofwhich is only a function of either the supporters’ states or the target state.When dealing with the acceptance phase, the marginal probability of sup-porters’ states p(S) appears in the equation. Although it is possible to reuseMCMC, it is impractical due to the large number of samples required to de-scribe p(S). The authors suggest using variants of VAE that optimize priors[150] to marginalize p(S) by decomposing the problem to the optimizationof pV AE(z|S), done via the encoder, and qV AE(S|z), done via the decoder.[SiamVAE] Semi-Supervised Visual Tracking Based on Variational SiameseNetwork: This work [173] modifies the objective of SiamFC [5] from predict-ing a score map to predicting a Gaussian heat map. First, both templateand search branches are readjusted to have their intermediate feature mapsrepresenting the mean and variance of the image distribution. Consequently,the template branch outputs the mean and variance of the latent represen-tation (i.e., feature maps) distribution similarly for the search branch. Theobjective is to close the gap between the search and template distributionsusing Wasserstein distance [33], which emits gradient signals from the meansquared error between the search and template distribution parameters µ, σ.This formulation results in a smooth similarity measurement instead of rely-ing on sharp bounding box measurements, enhancing model generalizability.To stabilize training and increase the model’s performance, the feature ex-tractor is trained in a VAE setup. This setup allows the model to capturethe underlying distribution before the supervised phase.[Depth-aware] Depth-aware Object Tracking With a Conditional Vari-ational Autoencoder: This research [63] benefits from depth informationprovided in CDTB dataset [107] using a Conditional VAE (CVAE) [137].The network is composed of three subnetworks. The main subnetwork is aSiamese cross-correlation network. In another subnetwork, VAE is leveraged273.1. TAXONOMYin a prior VAE that consists of a prior encoder and a mask decoder. Lastly,a subnetwork of a posterior CVAE consists of a depth-conditioned posteriorencoder and a depth decoder. While training, all three subnetworks con-tribute to the training objective in the following way: First, the search andtemplate features are extracted and cross-correlated using the Siamese net-work. Following that, the posterior encoder takes in the search image andthe depth map, which represents the condition, and outputs a sample con-catenated with the Siamese cross-correlation. The concatenation output isfed into the mask and depth decoders to output a segmentation mask and adepth map, respectively. The training objective is to minimize three terms:The cross-entropy between the ground truth segmentation mask and thepredicted one, the mean squared error between the ground truth depth mapand the predicted one, and the KL divergence between the prior distributionand the posterior distribution in the latent space. It is worth mentioningthat the posterior CVAE is removed during tracking. To produce the finaloutput, a sample is taken from the prior encoder to be concatenated withSiamese cross-correlation and fed into the prior mask decoder.3.1.3 Learning Paradigms3.1.3.1 Self-Supervised LearningBecause deep learning models require a vast amount of labelled trainingdata and acquiring these labelled data is equally exhaustive for humans andmachines, self-supervised learning was introduced. Self-supervised learningaims to exploit underlying implicit relations in the data. This paradigm hasbeen explored in video object segmentation [202], which is usually treatedas a twin field to visual tracking with different datasets and applications.Many self-supervised learning methods have been extended to videos basedon exploiting temporal relations between video frames. For instance, VFS[172] is self-supervised pre-trained to achieve a video frame correspondenceobjective on the video dataset Kinetics [67]. Moreover, in [154], a modelis prompted to predict colours in frames based on previously seen videoframes. In this survey, we explore self-supervised techniques for visual objecttracking. Usually, training with a self-supervised objective entails a cycle-consistency constraint, either explicitly or implicitly. A cycle is defined asa forward-tracking path followed by a backward one. Given a target patchPt, a predicted patch by forward tracking Ppf and a predicted patch bybackward tracking Ppb, a cycle-consistency objective encourages that:Ppf ≡ Pt (3.5)283.1. TAXONOMYwhere the identical shape and location of the equivalent patches characterizethe equivalence.[D-EM] Self-Supervised Learning for Visual Tracking and Recognitionof Human Hand: Despite being relatively dated, this work [169] laid downthe mathematical fundamentals for an approach to this family of trackers.The proposed mathematical formulation for object tracking leverages theExpectation-Maximization algorithm [31], which can be reformulated as themore recent maximum likelihood estimation for classification. The proce-dure goes as follows: First, a tracker in the form of a classifier is trained ina supervised manner on a small labelled dataset. After that, the trained un-reliable classifier calculates the distance between newly acquired unlabelleddata samples and those available from the previous stage. This stage iscalled the discriminative stage. Hence, the method is called DiscriminantExpectation-Maximization (D-EM). Having the new samples clustered withthe least distance samples from the small dataset, a new big dataset is formedto train a higher-quality classifier, which is the tracker in this case.[Joint-Task] Joint-task Self-supervised Learning for Temporal Corre-spondence: The main assumption in this method [94] is that for any twoconsecutive-in-time feature maps, there has to be a linear transformationrepresented by an affinity matrix that maps the features and locations fromone feature map to the other. This method obtains the affinity matrix by asimple dot product followed by a softmax operation on the corresponding fea-ture maps. Learning to match features requires sampling the correspondingframes, which involves many issues, such as staticity and the absence of thetarget frame. To alleviate the effect of these issues, a two-stage frameworkis introduced, where two affinity matrices Afp and App are learned. Theseaffinity matrices represent the linear transformation applied to the shapeand the location. Afp represents a transformation mapping the referenceframe to the target patch, while App represents a transformation mappingthe target and current patches. First, a region-level localization network usestwo greyscale inputs: reference patch and search frame. This phase aims todetermine the location transformation that, when applied using the learnedAfp, a new patch of the target object is located in the search frame. Second,fine-grained matching is applied using the colour information. During thisphase, App is obtained through the target and cropped search patches. Fur-thermore, an autoencoder is employed to generate the coloured target patchwith the encoder prompted by the current search patch and the decodermixing the latent representation with the App affinity matrix. The train-ing objective encompasses two regularization conditions. First, the cycle-consistency constraint, which in the context of this method can be achieved293.1. TAXONOMYby the orthogonality of the affinity matrices alongside the location matchingfrom the predicted and original locations of the object in the same framecyclically. Second, the concentration constraint necessitates pixels from thesame object not to drift from its center, moving as one unit over time.[CycleSiam] Self-supervised Object Tracking with Cycle-consistent SiameseNetworks: This method [186] employs the cycle-consistency described byequation 3.5 in SiamRPN [87]. The basic idea is to establish forward andbackward tracking passes for many frames at various timesteps. The lossis calculated upon the completion of each cycle. The loss function containstwo components, one for bounding box consistency and another for trackedobject mask consistency. It is worth mentioning that this method is trainedusing labels (i.e., supervised learning) to identify the object location in theinitial frame. However, the training was done on a general-purpose Ima-geNet challenge [128], and its performance was exclusively evaluated on theVisual Object Tracking (VOT) dataset [74].[S2SiamFC] Self-supervised Fully Convolutional Siamese Network forVisual Tracking: This work [134] harnesses SiamFC [5] to a novel self-supervised approach. The pipeline incorporates anti-clutter sample weight-ing, adversarial appearance masking, and hard negative mining. First, train-ing pairs are extracted from images by randomly selecting an area of an im-age to be the template and then selecting a larger area of the image aroundthe template to be the search. This work proposes an anti-clutter weight-ing algorithm to reduce the effect of areas that contain patterns lackinginformation. The anti-clutter algorithm assigns importance weights to eachtraining sample following the fact that the response map of the Siamesenetwork contains few positive responses for informative samples as opposedto many positive responses for cluttered samples. Second, to enhance thediscriminative abilities of the model, a part of the template of each trainingsample is masked. The masking is performed in a gradient-guided manner[90, 129, 200] by propagating the gradient of the response to locate the partsof intermediate feature maps that are most salient. The saliency channelsare then thresholded, and one is randomly selected, upsampled and appliedto the template to mask out an important part of the template, forcing themodel to calculate the correct response in the absence of the template andits presence. Finally, in another stage of training, images from the sameclass but with different appearances of a template in a training sample aremined using K-means clustering to be used as hard-negative samples.[UniTrack] Do Different Tracking Tasks Require Different AppearanceModels?: Although this method [162] does not train an end-to-end networkin a self-supervised tracking task, it paved the way for generic self-supervised303.1. TAXONOMYpre-trained models to unify different tracking tasks. This work suggeststhat all tracking tasks such as SOT [167], VOS [123], MOT [111], MOTS[152], and PoseTrack [1] require a base model that transforms images intoinformative feature maps that capture discriminative patterns. The basemodels, which have been empirically proven effective, are self-supervisedmodels trained on general-purpose image and video datasets. Experimentsconducted by the authors prove that VFS [172] is the best performant basemodel. Moving on with the base model embeddings, the tracking tasksare categorized into propagation (SOT, VOS) and association tasks (MOT,MOTS, PoseTrack). Cross-correlation is adopted as the relation modellingtechnique for our scope of propagation tasks. Different heads are employedfor the final output layer to meet the requirements of different tasks (e.g.,bounding box head for SOT, mask head for VOS, etc.).[Crop-Transform-Paste] Self-Supervised Tracking via Target-AwareData Synthesis: As the title suggests, this work [93] provides tracking modelswith different data augmentations that are suited for the visual trackingproblem. Given a training sequence, first, the target is cropped out of theinitial frame using the ground truth bounding box. Following that, the targetundergoes transformations that simulate the challenges present in the singleobject tracking datasets. These challenges include motion blur, occlusion,scale change, illumination variation, and deformation. The challenges canbe represented by transformations, including blur, cutout, rescale, colourjitter, and shear, respectively. Lastly, the cropped and transformed targetis pasted on a random location of random images of the same sequence.Henceforth, more data examples are obtained with information miming thedifferent conditions existing in single object tracking datasets.[ETC] Exploiting Temporal Coherence for Self-supervised Visual Track-ing by Using Vision Transformer: The main hypothesis in this work [205]is that there exist characteristics in the target object that are invariant toviewpoint change and deformation. The objective of this method is to ex-ploit the intrinsic supervisory signals in videos. The tracking problem isformulated assuming that two types of Samples exist: Unlabelled Samples(ULS) and Few Labelled Samples (FLS). On the one hand, while trainingusing ULS, no ground truth location of the target object is given. On theother hand, the initial location of the target object is given in FLS. Althoughframes are sampled equally for ULS to exploit the potential of unlabelledimages better, they are sampled variably for FLS to use the label infor-mation. Two types of relations between frames are assumed to exploit thetemporal coherence: intra-frame (frame-level) and inter-frame (sequence-level). Training sequences are represented by palindromes of frames where313.1. TAXONOMYsome T frames are reflected and appended to the training sequence to contain{Fi, Fi+1, Fi+2, ..., FT , ..., Fi+2, Fi+1, Fi} to train a cycle-consistent transformer.The main contribution in this work, the cycle-consistent transformer, im-plements equation 3.5 using an encoder-decoder architecture. An essentialstep before the transformer is encoding all frames in the palindrome usinga CNN. After that, the encoder encodes the template representation of Fiand the search representation of Fi+1 and feeds a joint representation forthe decoder, which predicts a representation for the template in Fi+1 whichin turn is used as a template representation in the next transformer block.Furthermore, the training objective is two-fold. One is to ensure the finalresult cycle consistency, and the other is to ensure the consistency of inter-mediate results. For instance, in a given palindrome, where template groundtruths are {F gti , F gti+1, F gti+2, ..., F gtT , ..., F gti+2, F gti+1, F gti } and predicted targetsin search images are {F si , F si+1, F si+2, ..., F sT , ..., F gti+2, F si+1, F si }, the model istrained such that:F si ≈ F gti , ..., F sT ≈ F gtT (3.6)[QCT] Consistency-based Self-supervised Visual Tracking by Using QueryCommunication Transformer: The same method of [205] is reintroduced inthis work [206] changing the transformer architecture to propagate the tem-plate and search embeddings as query information between instances of aquery interaction module. This method leverages the transformer architec-ture to enhance the model’s performance based on [205]. This enhancementis realized by the transformer’s capability to capture long-term dependenciesrepresented by the query token.3.1.3.2 Unsupervised LearningIn this category, the labels given in the tracking dataset are entirely excludedfrom the training process. As opposed to self-supervised learning, supervi-sory signals are not obtained from the ground truth in any training phase.Instead, the gradient signals come from an intrinsic source of information tothe data. For instance, given an image in a tracking setting, an unsupervisedmethod aims to teach the model how to track an arbitrary object or pattern,in some cases, not only a given labelled object.[UDT] Unsupervised Deep Tracking: This work [156] trains an unsu-pervised Siamese model using the cycle-consistency concept introduced insection 3.1.3.1. The notable difference between self-supervised methods andthis method is that they use the labels to localize the object initially. Incontrast, this method uses random patterns in the images to align with the323.1. TAXONOMYunsupervised learning paradigm. To further improve the localization accu-racy, more frames can be inserted in the cycle where the result templatefrom a previous frame is used to search in the next frame. Moreover, a cost-sensitive loss is introduced to eliminate noisy and uninformative patches thatmay result from random cropping. To account for noisy patches, all patchesare ranked for their loss values, and the highest 10% of them do not con-tribute to the final loss. Considering informative patches, a weight is givenaccording to motion by calculating the mean squared distance between theground truth synthesized labels and the response map. If the distance islarger than a certain threshold, then the patch is probably informative.[PUL] Progressive Unsupervised Learning for Visual Object Tracking:This method [166] implements an unsupervised way of training a Siamesetracking network such as SiamFC [5] and SiamDW [194]. The unsupervisedtraining pipeline involves three main components: Background discrimina-tion, temporal mining, and temporal correspondence. For background dis-crimination, first, EdgeBox [208] object detection model is employed to ex-tract 64 object proposals. Inspired by contrastive learning approaches ofSimCLR [18] and MoCo [53], each proposal is then augmented in two views,and the model is required to learn to assign high similarity between the sameobject views and low similarity between different objects. Furthermore, hardnegative mining facilitates feature learning and ensures non-trivial trainingobjectives. In essence, given an epoch E, hard negative mining is performedsuch that each batch contains T nearest neighbours of a positive patch ziaccording to the learned similarity measure in epoch E−1. In particular, themodel is first trained for 20 epochs without hard negative mining; then, sim-ilarity calculations are carried out using the learned similarity through the20 epochs. For temporal mining, a random initial frame and a tracking pat-tern are randomly picked from the frame. Following that, the backgrounddiscrimination model is then employed to track the picked pattern in thefollowing set of frames. The picked pattern and the last tracked instanceof the pattern collectively form a set representing the output of the tem-poral mining stage. For temporal correspondence, given that the temporalmining result depends on the training of background discrimination and thelocation of the randomly chosen pattern in the initial frame, the temporalmining result is certainly noisy. Therefore, a noise-robust loss is introducedto substitute the binary cross-entropy. The noise robust loss accounts forthe first moment (i.e., mean) and the second moment (i.e., variance) of thetarget maps based on the assumptions that the likelihood of a target map isthe Bernoulli distribution of the pixel location given some noise ϵ. Moreover,this likelihood can be approximated using the second-order Taylor approxi-333.1. TAXONOMYmation.[ContrastiveCycle] Contrastive Cycle Consistency Learning for Un-supervised Visual Tracking: This work [203] revisits many aspects of un-supervised tracking ranging from sampling training frames to the trainingprocess itself. The proposed unsupervised paradigm involves a samplingstrategy, backward tracking, and contrastive learning. First, this work em-phasizes that the sampling strategy should deal with the trade-off betweenappearance variation and temporal motion information. Although randomlysampling frames with a large interval between sampled frames helps increaseappearance variation, it also results in the collapse of the temporal flow inthe video. The sampling strategy followed in this work is to skip some frameswhere the exact number of skipped frames is determined empirically. Sec-ond, the backward tracking method is a step-by-step strategy. Particularly,for a set of frames, each frame is used in the forward and the backwardpasses. In the forward pass, a template is extracted using the Gaussian re-sponse from the previous step as a pseudo-label. Subsequently, a Siamesenetwork is applied to create another Gaussian map representing the searchedobject. The same happens for the backward pass. Hence, each intermedi-ate tracking result is considered when calculating consistency loss. Third,this work presents a novel contrastive learning approach to fit the trackingproblem. Namely, data augmentation is presented between two differenttemplate instances of the same target object. In practice, this way of dataaugmentation may slow the convergence, therefore, contrastive learning isperformed only in the last step of the backward pass encouraging responsemaps of different template views and the same search input to be the same.Finally, a conclusion worth mentioning is that not all known types of dataaugmentations fit in the tracking setting. Specifically, a data augmentationoperator for tracking should avoid destroying the smooth temporal transitioninformation. Hence, the Gaussian blur, noise and brightness augmentationoperators perform favourably against other operators such as rotation andcutout.3.1.4 Meta LearningBy exposing the model to diverse tracking scenarios within a comprehensivedataset, meta-learning empowers it to ”learn how to learn.” During tracking,the model leverages this meta-knowledge to rapidly adapt to a new targetwithin a limited number of initial frames [40]. This adaptation may involvefine-tuning internal parameters or selecting an optimal tracking strategybased on the object’s specific characteristics.343.1. TAXONOMYMeta-learning offers several advantages for single object tracking. First,it alleviates the need for extensive training data encompassing every ob-ject category. Second, it enables the tracker to handle unforeseen circum-stances through on-the-fly adaptation. Finally, meta-learning has the poten-tial to enhance real-time performance by facilitating the acquisition of effi-cient adaptation strategies. While ongoing research is crucial, meta-learningdemonstrates promise as a powerful tool for developing robust and adaptablesingle-object tracking algorithms.[Meta-Tracker] Fast and Robust Online Adaptation for Visual ObjectTrackers: This method [120] addresses the problems of initializing the set oftrainable parameters θ0 and finding an adaptive learning rate α. Concretely,given random initial parameters θ00 and α0, an initial frame x0 and a frameinterval δ, the model is prompted using the initial frame x0. Afterwards, afterone iteration, another frame xδ is sampled as a testing frame to evaluatethe quality of the predicted parameters. Subsequently, the model runs Titerations of optimization to find optimal θT0 . These values are consideredconstant when training the tracker to obtain a new set of parameters θiwhere i ̸= 0. For any subsequent frame, the same algorithm can be used inupdating α. However, it is found that α tends to be large when optimizedfor very few frames, thus leading to unstable training. Consequently, α isfixed for the subsequent frames after it is obtained through optimization inthe initial step.[GradNet] Gradient-Guided Network for Visual Object Tracking: Thisresearch [91] aims to find the optimal template representation through meta-optimization. The optimal template embeddings are generated through aprocess of three phases: Initial template embedding, loss gradient calcula-tion, and template embedding update. First, a Siamese network is employedto extract template and search embeddings and obtain the regular scoremap. Thereafter, an initial loss is calculated. The loss gradient w.r.t. ashallow layer of the template embedding branch is calculated. The templaterepresentation is then updated using an upsampled representation of theloss gradient and template embeddings from the shallow layer. Lastly, thefinal loss is calculated between the Gaussian pseudo label [5] and the cross-correlation between the updated template embeddings and search embed-dings. It is worth mentioning that the network tends to ignore the templategradient when updating multiple frames simultaneously. In that manner, asingle template is employed to search different frames from different videosto ensure the utilization of discriminative gradients that adjust the templateembeddings to perform effectively across all search images.[MLT] Deep Meta-Learning for Real-Time Target-Aware Visual Track-353.1. TAXONOMYing: This work [22] can be considered one of the pioneering meta-learningtechniques for object tracking. In particular, it employs a loss gradient as ameta-representation. The architecture comprises two networks: A Siamesematching network and a meta-learner. The Siamese matching network em-beds the template image and search images and finds the correlation be-tween the embedded representations. More concretely, the gradient of theloss w.r.t. the weights in the last layer in the matching network is used asan input for the meta-learner. During training, the meta-learner uses thegradient to infer new weights specific to the target object. The inferredweights are then concatenated with the last layer’s weights again to predicta target-aware response map. Considering the computational burden, themeta-learner is not run using every frame for the tracking phase. Instead,the past tracking results are saved in memory if the model is confident aboutits tracking result. After T frames pass, a sample of M frames with mini-mum entropy is drawn from the memory. After that, these frames are usedas input for the matching network to obtain the mean of the loss gradientof the last layer. The gradient is then fed into the meta-learner, which de-duces the new weights to be concatenated in the last layer of the matchingnetwork. Overall, the meta-learner is used to direct the matching networkusing the target information from confidently predicted targets.[DIMP] Learning Discriminative Model Prediction for Tracking: Thisresearch [7] depends on a combination of meta-learning approaches. Themodel comprises a feature extractor, a discriminative convolutional layer anda model predictor. At a glance, the overall idea is that once target and searchfeatures are extracted, target features are fed into the model predictor, whichinfers the weights of the discriminative convolutional layer. Shifting towardsmeta-learning techniques, first, a meta-representation choice is made for theloss function. The loss function is designed to ensure discrimination betweenforeground and background. Given a feature map output of a feature extrac-tion network, the loss aims to learn the parameters of the loss function suchthat the feature map matches a ground truth object center map. Further-more, the ground truth map is considered part of the meta-representation.Second, the learning rate is learned during the optimization in a data-drivenapproach using the steepest descent. Third, for initiating the weights of thediscriminative convolutional layer, a model-agnostic meta-learning approach[40] is used. The bounding box is predicted using the overlap maximizationapproach by [29]. During online tracking, augmented views are extractedfrom the first annotated frame and used as input to the model predictor toinitialize and optimize the convolutional discriminator. The discriminatoris optimized for other frames when a distractor peak is detected or after363.1. TAXONOMYpassing a certain number of frames. A memory of the 50 best-predictedframes is maintained at each run to ensure the optimal weight update forthe discriminator through the model predictor.[FCOS-MAML + RetinaMAML] Tracking by Instance Detection:This work [155] considers the single object tracking problem as detectingthe same instance of the same object in all the frames within a given se-quence. The basic idea is to select an efficient detector and propose a goodinitial point for the detector parameters θ0 so that it adapts quickly to newframes given the annotations of the first frame. The offline training is splitinto inner and outer optimization loops as in MAML [40]. A training (sup-port) image is sampled in the inner optimization loop, which is executed forfour iterations. Using the weights optimized by the inner loop, test (query)images are used to validate the generalization ability of the fitted detectorin the outer optimization loop. During that, the query images are used toupdate the initial weights to fit the whole dataset. Despite borrowing manyideas from MAML, different design choices were made. One choice worthmentioning is a kernel-wise learnable learning rate, where each convolutionkernel is associated with a learning rate that is optimized alongside the ini-tial weights θ0. Additionally, online domain adaptation is performed duringtracking. The model is trained online using the annotated first frame andgradually expands the support set as a memory containing the 30 best framesaccording to a preset threshold on the classification score. A pivotal choicein the backbone is to reduce the online and offline trainable weights as themodel learns the initial weights and the learning rate. This choice mitigatesthe problem of overfitting to the very few examples given in the current for-mulation. The most important aspect, however, is the detector selection. Inthis work, two single-stage anchor-based (RetinaNet) [101] and anchor-free(FCOS) [149] detectors are selected. Experiments show that anchor-baseddetectors have an advantage over anchor-free ones.[LMTU] High-Performance Long-Term Tracking with Meta-Updater:Although gears are shifted to long-term tracking when considering this method[28], it comprises many techniques that apply to the short-term single objecttracking as well. The main component of our concern is the meta-updater.The meta-updater is a component that decides whether the tracker can up-date its current template frame. The meta-updater is a three-layer LSTMnetwork. The input for this network is concatenated vectors of different rep-resentations of the target at the current frame. These representations arethe geometric information embedded in the bounding box, discriminationinformation represented by the maximum value of the response map, andappearance information represented by the L2 difference between the target373.1. TAXONOMYappearance in the first frame and the current frame. Furthermore, LSTMcells exploit the sequence information for different sequence frames. Themeta-learning aspect of this method is deployed by having a tracker runningon the sequence frame and having all mentioned representations recordedfor inner optimization of the meta-updater. The meta-updater is trained ina supervised way where the labels for the update decision variable are either1 or 0. A label is 1 if the IoU between a predicted box by the employedtracker and the ground truth box is greater than 0.5 and 0 if the IoU is ex-actly 0. The potential inputs associated with predictions having IoU greaterthan 0 and less than 0.5 are not considered in the training process for fasterconvergence.[CLNet] Compact Latent Network for Fast Adjusting Siamese Trackers:This work [35] hypothesizes that the Siamese trackers tend to misclassifysamples of the testing (tracking) data due to the bad choice of the deci-sion boundary that separates the training data. The rationale behind thetracker’s behaviour is that it does not sample enough decisive positive andnegative samples from the available testing data in the first frame. In otherwords, the Siamese trackers lack the ability to fully leverage the annota-tions in the first frame, resulting in missing information about the wholesequence. To alleviate this effect, many positive and negative samples mustbe drawn from the first frame alongside the training samples from the otherframes. Considering the easy solution of training on training samples andfine-tuning on testing samples, the small amount of testing data may leadto overfitting, and the same decision boundary problem may appear againin the training data. This work experiments with the previously mentionedassumptions on SiamRPN++ [86]. The proposed solution to the aforemen-tioned problem is a compact latent network that adapts the weights of thelast layer of the classification and regression branches to the testing samples.The compact latent network takes positive and negative samples from a hid-den feature map, then predicts their mean µ and standard deviation σ andfinally concatenates µ+, µ−, σ+, σ− into a compact latent variable that isused to update the last layer’s weights. The primary assumption in this workis that a Gaussian distribution represents the testing and training samples.Therefore, if we can incorporate the information about the distribution whiledetermining the decision boundary, then the decision boundary will consideralmost the whole population in the distribution, not only the sampled points.From a meta-learning perspective, this method can be seen as MAML [40]with a carefully designed subnetwork that considers assumptions about thetracking problem.[MGA-Net] Meta-graph Adaptation for Visual Object Tracking: This383.1. TAXONOMYmethod [165] aims to adjust the loss gradients of certain layers of a selectedtracker before using them to update the weights of these layers. Overall,The process of obtaining the refined gradients comprises two stages. Thefirst is embedding the gradients into proper low-dimensional space. Thesecond is combining the embedded gradient representations with an explicittarget representation. First, the gradient embedding phase involves buildinga graph of gradients, where the vertices are the gradients themselves and theedges are the similarities between the weights corresponding to the gradients.Using the constructed graph, a graph neural network embeds the graph intoa high semantic representation concatenated with the target embeddingsby a shallow encoder. A filter adaptation network combines the gradientsand the template embeddings. First, it concatenates the gradients and thetemplate representations and encodes them again. After that, the encodedjoint representation is decoded into new refined gradients. The meta-traininginvolves running the tracker, obtaining the gradients, and then updating thegradients in an inner loop using the gradient embedding and filter adaptationmodules. During online tracking, the inner optimization loop for gradientsis performed once every T frames using the most reliable results obtainedwhile running on the previous T frames. This update schedule optimizes theusage of the computational resources and mitigates overfitting to a specificframe in a sequence.[FastAndRobust] Fast and Robust Visual Tracking with Few-IterationMeta-Learning: This method [96] introduces MAML into a novel trans-former architecture. The main idea is to learn different initial parametersand learning rates for the classification and regression branches. This isbased on the assumption that both tasks require different adaptations. Dueto this complexity, the authors employ [37] to simplify the learning objectiveof determining the bounding box using a triplet of key points. Moreover, themodel supports online training to be able to adapt the model to new framesusing a few iterations of Conjugate Gradient (CG) and Newton’s method[29] and [130].[RTS] Robust Visual Tracking by Segmentation: This work [121] as-sumes that visual tracking with a bounding box prior is not robust to vari-ations such as occlusion because the object boundary is ill-defined by theparameters of the bounding box. To serve the purpose of segmentation-centred tracking, the model adapts a segmentation method called LearningWhat to Learn (LWL) [8]. Mainly, RTS consists of two branches: a segmen-tation branch based on LWL and an instance localization via the classifica-tion branch. Both branches adapt similar architecture but for different tasks.On the one hand, the segmentation branch aims to learn high-quality dis-393.1. TAXONOMYcriminative masks. On the other hand, the instance localization branch aimsto remedy the lack of robustness of the segmentation masks when dealingwith long and challenging tracking sequences. In the segmentation branch,a meta-learning model learns the weights of a segmentation feature encodergiven a previous memory of successful segmentation masks. In the instancelocalization branch, another meta-learning model learns the weights of aninstance classification model using DIMP [7], which adopted a loss functionfor robustness in the case of tracking videos. The feature maps producedby both branches are aggregated through a weighted addition and fed intoa decoder. The decoder, conditioned by both branches, finally outputs asegmentation map of the tracked object. Both branches are trained jointly,with the segmentation branch updated on every iteration and the instanceclassification updated on the first iteration only of the inner optimizationiterations.3.1.5 Continual LearningTypically, continual learning approaches are intended for long-term tracking,which is outside the scope of this survey. However, the continual learningmethodology is useful even for long, short-term sequences.[TrackingViaContinualLearning] Long-Term Visual Object Trackingvia Continual Learning: This work [190] applies continual learning via knowl-edge distillation [57]. Namely, the model consists of two main components:a dynamic memory and a continual learning module. The dynamic memoryis divided into positive samples short-term, positive samples long-term, andnegative samples short-term sub-memories. The continual learning moduleis trained through a classification loss and a continual knowledge distillationloss. In the warm-up phase of training, the model is trained using the posi-tive and negative samples of the short-term memory only and a cross-entropyclassification loss is computed. In the main stage of training, the joint train-ing, the previously warmed-up model weights are frozen, and a new modelis instantiated to be adapted to the positive samples from the long-termmemory. In this stage, the old model is used as a ground-truth provider,and the cross entropy between the predictions of the new model and the oldmodel’s predictions is minimized as a knowledge distillation component inthe total loss. The complete knowledge distillation algorithm is the same asthe Learning without Forgetting (LwF) algorithm [98]. The last piece of themethod is updating the long-term memory of positive samples. This is doneby comparing the highest scaler value of the response map to a threshold.The update happens if the value is less than a threshold. The memory is403.1. TAXONOMYalso updated every 10 frames.[Con] Visual Tracking by Adaptive Continual Meta-Learning: This re-search [23] exploits the same paradigm of knowledge distillation. In par-ticular, the model comprises a meta-learner and a tracking baseline. Themeta-learner aims to learn the following hyperparameters: the learning rate,the weighting parameters for knowledge distillation, current instance com-ponents of the loss, and one current-instance-loss-specific hyperparameter.For that purpose, the given dataset is divided into four episodes containingconsecutive frames. The first episode is responsible for the initial adaptationof model parameters. Using the remaining episodes, the model runs inneroptimization iterations on a given episode to obtain an initial prediction ofthe model weights while distilling the knowledge of the previous episode.Formally, given episodes D1, D2, D3 and D4, the inner optimization outputis the model parameters θ1, θ2 and θ3 where the focal loss [101] is employedto learn θ1 and the loss employed to learn θ2 and θ3 is a linear combination ofthe knowledge distillation and the focal losses. For the outer optimization,the model performs meta-training using all weights obtained from the inneroptimization steps. Furthermore, the outer loop aims to find not only theoptimal parameters for the model itself but also to optimize the meta-learnernetwork weights. In this stage, different combinations of the preset episodesare used to enhance generalization. Also, the loss used in this stage is thefocal loss with fixed hyperparameters in contrast to the inner loop focal loss.3.1.6 Domain AdaptationIn single object tracking, domain adaptation involves training a model on asource dataset encompassing various tracking tasks under varying conditions.This comprehensive training fosters the extraction of domain-agnostic fea-tures, such as motion patterns and object shapes that are crucial for trackingacross different scenarios. During tracking in a new, unseen target domain(e.g., a video with distinct lighting or background), the model can thenadapt its internal parameters or select the optimal tracking strategy basedon the specific characteristics of the target object and its environment. Thisadaptation process leverages the generalizable knowledge acquired from thesource domain while accounting for the target domain’s unique statisticalproperties.[MDNet] Learning Multi-Domain Convolutional Neural Networks forVisual Tracking: This method [116] is the first method to tackle the domainadaptation aspect of visual object tracking. The model encompasses twomain components: A CNN feature extractor and a sequence-dependent fully413.1. TAXONOMYconnected layer. The feature extractor is trained offline to gain domain-independent knowledge, while the last fully connected layer is trained onlineto attain domain knowledge. Using region proposals of a detection algo-rithm, the model classifies them as positive or negative targets, thus defin-ing an easier problem for the bounding box head to solve. Moreover, themodel incorporates hard negative mining [143] to learn non-trivial discrim-ination between the target and the background. Finally, the bounding boxis obtained via a bounding box regression head using features from a convo-lutional layer following [46].[CODA] Real-Time Deep Tracking via Corrective Domain Adaptation:This method [88] deals with the domain adaptation requirement by present-ing a detection-aided tracking methodology. The model consists of threemain components: A tracker, a single-shot detector and an algorithm fortracking output adjustment via the detector. First, the tracker is basedon hierarchical correlation filter [109] where outputs from different hierar-chy convolutional layers are inputs to kernelized correlation filter [56] minitrackers. These mini trackers are prompted to output a coarse tracking re-sult. The main contribution of the tracker is the additional domain adaptivelayers, which are multi-resolution convolutional layers applied to the fea-ture maps before being fed to the KCF trackers. These domain adaptivelayers learn the receptive field required for each task through learning im-portance weights for concatenated outputs of different convolutional filters3× 3, 5× 5, and 7× 7 following the design principal introduced in InceptionCNN [147]. After that, bounding box heads are applied to the output ofthe KCF trackers and an intermediate bounding box is drawn. Finally, themodel adjusts the tracker output by modifying the dimensions of the outputbounding box. The bounding box area is adjusted based on the average ofthe areas of positive bounding boxes obtained from the single-shot detector.A detector bounding box must have IoU and confidence scores that passpredefined thresholds to be considered positive.423.2. EXPERIMENTAL ANALYSIS3.2 Experimental AnalysisIn this section, we report the performance of the discussed trackers on differ-ent single object tracking datasets. We report selected tracking performancemeasures and the datasets used for training each of the discussed models inTab. 3.1. We also report the Expected Average Overlap metric used in therenowned Visual Object Tracking (VOT) challenge in Tab. 3.2.3.2.1 Selected DatasetsA set of the most popular single object tracking datasets was selected toreport the test performance of the surveyed trackers. The test sequencesinclude challenging situations such as Object Color Change (OCC), FastMotion (FM), In-Plane Rotation (IPR), Aspect Ratio Change (ARC), Il-lumination Variation (IV), OCClusion (OCC), Similar OBject (SOB), Low-Resolution Objects (LRO), Viewpoint Change (VC), Full OCclusion (FOC),Out-of-View (OV), Fast Camera Motion (FCM), DEFormation (DEF), LowResolution (LR), Absolute Motion (AM), Motion Blur (MB), Out-of-Planerotation (OPR), target ROTation (ROT), Background Clutter (BC), PartialOCclusion (POC), abrupt Camera Motion (CM), Scene COmplexity (SCO),Scale Variation (SV), MOtion Change (MOC). A detailed definition of eachattribute can be found in [39, 114]. A brief comparison of the selected testingdatasets is presented in Tab. 3.3. We also report the datasets each model wastrained on without specifically being single-object-tracking datasets. COCO[102] is a large scale object detection and segmenetation dataset with 200kimages. ILSVRC [128] is the ImageNet Large Scale Visual RecognitionChallenge that uses the ImageNet dataset, the largest open-access datasetwith 10M images intended for classification. It is also extended to othertasks, e.g. video object detection. ALOV++ [135] contains 315 sequencesfrom the Amsterdam Library of Ordinary Videos focused on robustness eval-uation with one tracking challenge per sequence. It provides annotations forevery 5 frame per sequence. UAV123 [112] is a single-object Unmanned-Aerial-Vehicle-perspective video dataset with 123 sequences. It also includesa subset for long-term tracking. NUS-PRO [85] was proposed as a newtracking challenge using a dataset of 365 YouTube sequences. Kinetics[67] is a large scale human action recognition with approximately 500k se-quences. NFS [42] is the highest frame rate (240 FPS) tracking datasetwith 100 sequences and 380k frames. NYUv2 [132] is an indoor segmenta-tion and depth estimation dataset with 1449 labelled examples with alignedRGB and depth images. YouTube-VOS [174] is a large-scale video object433.2. EXPERIMENTAL ANALYSISsegementation dataset with arround 5k sequences and 190k segementationannotations. YouTube-BB [125] is a general purpose large-scale datasetcomprises 380k sequences with 5.6M bounding box annotations. DAVIS[123] is a densely annotated (per-frame annotations) video object segmen-tation dataset. It contains 50 sequences with mask annotations per objectinstance.Due to the significance of the single object tracking problem, An ongoingannual challenge is held, namely the VOT challenge [74–78]. The VOT chal-lenge utilizes the VOT dataset, which consists of 60 sequences that involvemany challenging scenarios in tracking. The challenge does not require a spe-cific training set and does not provide one. However, the challenge prohibitsthe usage of 1000 sequences from GOT-10k [62] because they were used inrefreshing the sequences of the VOT20 dataset. It is worth mentioning thatthe optimal VOT bounding box is the one that surrounds the object whilecontaining minimal background pixels, which is in contrast to other datasetsthat require a box with its sides aligned with the image sides. The mainjudging metric employed by the challenge is the Expected Average Overlapdiscussed in Section 3.2.2.443.2. EXPERIMENTAL ANALYSISTable 3.1: Summary of the reported performance measures. The datasetsinvolved are: GOT-10k [62], LaSOT [38], LaSOText [39], TrackingNet [114],OTB50 [167], OTB100 [117], COCO [102], ILSVRC [128], ALOV++ [135],UAV123 [112], NUS-PRO [85], NFS [42], NYUv2 [132], YouTube-VOS [174],YouTube-BB [125], Kinetics300 [67], and DAVIS [123]. Please refer to section3.2.1 for a brief overview of the reported datasets.Method Training DatasetsGOT-10k LaSOT TrackingNetAO SR0.5 SR0.75 AUC PNorm P AUC PNorm PAutoregressive[163]ARTrack256GOT-10k, LaSOT, TrackingNet73.5 82.2 70.9 70.4 79.5 76.6 84.2 88.7 83.5ARTrack384 75.5 84.3 74.3 72.6 81.7 79.1 85.1 89.1 84.8ARTrack-L384 78.5 87.4 77.8 73.1 82.2 80.3 85.6 89.6 86.0[20]SeqTrack-L384GOT-10k, LaSOT, TrackingNet, COCO74.8 81.9 72.2 72.5 81.5 79.3 85.5 89.8 85.8SeqTrack-L256 74.5 83.2 72.0 72.1 81.7 79.0 85.0 89.5 84.9SeqTrack-B384 74.5 84.3 71.4 71.5 81.1 77.8 83.9 88.8 83.6SeqTrack-B256 74.7 84.7 71.8 69.9 79.7 76.3 83.3 88.3 82.2Memory-based sequence models[41] GOT-10k, LaSOT, TrackingNet, COCO, ILSVRC 64.2 73.7 57.5 60.6 63.3 69.3 80.3 76.7 85.1[188] Online 34.3 35.2 10.9 - - - - - -[161] GOT-10k, LaSOT, TrackingNet, COCO 67.9 78.3 61.8 67.1 71.8 77.2 81.8 86.8 79.3Masked image modelling[196] GOT-10k, LaSOT, TrackingNet, COCO 67.7 78.4 - 67.8 77.3 - 81.9 86.8 -[164] GOT-10k, LaSOT, TrackingNet, ILSVRC 75.9 86.8 72.0 52.7 63.9 60.2 - - -Generative adversarial networks[181] LaSOT, ILSVRC - - - 44.4 - 45.3 - - -Diffusion models[191] GOT-10k, LaSOT, TrackingNet, NFS 73.5 78.2 64.3 63.6 81.5 - - - -Self-supervised learning[162] ILSVRC - - - 35.1 - 32.6 59.1 - 51.2[93]SSSiamRPN++ILSVRC- - - 49.9 59.6 - - - -SSDIMP - - - 57.4 65.5 - - - -SSTransT - - - 66.3 75.6 - - - -[205]ULS GOT-10k - - - 44.7 46.1 38.2 - - -FLS GOT-10k - - - 50.6 56.2 47.8 - - -FLS GOT-10k, LaSOT, ILSVRC - - - 53.6 59.5 51.7 - - -[206]QCTCIM GOT-10k, LaSOT, ILSVRC, YouTube-VOS51.0 59.6 30.0 46,6 53.7 40.4 63.5 67.0 52.8QCTSIM 52.4 61.0 31.8 47.9 53.1 42.2 64.9 69.0 55.1Unsupervised learning[156] ILSVRC - - - - - - - - -[166]AlexPULILSVRC- - - - - - - - -ResPUL - - - - - - 54.6 63.0 48.5[203] ILSVRC - - - 28.9 29.8 26.1 - - -Meta-learning[91] ILSVRC - - - 36.5 - 35.1 - - -[22] ILSVRC - - - 34.5 - - - - -[7]DIMP-18GOT-10k, LaSOT, TrackingNet, COCO57.9 67.2 44.6 53.2 - - 72.3 78.5 66.6DIMP-50 61.1 71.7 49.2 56.9 - - 74.0 80.1 68.7[155]FCOS-MAMLGOT-10k, LaSOT, TrackingNet, COCO- - - 52.3 - - 75.7 82.2 -Retina-MAML - - - 48.0 - - 69.8 78.6 -[28] LaSOT - - - 57.2 - 57.2 - - -[35] COCO, ILSVRC, YouTube-BB - - - 49.9 57.4 49.4 - - -[96] GOT-10k, LaSOT 66.7 72.5 59.9 65.9 73.8 68.8 - - -[121] GOT-10k, LaSOT, YouTube-VOS, DAVIS - - - 69.7 76.2 73.3 81.6 86.0 79.4Continual learning[190] VOT13, VOT14, VOT15 - - - 42.4 - 38.8 - - -[23]ConTACT-18GOT-10k, LaSOT, TrackingNet, ILSVRC57.7 66.9 47.1 56.8 65.3 59.7 75.0 79.9 71.6ConTACT-50 59.4 68.7 48.7 65.8 65.3 62.7 75.8 80.7 72.7453.2. EXPERIMENTAL ANALYSISTable 3.1: Summary of the reported performance measures. The datasetsinvolved are: GOT-10k [62], LaSOT [38], LaSOText [39], TrackingNet [114],OTB50 [167], OTB100 [117], COCO [102], ILSVRC [128], ALOV++ [135],UAV123 [112], NUS-PRO [85], NFS [42], NYUv2 [132], YouTube-VOS [174],YouTube-BB [125], Kinetics300 [67], and DAVIS [123]. Please refer to section3.2.1 for a brief overview of the reported datasets.Method Training DatasetsLaSOText OTB50 OTB100AUC PNorm P AUC AUCAutoregressive[163]ARTrack256GOT-10k, LaSOT, TrackingNet46.4 56.5 52.3 - -ARTrack384 51.9 62.0 58.5 - -ARTrack-L384 52.8 62.9 59.7 - -[20]SeqTrack-L384GOT-10k, LaSOT, TrackingNet, COCO50.7 61.6 57.5 - -SeqTrack-L256 50.5 61.5 57.2 - -SeqTrack-B384 50.5 61.6 57.5 - -SeqTrack-B256 49.5 60.8 56.3 - -Memory-based sequence models[41] GOT-10k, LaSOT, TrackingNet, COCO, ILSVRC - - - - 71.9[188] Online - - - 61.3 65.2[161] GOT-10k, LaSOT, TrackingNet, COCO 46.3 - 52.7 - 69.2Masked image modelling[164] GOT-10k, LaSOT, TrackingNet, ILSVRC 71.8 81.8 78.1 - 69.6Generative adversarial networks[49] ALOV++, VOT16 - - - 57.6 61.8[139] VOT13, VOT14, VOT15 - - - 71.0 68.2[195] ILSVRC, UAV123, NUS-PRO - - - - 69.1[185] Online - - - - 71.5[51] VOT13, VOT14, VOT15 - - - 71.3 68.7[184] VOT13, VOT14, VOT15 - - - 68.5 66.1[181] LaSOT, ILSVRC - - - - 68.3Variational autoencoders[159] VOT13, VOT14, VOT16, ILSVRC - - - 62.4 57.4[81] - - - - 68.6 -Domain adaptation[116] VOT13, VOT14, VOT15 - - - 70.8 67.8[88] ILSVRC - - - 61.4 56.7463.2. EXPERIMENTAL ANALYSISTable 3.2: Summary of the reported performance measures on the VOTchallenge dataset from 2016 to 2020 [74–78].Method Training DatasetsVOT16 VOT17 VOT18 VOT19 VOT20EAO EAO EAO EAO EAOMemory-based sequence models[178] ILSVRC - 22.2 - - -[43] ILSVRC - 27.4 - - -[197] GOT-10k 37.5 - 39.8 - -[41] GOT-10k, LaSOT, TrackingNet, COCO, ILSVRC - - 44.7 - -[177]MemTrackClassificationILSVRC27.1 24.2 - - -MemDTC 26.8 25.0 - - -MemDTCClassification 29.8 26.5 - - -[188] Online 36.8 - 25.8 - -[161] GOT-10k, LaSOT, TrackingNet, COCO - - - - 50.1Generative adversarial networks[139] VOT13, VOT14, VOT15 32.3 - - - -[51] VOT13, VOT14, VOT15 34.7 - - - -[181] LaSOT, ILSVRC 36.0 - 34.7 36.1 -Variational autoencoders[173] GOT-10k 33.9 - - - -[63] NYUv2, YouTube-VOS 44.3 - 40.1 30.0 -Self-supervised learning[186] ILSVRC 39.8 - 31.7 - -[134] ILSVRC 23.2 - 19.0 - -[205]ULS GOT-10k - - 16.6 - -FLS GOT-10k - - 19.1 - -FLS GOT-10k, LaSOT, ILSVRC - - 20.9 - -Unsupervised learning[156] ILSVRC 30.1 - - - -[166]AlexPULILSVRC21.9 - - - -ResPUL 26.3 - - 19.8 -[203] ILSVRC - - 16.1 - -Meta-learning[120]Meta-CRESTVOT13, VOT14, VOT15, ILSVRC31.7 - - - -Meta-SDNet 31.4 - - - -[91] ILSVRC - 24.7 - - -[22] ILSVRC 42.1 - - - -[155]FCOS-MAMLGOT-10k, LaSOT, TrackingNet, COCO- - 39.2 - -Retina-MAML - - 45.2 - -[28] LaSOT - - - - -[35] COCO, ILSVRC, YouTube-BB - - - 31.3 -[165]MGA-SiamFCILSVRC26.3 - 23.7 - -MGA-SiamRPN++ 47.5 - 42.5 - -MGA-DCFNet 24.4 - 18.1 - -[96] GOT-10k, LaSOT - - 48.9 - -[121] GOT-10k, LaSOT, YouTube-VOS, DAVIS - - - - 50.647Table 3.3: A brief overview of the selected datasets used in reporting the different performance metrics of thediscussed trackers. Other mentioned training datasets in Tab. 3.1 and Tab. 3.1 are briefly discussed in section3.2.1.Dataset # Training Sequences # Testing Sequences # Classes Training and Testing Overlap Absence Label Additional Labels Evaluation Metrics ChallengesGOT-10k 9334 180 563 No Yes YesAverage Overlap,Success RateSV, ARC, FM, LR, IV, OCCLaSOTProtocol I: 1400Protocol II: 1120280 85Protocol I: YesProtocol II: NoYes NoArea Under the Success Plot,Precision,Normalized PrecisionCM, VC, ROT, SV, DEF, BC,POC, FOC, MB, IV, ARC, OV,LR, FMLaSOText Protocol III: 1400 150 85 Protocol III: No Yes NoArea Under the Success Plot,Precision,Normalized PrecisionCM, VC, ROT, SV, DEF, BC,POC, FOC, MB, IV, ARC, OV,LR, FMTrackingNet 30132 511 27 No No NoArea Under the Success Plot,Precision,Normalized PrecisionSV, ARC, FM, LR, OV, IV,CM, MB, BC, SOB, DEF, IPR,OPR, POC, FOCOTB-50 - 50 10 No No NoArea Under the Success Plot,PrecisionIV, SV, OCC, DEF, MB, FM,IPR, OPR, OV, BC, LROTB-100 - 100 16 No No NoArea Under the Success Plot,PrecisionIV, SV, OCC, DEF, MB, FM,IPR, OPR, OV, BC, LR483.2. EXPERIMENTAL ANALYSIS3.2.2 Performance Metrics3.2.2.1 Average Overlap (AO)The AO is the main adopted metric by GOT-10k [62]. It measures the meanaverage overlap between the ground truth bounding boxes and the predictedbounding boxes. Instead of directly averaging the overlaps in all sequences,a per-class average is calculated, and each class average overlap is weightedaccording to the number of class samples. Finally, all averages are reducedto a per-dataset overlap. Calculating the AO this way accounts for the classimbalance in the dataset.3.2.2.2 Success Rates (SR0.5 and SR0.75)The secodary measure adopted by GOT-10k [62] to indicate accuracy-robustnesstrade-off. The success rate is the percentage of predicted bounding boxesthat exceed an overlap threshold. The threshold can be either 0.5 or 0.75.As for the AO, the success rates are calculated per class and then reducedper dataset to accommodate the imbalance. Uniquely, the success rate cal-culation is performed three times and averaged to get a robust measure.3.2.2.3 Area Under the Success Curve (AUC)This is the main performance measure adopted by LaSOT [38, 39], Track-ingNet [114] and OTB [117, 167]. First, the success curve is obtained bydefining a 0 ∼ 1 range of thresholds as the x-axis and another 0 ∼ 1 rangeof success ratios. A success ratio at a certain threshold is the number offrames in the predicted bounding boxes that overlap with the ground truthbounding boxes greater than the threshold. The area under the curve isreported after forming the success curve for different thresholds.3.2.2.4 Precision (P)Tracker precision is the distance between the centres of the predicted andground truth bounding boxes in pixels. The reported precision, which isreported for LaSOT [38, 39] and TrackingNet [114], is the ratio of the framesthat has a precision greater than a certain threshold (usually = 20 pixels).3.2.2.5 Normalized Precision (PNorm)This measure was introduced in TrackingNet [114] and adopted in LaSOT[38, 39]. It mainly aims to cancel the effect of the object and image dimen-493.3. DISCUSSION AND FUTURE DIRECTIONSsions in calculating the precision. This is done by dividing the precision perframe by the ground truth bounding box size.3.2.2.6 Expected Average Overlap (EAO)Adopted by VOT challenge datasets since 2015 [74–78] and calculated byaveraging the average overlaps on a very large set of sequences. This accom-modates the high variance of the AO.3.2.3 Cumulative RemarksAlthough performance comparison between different trackers is a key tounderstanding them, we put forth that reporting the performance for thedifferent trackers is for completeness. We do not mean to compare differenttrackers that emerged in distant years, having different backbones and lever-aging different computation capabilities. Rather, we report the performanceto observe the evolution in single object tracking methodologies. Please re-fer to Section 3.1 for more information about how different algorithms mightaffect the performance.3.3 Discussion and Future DirectionsThis survey provides a comprehensive outlook on emerging theoretical tech-niques in deep learning and their application in single-object tracking. Inthis section, we summarize the main takeaways and suggest future pathwaysfor single-object tracking based on the trends we observe throughout thesurvey.Although this survey focuses on the new applications of non-traditionaltheoretical techniques in deep learning, it is worth mentioning that a big por-tion of the research in object tracking is shifting toward the vision trans-former paradigm. To this end, we investigate the increase in the employ-ment of vision transformers in single object tracking from 2020 to 2024 inFig. 3.7. In this investigation, six conferences, CVPR, ICCV, ECCV, AAAI,NeurIPS, and WACV, and four journals, TPAMI, TIP, IJCV, and TCSVT,among the top-tier venues in computer vision, are adopted. We count theresearch publications on single object tracking in these 10 venues. We re-fer to the publications that utilize a transformer, either purely or jointlywith a CNN, as ”Transformers.” For pure CNN-based approaches we re-fer to them as ”CNNs.” We refer to DCF-based or any method other thantransformer-based or CNN-based methods as ”Others.” From Fig 3.7, it can503.3. DISCUSSION AND FUTURE DIRECTIONS2020 2021 2022 2023 2024010203040TransformersCNNsOthersYearPublicationsCountFigure 3.7: A breakdown of the number of publications in single objecttracking that utilize transformers, CNNs, or any other approach such asDCF.be deduced that although the CNN-based approaches dominated single ob-ject tracking research before 2023, these approaches are losing the researchinterest to the transformer-based approaches. A similar trend can be seenwith methods other than CNN-based. Overall, the current research in singleobject tracking is centred around transformer-based approaches. Many fac-tors can justify this trend. One factor is that the transformer architectureis inherently of a bigger receptive field than that of CNNs, thus adaptingto changes in the scale of the target object. Another factor is the intro-duction of large-scale single object tracking datasets such as GOT-10k [62]and LaSOT [38], which suit the transformer architecture. A third factor isthe explicit attention paid to the template and search tokens. Moreover,by design, the transformers do not have the strong inductive bias that theCNNs have. These factors enable a more generic representation of the rela-tion between the template and the search images. The vision transformer’spowerful generic representation capability is particularly useful for the singleobject tracking task that requires capturing the critical relations from onlyone annotated example at tracking time. Moreover, the attention mecha-nism effectively handles long-range dependencies, crucial for single-objecttracking, especially long-term tracking.Another trend we observe is the incorporation of natural language.513.3. DISCUSSION AND FUTURE DIRECTIONSWith the current state of the Large Language Models, it is natural to haverelated applications in single object tracking. Mainly, the usage of LLMs-based techniques falls into two categories; one is natural language tracking[95, 97, 201] in which a tracker is prompted to track an object given a spec-ification provided in natural language. The other category, which is fairlyrecent, is the usage of the natural language in enhancing the tracking modelperformance [92]. The latter category is interesting because it enforces atracker to associate a generic language description with a specific visual ap-pearance, leveraging information that is usually unemployed. These methodspave the way for generic cross-modal models that integrate multiple tasks.As per this work, we observe the importance of explicit learning oftemporal correspondence. The current state-of-the-art techniques se-quence models effectively learn the temporal cues by autoregression. Follow-ing the success of autoregressive Large Language Models (LLMs) [10, 180],it is natural to extend the concept of autoregression in single object trackingto calculate appearance models for a dynamically changing template object.This way turns out to be effective when applied to a one-stream trackingframework such as [182].Furthermore, we highlight the significance of the backbone pretrainingin a supervised or self-supervised manner. More recent techniques employpretraining in a supervised way such as [164, 196] where a task of jointreconstruction of template and search images helps the model understandthe relationship between the template and its surroundings as well as thetemporal variations in a target object implicitly. In another context of self-supervised unlabelled pretraining, existing works emphasize the importanceof the cycle consistency constraint as a non-trivial pretext task that servesvisual tracking.Another observation is the powerful concept of generative models. Inparticular, the GAN-based methods enhance the tracking model discrimi-nation abilities. However, generative models such as diffusion models havestill not been discovered. Despite landing in object detection [17] and multi-object tracking [108], its application in single-object tracking is still far be-hind. As for the time being, diffusion is only used to generate more samplesbut not for an end-to-end single object tracking framework.As future pathways, we see that more research will employ learning au-toregressive temporal cues alongside learning robust visual representations.This is due to the increasing amount of research that learns either of themseparately. A joint approach can be implemented by penalizing a recon-struction error alongside maximizing the likelihood of the next autoregres-sive bounding box token. We also anticipate that a stronger pretext task for523.3. DISCUSSION AND FUTURE DIRECTIONSself-supervised tracking will be adopted. This is especially possible with theintroduction of many novel pretext tasks for learning discriminative visualfeatures and robust temporal features. An example of a novel pretext taskis Mixed Autoencoder [16], where the autoencoder is prompted to minimizesimilar object’s embeddings and maximize the different object embeddingsin the same image. Another novel pretext task is introduced in No MoreShortcuts (NMS) [30] where a model is prompted to predict the position ofa frame in a given sequence via two concurrent tasks, one is the skip rateand the other is whether the frame is out of order. This kind of temporal de-pendency capturing in a pretraining phase enables a holistic understandingof the changes that may occur to an object in a tracking setting.Finally, a promising direction is harnessing diffusion models to ob-ject tracking by following the method proposed by [17] with the diffusiondenoising process done on the bounding boxes.53Chapter 4Investigating Bounding BoxRegression Method forOne-Stream Single ObjectTracking Transformers4.1 Methodology4.1.1 Adopted Tracking FrameworkWe hypothesize that modern ViT-based single-object trackers, such as OS-Track [182] and ODTrack [199] lack good exploitation of the joint search andtemplate embeddings to extract the location information.Given a ViT-based backbone, the output after the joint feature extrac-tion and relation modelling phase is an embedding tensor of dimensions(B,H ×W,D) where B is the batch size, H ×W is the dimension of thetransformer patch, and D is the embedding dimension. For the ViT-basedtrackers represented by the three selected trackers, this output tensor is sub-jected to a CNN-based bounding box regression network that extracts threescore maps representing the center location, the size, and the offset of thecenter of the bounding box as shown in Fig. 4.1. Given that the input tensormight contain global correlation as well as local correlation, we hypothesizeit is essential that the bounding box network has multiple receptive fieldsthat enable the network to capture relationships on both the global and locallevels.4.1.2 Inception NetworkTo this end, we propose the usage of an Inception block [147] in two alter-natives as depicted in Fig. 4.2 and Fig. 4.3.The simpler inception block is adopted from [146]. As depicted in Fig. 4.2,it is composed of four parallel convolutional paths. The four convolutional544.1. METHODOLOGYTransformerBackboneBoundingBoxHeadTemplateSearchCenterMapSizeMapOffsetMapRedesignedInceptionBlockRedesignedInceptionBlockRedesignedInceptionBlockDownsamplingCNNDownsamplingCNNDownsamplingCNNFigure 4.1: The overall architecture of the proposed ViT-based single objecttracker bounding box head.paths share the functionality of mapping the input feature map to a new fea-ture space while each of them has a different receptive field. Namely, while abigger filter size of 7×7 that is realized by the two 3×3 convolutions shownin the upper-most branch foster a more global understanding of attentionalrelationships in the input tensor, the smaller ones (1 × 1, 3 × 3) focus ona local context. After feeding the embedding tensor into the convolutionallayers, the obtained outputs are concatenated and fed into a 1× 1 convolu-tional filter to reduce the number of channels to the original input numberof channels, hence, speeding up the computations.The more sophisticated inception block is based on the same idea butwith a different implementation. Given the intrinsic adaptive receptive fieldthat Deformable Convolutional Networks have [207], we design a deformable-based inception block of convolutions that have filter size of 3 × 3 whereeach filter performs deformable convolution operation and the final outputis concatenated and fed to a regular 1× 1 convolution as shown in Fig. 4.3.The deformable convolutions enable a larger receptive field for the samefilter size by learning the sampling positions of the input. In addition to thedeformable convolution, we add a branch of regular convolution to considerlocal contiguous context, therefore simplifying the learning objective anddecreasing convergence time.554.2. EXPERIMENTSConv1× 1AVG PoolConv3× 3Conv3× 3 Conv3× 3Conv1× 1Conv1× 1Conv1× 1Conv1× 1CFigure 4.2: The simple Inception subnetwork architecture adapted from[146]. The blue-filled circle indicates concatenation operation. Filter sizesare shown in the figure. Please refer to [146] for more details about theparameters of each convolution.DeformConv3x3RegularConv3x3Conv1x1CFigure 4.3: The deformable inception block.4.2 Experiments4.2.1 Implementation DetailsWe adopt the ViT-base [36] as the visual encoder with MAE pretrainingweights [52] to train OSTrack [182] and ODTrack [199] using our proposedbounding box regression networks. The template image shape is 192×192×3while the search image shape is 384×384×3. All models are trained for 100epochs solely on the GOT-10k dataset [62]. Training is conducted on a serverwith two 50 GB GPUs. The batch size is set to 8. AdamW [106] is usedto optimize the training objective. We sample 40000 examples per epoch totrain networks based on ODTrack following the configurations found in theircode repository, while the number of samples per epoch for OSTrack-basednetworks is 60000. We keep the learning rate and the weight decay thateach model was originally trained with. We report the performance of the564.2. EXPERIMENTSTable 4.1: Performance comparison against the state-of-the-art methods.We emphasize that our models are only trained on the training split of theGOT-10k dataset. Significant results are in boldMethodGOT-10k [62] UAV123 [113] OTB2015 [168]AO SR0.5 SR0.75 AUC AUCSiamFC [6] 34.8 35.3 9.8 46.5 58.2ATOM [29] 55.6 63.4 40.2 65.0 -DiMP [7] 61.1 71.7 49.2 65.4 68.4SiamRCNN [153] 64.9 72.8 59.7 64.9 70.1Stark [176] 68.8 78.1 64.1 69.1 68.5SBT-B [171] 69.9 80.4 63.6 - 70.9Mixformer [26] 70.7 80.0 67.8 70.4 -OSTrack [182] 73.7 83.2 70.8 70.7 -AiATrack [44] 69.6 80.0 63.2 70.6 69.6SeqTrack [20] 74.5 84.3 71.4 68.6 -GRM [45] 73.4 82.9 70.4 70.2 -VideoTrack [170] 72.9 81.9 69.8 69.7 -ARTrack [163] 75.5 84.3 74.3 70.5 -ODTrack-B [199] 77.0 87.9 75.1 - 72.3ODTrack-B (Our machine) 75.6 86.1 72.7 65.8 68.5ODTrack-B-Inception (Ours) 77.3 87.5 75.5 65.8 69.0two proposed variants, namely the Inception module and the DeformableInception module. The basic convolutional block used in both of them is astack of conv, ReLU, and BatchNorm. The filter details of the convolutionsfor each variant are shown in Fig. 4.2 and Fig. 4.3.4.2.2 State of the Art ComparisonWe compare our method to 14 methods on three leading datasets, GOT-10k [62], UAV123 [113] and OTB2015 [168]. The results shown in Table 4.1indicate that our method variants outperform the state-of-the-art techniqueson the three datasets when compared with retrained models on the samemachines.4.2.2.1 GOT-10kGeneric Object Tracking dataset is a large-scale single object tracking dataset.It contains 9334 training sequences, 180 validation sequences and 180 testsequences. The testing protocol of the dataset requires models to be trainedonly on the GOT-10k training sequences. The training and test sequencesare not of overlapping classes. As reported in Table 4.1, our best performant574.2. EXPERIMENTSTable 4.2: Investigation of the performance of the two proposed boundingbox networks on two modern one-stream trackers. Significant results are inbold. All trackers are retrained from MAE [52] pretraining weights on ourmachine using the same configurations provided in their repositories for faircomparison.MethodGOT-10k [62] UAV123 [113] OTB2015 [168]AO SR0.5 SR0.75 AUC AUCOSTrack-B [182] 70.1 79.0 66.3 65.79 66.68OSTrack-B-Inception (Ours) 71.6 80.7 66.9 66.01 68.18OSTrack-B-deformable (Ours) 70.8 79.8 67.1 67.78 66.98ODTrack-B [199] 75.6 86.1 72.7 65.79 68.46ODTrack-B-Inception (Ours) 77.3 87.5 75.5 65.79 69.03ODTrack-B-deformable (Ours) 76.8 87.2 73.3 64.30 68.89Table 4.3: Study of the performance of ODTrack with deformable convolu-tion Inception module. ”deformable only” refers to the deformable branchwithout the regular convolution while ”Conv only” refers to the originalODTrack [199]. ”Both” refers to using the bypass branch. Significant re-sults are in bold.MethodGOT-10k [62]AO(%) SR0.5(%) SR0.75(%)Deformable only 75.4 86.6 73.3Conv only 75.6 86.1 72.7Both 76.8 87.2 73.3variant outperforms the state-of-the-art ODTrack-base [199] trained on ourmachine by 1.7% in Average Overlap (AO). Moreover, for the Success Rate at0.5 IoU threshold (SR0.5) and the (SR0.75), our model outperforms ODTrackby 1.4% and 2.8% respectively.4.2.2.2 UAV123UAV123 is a aerial viewpoint video dataset that contains 123 challengingsequences. It contains training and testing sequences. However, we only usethe testing set for evaluation purposes without training on the training set.According to the shown results in Table 4.1, our method outperforms thealigned state-of-the-art techniques while it is only trained on the GOT-10k584.2. EXPERIMENTStraining split.4.2.2.3 OTB2015Object Tracking Benchmark [168] is a pioneering testing set for single objecttracking that consists of 100 challenging frames collected from YouTube.The dataset labels are publicly available. The performance pattern on thisdataset is similar to the one in the GOT-10k as they are both datasets ofthe same natural image domain.4.2.3 Ablation StudyWe examine the effect of the proposed simple Inception and deformable con-volution Inception on the two adopted modern trackers stated in section 4.1.As reported in Table 4.2, our Inception and deformable inception methodsoutperform the locally trained models on the GOT-10k test set consider-ing the Average overlap by 1.5% and 0.6% respectively. The pattern isconsistent despite the different tracking ideas implemented in the differentmodules, indicating the effectiveness of the inception and deformable incep-tion convolutional networks in learning a suitable receptive field to processthe joint search and template embeddings. Notably, the pure Inception ismore powerful enhancing the performance on all three datasets. This indi-cates the higher learning capacity of the inception module compared to thedeformable inception. We also investigate the effect of the skip branch thathas the 3 × 3 convolutional filter of the deformable convolution Inceptionmodule in Table 4.3. It can be shown that the adopted variant is the bestperformant deformable module, indicating the importance of the 3× 3 con-volution skip connection in facilitating the learning objective and increasingmodel capacity.4.2.4 VisualizationWe qualitatively visualize the output of our two bounding box regressionnetworks in Fig. 4.4. As depicted, the regular Inception has a better bound-ing box quality. This indicates the high model capacity of the inceptionmodule that enables the best exploitation of the visual features given by theViT in ODTrack [199].59Figure 4.4: Success cases of the proposed methods on the OTB2015 Basketball sequence. It can be noted that thepure Inception network is most accurate especially when comparing the first column of frames. Although boundingboxes converge at frame #50, the initially predicted bounding boxes at frame #2 indicate the fast adaptation ofthe Inception bounding box network.604.3. CONCLUSION4.3 ConclusionIn this work, we show the significance of the bounding box regression giventhe visual interaction features of a one-stream tracking transformer. Weshow that a bounding box-regression network lacking a suitable receptivefield can hinder the performance of a transformer tracking model due to thelack of good exploitation of the visual features provided by the backbone.Experiments show that our methods of Inception and deformable inceptionbounding box networks perform favourably against state-of-the-art ODTrackby 1.7% of AO on the GOT-10k [62] dataset. As future pathways, this workcan be extended to ablate the different emerging CNN techniques and findthe CNN bounding box network that strikes the balance between a goodinductive bias and a global receptive field.61Chapter 5Learning Deformable MaskedVisual Representations forSingle Object Tracking5.1 MethodologyWe propose deformable masking modules for robust single-object tracking.Overall, our method’s novelty lies in the vision transformer encoder block.The overall architecture of the end-to-end tracker is shown in Fig 5.1. Wepropose two variants of the deformable masking model depicted in Fig. 5.2.We name the tracking model DMTrack. The rest of the section is organizedas follows: The theory behind our approach is delineated in section 5.1.1.Following that, the different components of our module are discussed insection 5.1.2. Finally, the bounding box prediction head and the loss functionare outlined in section 5.1.3.5.1.1 Problem StatementTable 5.1: The average overlap of two SOTA trackers compared to ours onGOT-10k validation and test sequences at the end of training on the trainingsplit of GOT-10k [62].OSTrack-B [182] ODTrack-B [199] DMTrack-B (Ours)AOV alidation(%) 89.3 87.0 87.8AOTesting(%) 71.1 77.0 79.1∆ AOTesting−V alidation(%) −18.2 −10.0 −8.7To holistically explain DMTrack, it is important to provide a brief overviewof the dominant one-stream vision transformer paradigm [12, 26, 182]. For-mally, given a template image z ∈ R3×Hz×Wz that only contains the ap-pearance of an object to be tracked in the first frame of an arbitrary se-quence of search images S = {xi|1 ≤ i ≤ N}, where x ∈ R3×Hx×Wx and625.1. METHODOLOGYTemplateSearch192× 192× 3384× 384× 3Encoder withDeformableMasking×LReshapeandBoundingBoxNetworkPatchifyandLinearProjectionE1E2EL−1FusionELMasking DeformableConvMasking DeformableConvMasking DeformableConvFigure 5.1: End-to-end single object tracker based on the deformable mask-ing module.635.1. METHODOLOGYN is the sequence length, it is required to determine the coordinates ofbounding boxes B = {bi|1 ≤ i ≤ N} that determine the location and di-mensions of the tracked object in each frame of the search sequence S. Adeep learning model typically approaches the problem of single object track-ing as an approximation to a function ω that maps a pair of (z, x) to b.Predominantly, ω is broken into three stages: feature extraction ω1, rela-tion modelling ω2 and bounding box prediction ω3. In conventional Dis-criminative Correlation Filters [65] and Siamese Convolutional Networks[5, 65, 148], the three phases are represented by three separate modules,each of which output is dependent on the input from its preceding phase.In the recently widely adopted one-stream vision transformer paradigm, ω1and ω2 jointly leverage the transformer’s powerful architecture. For a one-stream vision transformer, the template and search pair z, xi are embeddedinto Nz = (Hz ×Wz)/p2 and Nx = (Hx ×Wx)/p2 patches. The patcheszp ∈ RNz×(3∗p2) and xp ∈ RNs×(3∗p2), are of resolution p× p. These patchesare fed into a stack of L vision transformer layers. At a finer level, eachtransformer layer performs the self-attention operation:A = Softmax(QKT√dk) · V= Softmax([Qz;Qx][Kz;Kx]T√dk) · [Vz;Vx](5.1)where Q, K, and V are the query, key, and value, respectively. dk is thedimensionality of the key. The output of the joint phase of relation modellingand feature extraction is an attentional feature map A of shape (HA,WA).This paradigm has effectively shown that a joint phase of feature encod-ing and relation modelling is crucial for visual tracking. However, the jointinteraction has to be carefully handled as it might lead to overfitting somerepresentations. This is mainly because the model can change the weightsof each phase without carefully designed constraints. As a result, some easyrepresentations or relations could be learned. This problem can be demon-strated empirically on two trackers, OSTrack [182], and ODTrack [199], asshown in Tab. 5.1. The reported values are adopted from the original papersand raw training logs in public repositories.To address this issue, we propose the deformable masking module shownin Fig 5.2 in the next section.645.1. METHODOLOGY5.1.2 Learning Robust Tracking FeaturesIn this section, we explain our model in detail. The main building blockin our model is the deformable masking module. It consists of a maskingsubmodule followed by a deformable convolution submodule.5.1.2.1 Deformable Convolution ModuleWe adopt the deformable convolution architecture proposed by [207]. Thedeformable convolution submodule comprises three regular convolutions, asshown in Fig. 5.3. The deformable convolution process can be considereda regular convolution with two modifications. First, the input feature mapis fed into a regular convolution to determine the sampling locations wherethe weights of the deformable convolution are applied to the input of thedeformable convolution. Second, another convolution modulates the inputfeature map of the deformable convolution. The modulation parameters arecalculated by a regular convolution with a softmax activation function toensure a range from 0 to 1. The modulation parameters are then element-wise multiplied by the original feature map to emphasize more importantpixels and discard less important ones. Formally, the output y(p) at a specificindex p of a modulated deformable convolution of input x is described by:y(p) =I∑i=1wi · x(p+ pi +∆pi) ·mi (5.2)where I is the number of indices of the convolutional filter for iteratingall possible filter locations, e.g. for a 3 × 3 convolutional filter, I = 9. wi,∆pi and mi are the filter weight, the predicted offset and the modulationfactor, respectively, all at the element i of the filter.The rationale behind deformable convolutions is simple. Mainly, themodel requires a subnetwork that masks some representations but then re-generates them so that it does not confuse the transformer layers. Thedeformable convolution can use information from a variable receptive fieldwhich respects the transformer’s assumption of less inductive bias.5.1.2.2 Masking ModuleTo mitigate the effect of learning simple representations or relations, theoutput of each transformer layer is masked. The selected area to mask isat random following a uniform distribution. The masking process involvesthree main design choices: masking areas (where and how), masking ratio655.1. METHODOLOGY(how many) and masking order (when). First, in terms of where the ar-eas of masking are, this refers to which portions of the input attentionalfeature map should be masked. We define two main dimensions to maskalong. As shown in Fig. 5.2, the output of the transformer has two possibledimensions to mask along, namely the embedding dimension and the com-bined search and template tokens dimension. This means that the possiblemasked portions are in the form of rectangular prisms, as shown in Fig 5.4.In terms of how the masking is performed, it can either be contiguous areasor sparsely distributed in the feature map. Second, the masking ratio de-termines the volume of the masked portion and is to be decided empirically.Third, the masking order means whether the masking should be performedon the output of the layer l of the transformer or the output of the deformableconvolution. Although masking the transformer representations first beforeapplying the deformable convolution seems intuitively better, we experimen-tally find that masking the deformable convolution output achieves similarperformance as reported in Table 5.6.5.1.2.3 FusionA significant part of the proposed method is the information flow betweenthe deformable masking module and the layers of the vision transformer.Concretely, the information flow starts at the vision transformer layer l0that feeds its output into the first deformable masking module dmm0. Afterthat, we investigate several possible choices of paths. At the macro level, wedesign two variants of DMTrack shown in Fig. 5.2. The macro choice hereis whether to fuse the outputs of each deformable masking module directlywith the output of the last layer of the vision transformer (variant a) orfuse the outputs of the corresponding vision transformer and deformablemasking module at each layer and feed the fused map to the next layer ofthe vision transformer (variant b). At the micro level, we analyze threedifferent choices shown in Fig 5.5, namely addition, learnable single-stageweighted concatenation, and learnable multi-stage weighted concatenation.Other ways of information flow might be more sophisticated submodules suchas cross-attention. However, the focal point of this work is the deformablemasking, not the fusion between inputs. Moreover, any sophisticated fusionmodule increases the computational complexity and the risk of overfittingthe representations, which we try to avoid. Our model employs learnablemulti-stage concatenation, which can be described by:F oj =Wj ⊛ (F ij ∥ F oj−1) (5.3)665.1. METHODOLOGYwhere F oj is the output feature map at stage j, Wj is the weight of the1 × 1 convolution, ⊛ is the convolution operation, F ij is the input featuremap at stage j, and F oj−1 is the output feature map from the previous stagej − 1.5.1.3 Head and LossTo achieve the end goal of single object tracking, the output of the visiontransformer is fed into a bounding box prediction network. The boundingbox network is adopted from OSTrack [182]. It consists of three parallel con-volutional branches that output three score maps that contribute to bound-ing box coordinates calculation. These maps are the size ∈ R2×Hxp ×Wxp ,the offset ∈ R2×Hxp ×Wxp and the center ∈ R1×Hxp ×Wxp . The bounding box iscalculated as follows:xmax = argmaxxCenterymax = argmaxyCenterx = xmax +Offset(0, xmax, ymax)y = ymax +Offset(1, xmax, ymax)w = Size(0, xmax, ymax)h = Size(1, xmax, ymax)(5.4)where (xmax, ymax) are the coordinates of the maximum response in thecenter score map.The optimization objective consists of three components. First, the Fo-calLoss [101] is utilized to guide the center score map to be close to a Gaus-sian score map at the center of the object. Second, a bounding box is calcu-lated and compared to the ground truth bounding box using the other twocomponents of the objective function, namely the L1 and the GIOU (Gen-eralized IOU) [127] losses. Finally, the different components are combinedas follows:Ltotal = λ1Lclassification + λ2L1 + λ3LGIOU (5.5)where λ1, λ2, λ3 are weighting factors and set to 1, 5 and 2 respectively.5.1.4 Tracking modeDuring tracking, while the deformable convolutions are kept, the maskingmodule works as an identity module by passing the input without masking.675.2. EXPERIMENTAL ANALYSISE1E2ELBatchesTokensEmbeddingsE1ELFusionE2FusionDeformbleMaskingModuleDeformbleMaskingModule(A) (B)DeformbleMaskingModuleDeformbleMaskingModuleFusionFusionBatchesTokensEmbeddingsFigure 5.2: Two variants of the deformable masking module. (A) is denotedas the bypass variant where outputs of the deformable masking module ateach layer are fused. (B) is the sequential variant where the outputs fromthe last layer’s deformable masking module are fused with the output of theencoder and then fed into the next layer transformer encoder. The numberof the deformable masking modules is L− 1 for any of the variants.The main hypothesis behind this decision is that the learned representationsare robust enough that masking during tracking would not hurt or increasethe performance by much.5.2 Experimental Analysis5.2.1 Implementation Details5.2.1.1 Training DetailsWe adopt ODTrack [199] as a base tracker. The ViT-base [36] is utilized asthe visual encoder. The ViT weights are initialized using the Masked AutoEncoder (MAE) [52] weights. Due to computational burdens, we use thetraining splits of GOT-10k [62], LaSOT [38] and COCO [102] only to trainthe full model. This is differnet from the literature where the training split ofTrackingNet [114] is also used to train the model. The full model is trainedfor 360 epochs. We leverage AdamW [105] as the optimizer. We set thelearning rate to 1×10−5 for the ViT encoder and 1×10−4 for the rest of thenetwork. We reduce the learning rate by 10 after 240 epochs. We decay theweights of the whole network by a factor of 10−4. We sample 60000 trainingexamples per epoch. we sample Frames from the same sequence in an intervalof 400 frames. The batch size is 8 on two Nvidia A6000 GPUs each of 48GB of VRAM. We follow the video-level tracking method in ODTrack [199].685.2. EXPERIMENTAL ANALYSISOffsetsFeatureMapModulatedFeatureMapDeformedFeatureMapModulationOffsetConvDeformableConvFigure 5.3: Deformable convolution network. In this work, the offset con-volution is a stack of 3 × 3 convolution and layer normalization [2]. Themodulation convolution is a stack of 3 × 3 convolution, softmax activationand layer normalization. We adopt layer normalization in this submoduleaccording to convergence time and test metrics results.(A) (B) (C)Figure 5.4: Possible masking areas of a transformer feature map at a certainlayer. (A) masking is carried on the embedding dimension where severalrepresentations are removed from all combined search and template tokens.(B) masking is carried on the token dimension where several tokens areremoved. (C) masking is performed on both dimensions.695.2. EXPERIMENTAL ANALYSIS(A) (B)(C)+CC Conv1× 1Conv1× 1Conv1× 1CFigure 5.5: Three ways of information flow in the fusion block. (A) is a sim-ple addition of all feature maps input to the fusion module. (B) is learnablesingle-stage weighted concatenation, where the inputs to the fusion moduleare first concatenated and then fed into a 1 × 1 convolution to output afeature map the same size as any of the inputs. (C) is learnable multi-stageweighted concatenation, in which each two input feature maps are concate-nated and fed into a 1 × 1 convolution which outputs a new feature mapthat is then concatenated with the next input to be fed into the next 1× 1convolution. The total number of 1× 1 convolutions is L− 1 where L is thenumber of layers of the transformer.705.2. EXPERIMENTAL ANALYSISTable 5.2: Performance comparison with state-of-the-art tracking methods.Our model outperforms the state-of-the-art ViT-base methods by a largemargin while competing with bigger ViT-Large and ViT-Giant. The twobest results are red and green. The Asterisk ∗ indicates that all modelstested on the GOT-10k test set are solely trained on the GOT-10k trainingset.LaSOT [38] LaSOText [39] TrackingNet [114] GOT-10k∗ [62]TrackerSUC PNorm P SUC PNorm P SUC PNorm P AO SR0.5 SR0.75Trackers with ViT-BaseTransT [21] 64.9 73.8 69.0 - - - 81.4 86.7 80.3 67.1 76.8 60.9AutoMatch [193] 58.3 - 59.9 37.6 - 43.0 76.0 - 72.6 65.2 76.6 54.3STARK [176] 67.1 77.0 - - - - 82.0 86.9 - 68.8 78.1 64.1KeepTrack [110] 67.1 77.2 70.2 48.2 - - - - - - - -SBT [171] 66.7 - 71.1 - - - - - - 70.4 80.8 64.7AiATrack [44] 69.0 79.4 73.8 47.7 55.6 55.4 - - - 69.6 80.0 63.2SimTrack [15] 70.5 79.7 - - - - - - - 69.8 78.8 66.0OSTrack [182] 71.1 81.1 77.6 50.5 61.3 57.6 83.9 88.5 83.2 73.7 83.2 70.8SwinTrack [100] 71.3 - 76.5 49.1 - 55.6 84.0 - 82.8 72.4 80.5 67.8DropTrack [164] 71.8 81.8 78.1 52.7 63.9 60.2 - - - 75.9 86.8 72.0SeqTrack [20] 72.5 81.5 79.3 50.7 61.6 57.5 85.5 89.8 85.8 74.8 81.9 72.2ARTrack [163] 72.6 81.7 79.1 51.9 62.0 58.5 85.1 89.1 84.8 75.5 84.3 74.3CiteTracker [92] 69.7 78.6 75.7 - - - 84.5 89.0 84.2 74.7 84.3 73.0ROMTrack [12] 71.4 81.4 78.2 51.3 62.4 58.6 84.1 89.0 83.7 74.2 84.3 72.4MixViT-B [26] 70.4 80.4 76.7 - - - 84.5 89.1 83.7 72.5 82.4 69.9LoRAT-B [99] 72.9 82.0 79.1 51.7 65.1 59.2 84.1 88.5 83.2 74.0 83.0 73.2ARTrackV2 [3] 73.0 82.0 79.6 52.9 63.4 59.1 85.7 89.8 85.5 77.5 86.0 75.5ODTrack-B [199] 73.2 83.2 80.6 52.4 63.9 60.1 85.1 90.1 84.9 77.0 87.9 75.1DMTrack (Ours) 73.2 83.4 80.7 53.5 65.1 61.5 85.7 91.7 90.9 79.1 89.3 77.7Trackers with Bigger ViTsLoRAT-G [99] 76.2 85.4 83.5 54.9 69.0 63.2 85.9 90.2 86.3 79.0 87.8 81.0MixViT-L [26] 72.4 82.2 80.1 - - - 85.4 90.2 85.7 75.7 85.3 75.1ODTrack-L [199] 74.0 84.2 82.3 53.9 65.4 61.7 86.1 91.0 86.7 78.2 87.2 77.3ARTrackV2-L [3] 73.6 82.8 81.1 53.4 63.7 60.2 86.1 90.4 86.2 79.5 87.8 79.6We train another version of our model solely on the GOT-10k to complywith the one-shot protocol [62]. This version is trained for 119 epochs, anda factor of 10 reduces the learning rate after 80 epochs. Other settings arekept the same as the full version.5.2.1.2 Tracking DetailsWe adopt the same tracking (inference) settings by ODTrack [199]. Theproposed model uses three reference images per inference iteration. More-over, we compare our model’s number of parameters and MACs (Multiply-Accumulate operations) to other state-of-the-art vision transformers in table5.3.715.2. EXPERIMENTAL ANALYSISTable 5.3: Demonstration of DMTrack efficiency according to MACs, numberof parameters, and GOT-10k test set Average Overlap (AO). Our modelexceeds computationally comparable models in terms of AO by at least 2%.Moreover, it matches the performance of a much bigger model, LORAT-G.Tracker Search Size MACs (G) # of Parameters (M) AO (%) on GOT-10k TestsetTrackers with ViT-baseSeqTrack [20] 384 148 89 74.5ODTrack-B [199] 384 73 92 77.0LoRAT-B [99] 378 97 99 74.0DMTrack (Ours) 384 91 92.14 79.1Trackers with Bigger ViTsLoRAT-L [99] 378 325 336 77.5LoRAT-G [99] 378 1161 1216 79.0Table 5.4: Experi-menting necessity ofdeformable convolu-tions.AlternativeGOT-10kAO SR0.5 SR0.75Regular 67.0 72.6 60.1Deformable 79.0 89.3 77.5Remove 77.0 87.9 75.1Table 5.5: Experiment-ing selected counts ofmasked representations.# of Masked ColumnsGOT-10kAO SR0.5 SR0.750 78.4 88.4 77.810 79.0 89.3 77.550 78.3 88.7 77.0Table 5.6: Experiment-ing the masking orderat the best number ofmasked representations10.AlternativeGOT-10kAO SR0.5 SR0.75Deformable + Masking 78.1 88.0 76.6Masking + Deformable 79.0 89.3 77.55.2.2 Comparison with Current State-of-the-Art TrackersWe empirically verify the effectiveness of our model by comparing it with thestate-of-the-art (SOTA) trackers on 7 leading visual tracking benchmarks.The overall results are shown in Table 5.2.5.2.2.1 LaSOTLaSOT [38] is large-scale highly challenging single object tracking dataset.It includes 1400 sequences, among which 1120 are used for training and 280for testing with an average sequence length of 2500 frames. The performanceof DMTrack on LaSOT is reported in Table 5.2.5.2.2.2 LaSOTextLaSOText [39] is an extension of the test set of LaSOT [38] by 150 challeng-ing long-term test sequences. We report the performance on LaSOText asdepicted in Table 5.2.725.2. EXPERIMENTAL ANALYSISTable 5.7: Experiment-ing the type of maskedrepresentations.AlternativeGOT-10kAO SR0.5 SR0.75Token 78.1 88.2 75.6Embeddings 79.0 89.3 77.5Both 55.0 59.1 50.0Table 5.8: Experi-menting high-level fu-sion variants.AlternativeGOT-10kAO SR0.5 SR0.75Bypass 79.0 89.3 77.5Sequential 58.2 66.3 43.7Table 5.9: Experi-menting the low-levelfusion variants.AlternativeGOT-10kAO SR0.5 SR0.75Add 78.1 87.3 75.7Multi-level Fusion 79.1 89.3 77.7One-level Fusion 76.9 87.3 75.0Table 5.10: State-of-the-art demonstration on UAV123 and OTB-100ATOM [29] DiMP [7] TransT [21] MixViT [26] SeqTrack [20] SBT [171] OSTrack-B [182] ODTrack-B [199] ARTrack-B [163] ARTrackV2-B [3] DMTrack (Ours)UAV123 [112] 63.2 64.3 68.1 68.7 68.5 - 70.7 - 67.7 69.9 68.6OTB-100 [168] 66.3 68.4 69.6 70.0 - 70.9 - 72.3 - - 70.45.2.2.3 TrackingNetTrackingNet [114] is a large-scale, short-term, challenging single object track-ing dataset. It comprises 30132 training sequences and 511 testing sequences.As per the results in Table 5.2, DMTrack outperforms ARTrackV2 [3] re-garding the PNorm by 1.9% and ODTrack [199] regarding the same metricby 1.6%. Moreover, DMTrack outperforms the best model regarding theprecision metric by 4.2%.5.2.2.4 GOT-10kGOT-10k is a challenging large-scale single object tracking dataset that con-tains 9334 training, 180 validation, and 180 test sequences. Trackers evalu-ated on GOT-10k must be solely trained on its training split following theone-shot protocol. Our DMTrack sets a new SOTA on GOT-10k AverageOverlap (AO) of 79.1% at a cheap cost of an additional 140k parameters.The new SOTA is above the previous one of ARTrackV2 [199] by 1.5%. Wealso establish a new SR0.5 boundary at 89.3% above the closest ODTrack[199] by 1.4%.5.2.2.5 UAV123 and OTB100We extend our experiments on two small-scale datasets to verify the gener-alizability of our model. UAV123 [112] is a drone-view dataset that contains123 test sequences. OTB100 [168] is a challenging 100 test sequences. Wereport the results in Table 5.10.735.2. EXPERIMENTAL ANALYSIS5.2.3 Ablation Study and AnalysisTo gain a deep, comprehensive understanding of the different components ofour model, we conduct several ablation experiments on the GOT-10k, OTBand UAV123 datasets. During all ablations, DMTrack is only trained on theGOT-10k training sets. In all tables associated with ablations, the row withdefault settings is coloured.5.2.3.1 Necessity of Deformable ConvolutionWe demonstrate the importance of deformable convolutions by testing threedifferent variants of our model as shown in Table 5.4. The conducted exper-iments show that deformable convolutions enhance the model’s performancecompared to regular convolutions. This experiment shows that deformableconvolutions can overcome the random masking and sample robust represen-tations from the transformer attentional feature maps. Moreover, deformableconvolutions force the transformer to output more robust maps than trainingthe network without convolutions.5.2.3.2 Masking RatioAs masking is carried on deep representations of the transformer, it is in-tuitive to think that the masking ratio has to be low so that the trans-former does not lose important information. This can be considered anadditional dropout regularization layer [141]. As depicted in Table 5.5, ex-periments demonstrate that masking number of the embedding yields themost favourable outcomes.5.2.3.3 When to MaskWe validate the intuitive decision of masking the transformer representationsbefore feeding them into the deformable masking module in Table 5.6. Itcan be shown that masking the transformer representations (before) behavessimilarly to masking the deformable representations (after). The defaultsetting is before.5.2.3.4 What to MaskWe test the three scenarios presented in Fig. 5.4 and put forth the results inTable 5.7. From the table,masking on the embedding dimension (vari-ant A) works the best. This can be justified as follows. Provided that thisdimension represents the embedding of the various combined template and745.2. EXPERIMENTAL ANALYSISsearch tokens as shown in Fig 5.2, this ablation shows that a tracking modellearns easy combinations of template and search embeddings that degradesits test performance, hence, uniformly masking portions of the learned em-beddings improves its quality. Nevertheless, when masking along the tokendimension or masking along both the embedding and token dimension con-currently, the training and test AO fall by %, indicating that the informationrepresented in the token dimension is crucial.5.2.3.5 High-Level FusionWe ablate the different alternatives of information transmission betweenlayers shown in Fig 5.2. The results shown in Table 5.8 are in favour ofthe bypass alternative (variant A), which indicates the transformer’s abil-ity to leverage the new representations. The sequential alternative pushesthe transformer to depend on the location prior learned by the deformableconvolutions, which hinders the transformer’s generalizability.5.2.3.6 Low-Level FusionAmong the three alternatives for low-level fusion (information passing be-tween layers) depicted in Fig. 5.5, we choose the learnable multi-stageconcatenation (variant C) to be the default. Experimental results in Ta-ble 5.9 support this choice. Intuitively, the chosen variant leverages more1× 1 convolutions, increasing the overall model flexibility and expressivity.5.2.4 VisualizationWe qualitatively show the effectiveness of DMTrack on different trackers onthe GOT-10k testset, the OTB-100 and the UAV123 benchmarks in Fig 5.6.In especially challenging hard distractors and extreme occlusion scenarios,our model tightly tracks the object of interest, indicating that it learnedhigh-quality representations.We also test our model on two sequences recorded by webcam to quali-tatively demonstrate the zero-shot performance of DMTrack as depicted inFig 5.7. The figure shows that DMTrack tracks newly unseen objects innovel sequences with a highly accurate bounding box.5.2.5 LimitationsAlthough our model sets a new SOTA upfront, it has limitations. Onelimitation is the increased complexity of the model with the addition of the755.2. EXPERIMENTAL ANALYSISFigure 5.6: Visual results of DMTrack on three different datasets.Figure 5.7: Visual results of DMTrack on two webcam sequences.765.3. DISCUSSION AND FUTURE DIRECTIONSdeformable convolutions. This led to performing more MACs than the vanillaODTrack [199]. Another limitation intrinsic to the ViT backbone is the highdemand for resources. Some improvements might be enhancing the ViT tobe more efficient or incorporating a more tracking-specific pretraining todecrease the training time. We excluded TrackingNet[114] from training ourmodel because of the limited computational resources, resulting in limitedperformance on LaSOT datasets [38, 39].5.3 Discussion and Future DirectionsIn this chapter, we introduced DMTrack, a new tracker that achieves state-of-the-art performance for single object tracking leveraging robust represen-tations learned by the proposed deformable masking module. Our modeloutperforms much bigger models, such as LORAT-G and ODTrack-L, whilebeing trained using fewer resources. Although achieving favourable per-formance, there is still room for improvement. As a future pathway, thedeformable masking module might be replaced by an adaptive module thatdecides the masking ratio for better performance. Moreover, to address theViT resources limitation, our method might be installed in a recent archi-tecture such as Mamba [204].77Chapter 6ConclusionsBased on the work from the three main chapters of the thesis, it is evidentthat the landscape of single-object tracking is evolving rapidly, with emergingtheoretical techniques in deep learning playing a crucial role in shaping itstrajectory.Chapter 3 provided a comprehensive survey of these emerging tech-niques, highlighting the shift towards the vision transformer paradigm insingle-object tracking research. This transition is attributed to several fac-tors, including the inherent advantages of transformer architectures, such aslarger receptive fields and robust representation capabilities, as well as theavailability of large-scale tracking datasets that complement the transformerarchitecture. Additionally, the incorporation of natural language and the ex-plicit learning of temporal correspondence have further enriched the trackingmodels, paving the way for generic cross-modal models and enhancing theirability to capture temporal dependencies effectively.Furthermore, the significance of backbone pretraining, both in supervisedand self-supervised manners, has been underscored, along with the potentialof generative models, particularly in enhancing the discrimination abilitiesof tracking models. As future pathways, it is envisioned that research willincreasingly focus on integrating autoregressive temporal cues with robustvisual representations, potentially through joint approaches that penalize re-construction errors while maximizing the likelihood of autoregressive bound-ing box tokens. Additionally, novel pretext tasks for self-supervised tracking,such as Mixed Autoencoder and No More Shortcuts, are expected to be ex-plored to further enhance the discriminative and temporal features learnedby tracking models.Chapter 4 delved into the significance of bounding box regression in con-junction with visual interaction features of one-stream tracking transformers.It was demonstrated that the performance of transformer tracking modelsheavily depends on the effectiveness of the bounding box regression network,with experiments showcasing the superiority of Inception and deformableinception bounding box networks over existing methods. As a future path-way, further exploration into emerging CNN techniques to find the optimal78bounding box network that balances inductive bias and global receptive fieldis suggested.Chapter 5 introduced DMTrack, a novel tracker leveraging deformablemasking modules to achieve state-of-the-art performance in single-objecttracking while using fewer resources compared to larger models. Despite itsfavourable performance, there remains room for improvement, particularlythrough the exploration of adaptive modules to enhance the masking ratioand the integration of DMTrack into recent architectures like Mamba toaddress resource limitations.In conclusion, the thesis underscores the dynamic nature of single-objecttracking research, driven by the adoption of emerging theoretical techniquesin deep learning. By leveraging the strengths of vision transformers, naturallanguage incorporation, explicit learning of temporal correspondence, back-bone pretraining, generative models, and innovative bounding box regressiontechniques, the field is poised for further advancements toward more robustand efficient tracking models.79Bibliography[1] Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, LeonidPishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Pose-track: A benchmark for human pose estimation and tracking. In 2018IEEE/CVF Conference on Computer Vision and Pattern Recognition,pages 5167–5176, 2018.[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layernormalization. arXiv preprint arXiv:1607.06450, 2016.[3] Yifan Bai, Zeyang Zhao, Yihong Gong, and Xing Wei. Artrackv2:Prompting autoregressive tracker where to look and how to describe,2024.[4] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua.Cvae-gan: fine-grained image generation through asymmetric train-ing. In Proceedings of the IEEE international conference on computervision, pages 2745–2754, 2017.[5] Luca Bertinetto, Jack Valmadre, Joa˜o F. Henriques, Andrea Vedaldi,and Philip H. S. Torr. Fully-convolutional siamese networks for objecttracking. In Gang Hua and Herve´ Je´gou, editors, Computer Vision –ECCV 2016 Workshops, pages 850–865, Cham, 2016. Springer Inter-national Publishing.[6] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi,and Philip HS Torr. Fully-convolutional siamese networks for objecttracking. In Computer Vision–ECCV 2016 Workshops: Amsterdam,The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II14, pages 850–865. Springer, 2016.[7] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte.Learning discriminative model prediction for tracking. In Proceed-ings of the IEEE/CVF International Conference on Computer Vision(ICCV), October 2019.80Bibliography[8] Goutam Bhat, Felix Ja¨remo Lawin, Martin Danelljan, Andreas Robin-son, Michael Felsberg, Luc Van Gool, and Radu Timofte. Learningwhat to learn for video object segmentation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,2020, Proceedings, Part II 16, pages 777–794. Springer, 2020.[9] A. Blake and M. Isard. A mixed-state condensation tracker with au-tomatic model-switching. In Computer Vision, IEEE InternationalConference on, page 107, Los Alamitos, CA, USA, jan 1998. IEEEComputer Society.[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared DKaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam,Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss,Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh,Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen,Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark,Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever,and Dario Amodei. Language models are few-shot learners. InH. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, edi-tors, Advances in Neural Information Processing Systems, volume 33,pages 1877–1901. Curran Associates, Inc., 2020.[11] Lei Cai, Hongyang Gao, and Shuiwang Ji. Multi-stage variationalauto-encoders for coarse-to-fine image generation. In Proceedings ofthe 2019 SIAM International Conference on Data Mining, pages 630–638. SIAM, 2019.[12] Y. Cai, J. Liu, J. Tang, and G. Wu. Robust object modeling for visualtracking. In 2023 IEEE/CVF International Conference on ComputerVision (ICCV), pages 9555–9566, Los Alamitos, CA, USA, oct 2023.IEEE Computer Society.[13] Lahiru D Chamain, Siyu Qi, and Zhi Ding. End-to-end image classifi-cation and compression with variational autoencoders. IEEE Internetof Things Journal, 9(21):21916–21931, 2022.[14] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman. Maskgit:Masked generative image transformer. In 2022 IEEE/CVF Conferenceon Computer Vision and Pattern Recognition (CVPR), pages 11305–11315, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.81Bibliography[15] Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Wei-hao Gan, Wei Wu, and Wanli Ouyang. Backbone is all your need:A simplified architecture for visual object tracking. In Computer Vi-sion – ECCV 2022: 17th European Conference, page 375–392, Berlin,Heidelberg, 2022. Springer-Verlag.[16] Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit-YanYeung. Mixed autoencoder for self-supervised visual representationlearning. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), pages 22742–22751, June2023.[17] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet:Diffusion model for object detection. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision (ICCV), pages 19830–19843, October 2023.[18] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hin-ton. A simple framework for contrastive learning of visual representa-tions. In Hal Daume´ III and Aarti Singh, editors, Proceedings of the37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 1597–1607. PMLR,13–18 Jul 2020.[19] Xiangru Chen, Yanan Sun, Mengjie Zhang, and Dezhong Peng. Evolv-ing deep convolutional variational autoencoders for image classifica-tion. IEEE Transactions on Evolutionary Computation, 25(5):815–829,2020.[20] Xin Chen, Houwen Peng, Dong Wang, Huchuan Lu, and Han Hu.Seqtrack: Sequence to sequence learning for visual object tracking. InProceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), pages 14572–14581, June 2023.[21] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, andHuchuan Lu. Transformer tracking. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR),pages 8126–8135, June 2021.[22] J. Choi, J. Kwon, and K. Lee. Deep meta learning for real-time target-aware visual tracking. In 2019 IEEE/CVF International Conferenceon Computer Vision (ICCV), pages 911–920, Los Alamitos, CA, USA,nov 2019. IEEE Computer Society.82Bibliography[23] Janghoon Choi, Sungyong Baik, Myungsub Choi, Junseok Kwon, andKyoung Mu Lee. Visual tracking by adaptive continual meta-learning.IEEE Access, 10:9022–9035, 2022.[24] S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metricdiscriminatively, with application to face verification. In 2005 IEEEComputer Society Conference on Computer Vision and Pattern Recog-nition (CVPR’05), volume 1, pages 539–546 vol. 1, 2005.[25] Dorin Comaniciu and Peter Meer. Mean shift: A robust approach to-ward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell.,24:603–619, 2002.[26] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. Mixformer:End-to-end tracking with iterative mixed attention. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recogni-tion, pages 13608–13618, 2022.[27] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu,and Yichen Wei. Deformable convolutional networks. In 2017 IEEEInternational Conference on Computer Vision (ICCV), pages 764–773,2017.[28] Kenan Dai, Yunhua Zhang, Dong Wang, Jianhua Li, Huchuan Lu,and Xiaoyun Yang. High-performance long-term tracking with meta-updater. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), June 2020.[29] M. Danelljan, G. Bhat, F. Khan, and M. Felsberg. Atom: Accuratetracking by overlap maximization. In 2019 IEEE/CVF Conference onComputer Vision and Pattern Recognition (CVPR), pages 4655–4664,Los Alamitos, CA, USA, jun 2019. IEEE Computer Society.[30] Ishan Rajendrakumar Dave, Simon Jenni, and Mubarak Shah. Nomore shortcuts: Realizing the potential of temporal self-supervision,2023.[31] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihoodfrom incomplete data via the em algorithm. Journal of the RoyalStatistical Society. Series B (Methodological), 39(1):1–38, 1977.[32] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 200983BibliographyIEEE Conference on Computer Vision and Pattern Recognition, pages248–255, 2009.[33] Michel Deudon. Learning semantic similarity in a continuous space. InS. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi,and R. Garnett, editors, Advances in Neural Information ProcessingSystems, volume 31. Curran Associates, Inc., 2018.[34] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat ganson image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin,P.S. Liang, and J. Wortman Vaughan, editors, Advances in NeuralInformation Processing Systems, volume 34, pages 8780–8794. CurranAssociates, Inc., 2021.[35] Xingping Dong, Jianbing Shen, Ling Shao, and Fatih Porikli. Clnet:A compact latent network for fast adjusting siamese trackers. In Euro-pean Conference on Computer Vision, pages 378–395. Springer, 2020.[36] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis-senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani,Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit,and Neil Houlsby. An image is worth 16x16 words: Transformers forimage recognition at scale. In International Conference on LearningRepresentations, 2021.[37] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang,and Qi Tian. Centernet: Keypoint triplets for object detection. In 2019IEEE/CVF International Conference on Computer Vision (ICCV),pages 6568–6577, 2019.[38] H. Fan, L. Lin, F. Yang, P. Chu, G. Deng, S. Yu, H. Bai, Y. Xu, C. Liao,and H. Ling. Lasot: A high-quality benchmark for large-scale singleobject tracking. In 2019 IEEE/CVF Conference on Computer Visionand Pattern Recognition (CVPR), pages 5369–5378, Los Alamitos, CA,USA, jun 2019. IEEE Computer Society.[39] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, SijiaYu, Harshit, Mingzhen Huang, Juehuan Liu, Yong Xu, Chunyuan Liao,Lin Yuan, and Haibin Ling. Lasot: A high-quality large-scale singleobject tracking benchmark. Int. J. Comput. Vision, 129(2):439–461,feb 2021.84Bibliography[40] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup andYee Whye Teh, editors, Proceedings of the 34th International Con-ference on Machine Learning, volume 70 of Proceedings of MachineLearning Research, pages 1126–1135. PMLR, 06–11 Aug 2017.[41] Zhihong Fu, Qingjie Liu, Zehua Fu, and Yunhong Wang. Stmtrack:Template-free visual tracking with space-time memory networks. InProceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), pages 13774–13783, June 2021.[42] H. Galoogahi, A. Fagg, C. Huang, D. Ramanan, and S. Lucey. Needfor speed: A benchmark for higher frame rate object tracking. In2017 IEEE International Conference on Computer Vision (ICCV),pages 1134–1143, Los Alamitos, CA, USA, oct 2017. IEEE ComputerSociety.[43] Junyu Gao, Tianzhu Zhang, and Changsheng Xu. Graph convolutionaltracking. In 2019 IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), pages 4644–4654, 2019.[44] Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, and Jun-song Yuan. Aiatrack: Attention in attention for transformer visualtracking. In Computer Vision – ECCV 2022: 17th European Confer-ence, page 146–164, Berlin, Heidelberg, 2022. Springer-Verlag.[45] Shenyuan Gao, Chunluan Zhou, and Jun Zhang. Generalized relationmodeling for transformer tracking. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR),pages 18686–18695, June 2023.[46] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Richfeature hierarchies for accurate object detection and semantic segmen-tation. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 580–587, 2014.[47] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozair, Aaron Courville, and Y. Bengio. Genera-tive adversarial networks. Advances in Neural Information ProcessingSystems, 3, 06 2014.[48] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dong-dong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion85Bibliographymodel for text-to-image synthesis. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR),pages 10696–10706, June 2022.[49] Jie Guo, Tingfa Xu, Shenwang Jiang, and Ziyi Shen. Generating reli-able online adaptive templates for visual tracking. In 2018 25th IEEEInternational Conference on Image Processing (ICIP), pages 226–230,2018.[50] Changhee Han, Hideaki Hayashi, Leonardo Rundo, Ryosuke Araki,Wataru Shimoda, Shinichi Muramatsu, Yujiro Furukawa, GiancarloMauri, and Hideki Nakayama. Gan-based synthetic brain mr imagegeneration. In 2018 IEEE 15th International Symposium on Biomed-ical Imaging (ISBI 2018), pages 734–738, 2018.[51] Yamin Han, Peng Zhang, Wei Huang, Yufei Zha, Garth DouglasCooper, and Yanning Zhang. Robust visual tracking based on ad-versarial unlabeled instance generation with label smoothing loss reg-ularization. Pattern Recognition, 97:107027, 2020.[52] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dolla´r, andRoss Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pat-tern recognition, pages 16000–16009, 2022.[53] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.Momentum contrast for unsupervised visual representation learning.In Proceedings of the IEEE/CVF conference on computer vision andpattern recognition, pages 9729–9738, 2020.[54] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Maskr-cnn. In Proceedings of the IEEE International Conference on Com-puter Vision (ICCV), Oct 2017.[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deepresidual learning for image recognition. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPR),June 2016.[56] Joa˜o F. Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista.High-speed tracking with kernelized correlation filters. IEEE Trans-actions on Pattern Analysis and Machine Intelligence, 37(3):583–596,2015.86Bibliography[57] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledgein a neural network. arXiv preprint arXiv:1503.02531, 2015.[58] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prob-abilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal-can, and H. Lin, editors, Advances in Neural Information ProcessingSystems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.[59] Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mo-hammad Norouzi, and Tim Salimans. Cascaded diffusion models forhigh fidelity image generation. J. Mach. Learn. Res., 23(1), jan 2022.[60] Sepp Hochreiter and Ju¨rgen Schmidhuber. Long short-term memory.Neural computation, 9:1735–80, 12 1997.[61] Weiming Hu, Qiang Wang, Li Zhang, Luca Bertinetto, and Philip H.S.Torr. Siammask: A framework for fast online object tracking andsegmentation. IEEE Transactions on Pattern Analysis and MachineIntelligence, 45(3):3072–3089, 2023.[62] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A largehigh-diversity benchmark for generic object tracking in the wild.IEEE Transactions on Pattern Analysis and Machine Intelligence,43(5):1562–1577, 2021.[63] Wenhui Huang, Jason Gu, and Yinchen Guo. Depth-aware objecttracking with a conditional variational autoencoder. IEEE Access,9:94537–94547, 2021.[64] Yucheng Huang, Ziwang Xiao, Eksan Firkat, Jinlai Zhang, Dan-feng Wu, and Askar Hamdulla. Spatio-temporal mix deformable fea-ture extractor in visual tracking. Expert Systems with Applications,237:121377, 2024.[65] S. Javed, M. Danelljan, F. Khan, M. Khan, M. Felsberg, and J. Matas.Visual object tracking with discriminative filters and siamese networks:A survey and outlook. IEEE Transactions on Pattern Analysis andMachine Intelligence, 45(05):6552–6574, may 2023.[66] Rudolph Emil Kalman. A new approach to linear filtering and predic-tion problems. Transactions of the ASME–Journal of Basic Engineer-ing, 82(Series D):35–45, 1960.87Bibliography[67] Will Kay, Joa˜o Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier,Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back,Apostol Natsev, Mustafa Suleyman, and Andrew Zisserman. The ki-netics human action video dataset. ArXiv, abs/1705.06950, 2017.[68] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.Bert: Pre-training of deep bidirectional transformers for language un-derstanding. In Proceedings of naacL-HLT, volume 1, page 2, 2019.[69] Zia Khan, T. Balch, and F. Dellaert. Mcmc-based particle filtering fortracking a variable number of interacting targets. IEEE Transactionson Pattern Analysis and Machine Intelligence, 27(11):1805–1819, 2005.[70] Diederik P Kingma and MaxWelling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013.[71] Thomas N. Kipf and Max Welling. Semi-supervised classification withgraph convolutional networks. In International Conference on Learn-ing Representations, 2017.[72] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.Pointrend: Image segmentation as rendering. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), June 2020.[73] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: Theefficient transformer. In International Conference on Learning Repre-sentations, 2020.[74] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, RomanPflugfelder, Luka Cehovin Zajc, Tomas Vojir, Goutam Bhat, AlanLukezic, Abdelrahman Eldesokey, Gustavo Fernandez, Alvaro Garcia-Martin, Alvaro Iglesias-Arias, A. Aydin Alatan, Abel Gonzalez-Garcia,Alfredo Petrosino, Alireza Memarmoghadam, Andrea Vedaldi, AndrejMuhic, Anfeng He, Arnold Smeulders, Asanka G. Perera, Bo Li, BoyuChen, Changick Kim, Changsheng Xu, Changzhen Xiong, Cheng Tian,Chong Luo, Chong Sun, Cong Hao, Daijin Kim, Deepak Mishra, Dem-ing Chen, Dong Wang, Dongyoon Wee, Efstratios Gavves, Erhan Gun-dogdu, Erik Velasco-Salido, Fahad Shahbaz Khan, Fan Yang, Fei Zhao,Feng Li, Francesco Battistone, George De Ath, Gorthi R. K. S. Sub-rahmanyam, Guilherme Bastos, Haibin Ling, Hamed Kiani Galoogahi,Hankyeol Lee, Haojie Li, Haojie Zhao, Heng Fan, Honggang Zhang,88BibliographyHorst Possegger, Houqiang Li, Huchuan Lu, Hui Zhi, Huiyun Li,Hyemin Lee, Hyung Jin Chang, Isabela Drummond, Jack Valmadre,Jaime Spencer Martin, Javaan Chahl, Jin Young Choi, Jing Li, JinqiaoWang, Jinqing Qi, Jinyoung Sung, Joakim Johnander, Joao Henriques,Jongwon Choi, Joost van de Weijer, Jorge Rodriguez Herranz, Jose M.Martinez, Josef Kittler, Junfei Zhuang, Junyu Gao, Klemen Grm,Lichao Zhang, Lijun Wang, Lingxiao Yang, Litu Rout, Liu Si, LucaBertinetto, Lutao Chu, Manqiang Che, Mario Edoardo Maresca, Mar-tin Danelljan, Ming-Hsuan Yang, Mohamed Abdelpakey, MohamedShehata, Myunggu Kang, Namhoon Lee, Ning Wang, Ondrej Miksik,P. Moallem, Pablo Vicente-Monivar, Pedro Senna, Peixia Li, PhilipTorr, Priya Mariam Raju, Qian Ruihe, Qiang Wang, Qin Zhou, QingGuo, Rafael Martin-Nieto, Rama Krishna Gorthi, Ran Tao, RichardBowden, Richard Everson, Runling Wang, Sangdoo Yun, SeokeonChoi, Sergio Vivas, Shuai Bai, Shuangping Huang, Sihang Wu, SimonHadfield, Siwen Wang, Stuart Golodetz, Tang Ming, Tianyang Xu,Tianzhu Zhang, Tobias Fischer, Vincenzo Santopietro, Vitomir Struc,Wang Wei, Wangmeng Zuo, Wei Feng, Wei Wu, Wei Zou, WeimingHu, Wengang Zhou, Wenjun Zeng, Xiaofan Zhang, Xiaohe Wu, Xiao-Jun Wu, Xinmei Tian, Yan Li, Yan Lu, Yee Wei Law, Yi Wu, YiannisDemiris, Yicai Yang, Yifan Jiao, Yuhong Li, Yunhua Zhang, YuxuanSun, Zheng Zhang, Zheng Zhu, Zhen-Hua Feng, Zhihui Wang, andZhiqun He. The sixth visual object tracking vot2018 challenge re-sults. In Proceedings of the European Conference on Computer Vision(ECCV) Workshops, September 2018.[75] Matej Kristan, Alesˇ Leonardis, Jiˇr´ı Matas, Michael Felsberg,Roman Pflugfelder, Joni-Kristian Ka¨ma¨ra¨inen, Martin Danelljan,Luka Cˇehovin Zajc, Alan Lukezˇicˇ, Ondrej Drbohlav, Linbo He, YushanZhang, Song Yan, Jinyu Yang, Gustavo Ferna´ndez, Alexander Haupt-mann, Alireza Memarmoghadam, A´lvaro Garc´ıa-Mart´ın, AndreasRobinson, Anton Varfolomieiev, Awet Haileslassie Gebrehiwot, Be-dirhan Uzun, Bin Yan, Bing Li, Chen Qian, Chi-Yi Tsai, Chris-tian Micheloni, Dong Wang, Fei Wang, Fei Xie, Felix Jaremo Lawin,Fredrik Gustafsson, Gian Luca Foresti, Goutam Bhat, Guangqi Chen,Haibin Ling, Haitao Zhang, Hakan Cevikalp, Haojie Zhao, HaoranBai, Hari Chandana Kuchibhotla, Hasan Saribas, Heng Fan, Hos-sein Ghanei-Yakhdan, Houqiang Li, Houwen Peng, Huchuan Lu, HuiLi, Javad Khaghani, Jesus Bescos, Jianhua Li, Jianlong Fu, JiaqianYu, Jingtao Xu, Josef Kittler, Jun Yin, Junhyun Lee, Kaicheng89BibliographyYu, Kaiwen Liu, Kang Yang, Kenan Dai, Li Cheng, Li Zhang, Li-jun Wang, Linyuan Wang, Luc Van Gool, Luca Bertinetto, MatteoDunnhofer, Miao Cheng, Mohana Murali Dasari, Ning Wang, NingWang, Pengyu Zhang, Philip H. S. Torr, Qiang Wang, Radu Timofte,Rama Krishna Sai Gorthi, Seokeon Choi, Seyed Mojtaba Marvasti-Zadeh, Shaochuan Zhao, Shohreh Kasaei, Shoumeng Qiu, ShuhaoChen, Thomas B. Scho¨n, Tianyang Xu, Wei Lu, Weiming Hu, Wen-gang Zhou, Xi Qiu, Xiao Ke, Xiao-Jun Wu, Xiaolin Zhang, XiaoyunYang, Xuefeng Zhu, Yingjie Jiang, Yingming Wang, Yiwei Chen,Yu Ye, Yuezhou Li, Yuncon Yao, Yunsung Lee, Yuzhang Gu, ZezhouWang, Zhangyong Tang, Zhen-Hua Feng, Zhijun Mai, Zhipeng Zhang,Zhirong Wu, and Ziang Ma. The eighth visual object tracking vot2020challenge results. In Adrien Bartoli and Andrea Fusiello, editors, Com-puter Vision – ECCV 2020 Workshops, pages 547–601, Cham, 2020.Springer International Publishing.[76] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, RomanPflugfelder, Luka Cˇehovin Zajc, Toma´sˇ Voj´ır, Gustav Ha¨ger, AlanLukezˇicˇ, Gustavo Fernandez Dominguez, Abhinav Gupta, AlfredoPetrosino, Alireza Memarmoghadam, Alvaro Garcia-Martin, Andre´sMontero, Andrea Vedaldi, Andreas Robinson, Andy Ma, Anton Var-folomieiev, and Zhizhen Chi. The visual object tracking vot2016 chal-lenge results. volume 9914, pages 777–823, 10 2016.[77] Matej Kristan, Alesˇ Leonardis, Jiri Matas, Michael Felsberg, RomanPflugfelder, Luka Cehovin Zajc, Toma´s Voj´ır, Gustav Ha¨ger, AlanLukezˇic, Abdelrahman Eldesokey, Gustavo Ferna´ndez, Alvaro Garcia-Martin, A. Muhic, Alfredo Petrosino, Alireza Memarmoghadam, An-drea Vedaldi, Antoine Manzanera, Antoine Tran, Aydin Alatan, Bog-dan Mocanu, Boyu Chen, Chang Huang, Changsheng Xu, Chong Sun,Dalong Du, David Zhang, Dawei Du, Deepak Mishra, Erhan Gun-dogdu, Erik Velasco-Salido, Fahad Shahbaz Khan, Francesco Battis-tone, Gorthi R. K. Sai Subrahmanyam, Goutam Bhat, Guan Huang,Guilherme Bastos, Guna Seetharaman, Hongliang Zhang, HouqiangLi, Huchuan Lu, Isabela Drummond, Jack Valmadre, Jae-chan Jeong,Jae-il Cho, Jae-Yeong Lee, Jana Noskova, Jianke Zhu, Jin Gao, JingyuLiu, Ji-Wan Kim, Joa˜o F. Henriques, Jose M. Mart´ınez, Junfei Zhuang,Junliang Xing, Junyu Gao, Kai Chen, Kannappan Palaniappan, KarelLebeda, Ke Gao, Kris M. Kitani, Lei Zhang, Lijun Wang, Lingx-iao Yang, Longyin Wen, Luca Bertinetto, Mahdieh Poostchi, Mar-tin Danelljan, Matthias Mueller, Mengdan Zhang, Ming-Hsuan Yang,90BibliographyNianhao Xie, Ning Wang, Ondrej Miksik, P. Moallem, Pallavi M.Venugopal, Pedro Senna, Philip H. S. Torr, Qiang Wang, Qifeng Yu,Qingming Huang, Rafael Mart´ın-Nieto, Richard Bowden, Risheng Liu,Ruxandra Tapu, Simon Hadfield, Siwei Lyu, Stuart Golodetz, SunglokChoi, Tianzhu Zhang, Titus Zaharia, Vincenzo Santopietro, Wei Zou,Weiming Hu, Wenbing Tao, Wenbo Li, Wengang Zhou, Xianguo Yu,Xiao Bian, Yang Li, Yifan Xing, Yingruo Fan, Zheng Zhu, ZhipengZhang, and Zhiqun He. The visual object tracking vot2017 challengeresults. In 2017 IEEE International Conference on Computer VisionWorkshops (ICCVW), pages 1949–1972, 2017.[78] Matej Kristan, Jir´ı Matas, Alesˇ Leonardis, Michael Felsberg, RomanPflugfelder, Joni-Kristian Ka¨ma¨ra¨inen, Luka Cehovin Zajc, Ondrej Dr-bohlav, Alan Lukezic, Amanda Berg, Abdelrahman Eldesokey, JaniKa¨pyla¨, Gustavo Ferna´ndez, Abel Gonzalez-Garcia, Alireza Memar-moghadam, Andong Lu, Anfeng He, Anton Varfolomieiev, AntoniChan, Ardhendu Shekhar Tripathi, Arnold Smeulders, Bala SurajPedasingu, Bao Xin Chen, Baopeng Zhang, Baoyuan Wu, Bi Li, BinHe, Bin Yan, Bing Bai, Bing Li, Bo Li, Byeong Hak Kim, ChaoMa, Chen Fang, Chen Qian, Cheng Chen, Chenglong Li, ChengquanZhang, Chi-Yi Tsai, Chong Luo, Christian Micheloni, Chunhui Zhang,Dacheng Tao, Deepak Gupta, Dejia Song, Dong Wang, EfstratiosGavves, Eunu Yi, Fahad Shahbaz Khan, Fangyi Zhang, Fei Wang,Fei Zhao, George De Ath, Goutam Bhat, Guangqi Chen, GuangtingWang, Guoxuan Li, Hakan Cevikalp, Hao Du, Haojie Zhao, HasanSaribas, Ho Min Jung, Hongliang Bai, Hongyuan Yu, Houwen Peng,Huchuan Lu, Hui Li, Jiakun Li, Jianhua Li, Jianlong Fu, Jie Chen, JieGao, Jie Zhao, Jin Tang, Jing Li, Jingjing Wu, Jingtuo Liu, JinqiaoWang, Jinqing Qi, Jinyue Zhang, John K. Tsotsos, Jong Hyuk Lee,Joost van de Weijer, Josef Kittler, Jun Ha Lee, Junfei Zhuang, KangkaiZhang, Kangkang Wang, Kenan Dai, Lei Chen, Lei Liu, Leida Guo,Li Zhang, Liang Wang, Liangliang Wang, Lichao Zhang, Lijun Wang,Lijun Zhou, Linyu Zheng, Litu Rout, Luc Van Gool, Luca Bertinetto,Martin Danelljan, Matteo Dunnhofer, Meng Ni, Min Young Kim,Ming Tang, Ming-Hsuan Yang, Naveen Paluru, Niki Martinel, PengfeiXu, Pengfei Zhang, Pengkun Zheng, Pengyu Zhang, Philip H.S. Torr,Qi Zhang Qiang Wang, Qing Guo, Radu Timofte, Rama Krishna Gor-thi, Richard Everson, Ruize Han, Ruohan Zhang, Shan You, Shao-Chuan Zhao, Shengwei Zhao, Shihu Li, Shikun Li, Shiming Ge, ShuaiBai, Shuosen Guan, Tengfei Xing, Tianyang Xu, Tianyu Yang, Ting91BibliographyZhang, Toma´s Vojir, Wei Feng, Weiming Hu, Weizhao Wang, WenjieTang, Wenjun Zeng, Wenyu Liu, Xi Chen, Xi Qiu, Xiang Bai, Xiao-JunWu, Xiaoyun Yang, Xier Chen, Xin Li, Xing Sun, Xingyu Chen, Xin-mei Tian, Xu Tang, Xue-Feng Zhu, Yan Huang, Yanan Chen, YanchaoLian, Yang Gu, Yang Liu, Yanjie Chen, Yi Zhang, Yinda Xu, YingmingWang, Yingping Li, Yu Zhou, Yuan Dong, Yufei Xu, Yunhua Zhang,Yunkun Li, Zeyu Wang Zhao Luo, Zhaoliang Zhang, Zhen-Hua Feng,Zhenyu He, Zhichao Song, Zhihao Chen, Zhipeng Zhang, Zhirong Wu,Zhiwei Xiong, Zhongjian Huang, Zhu Teng, and Zihan Ni. The seventhvisual object tracking vot2019 challenge results. In 2019 IEEE/CVFInternational Conference on Computer Vision Workshop (ICCVW),pages 2206–2241, 2019.[79] Janani Kugarajeevan, Thanikasalam Kokul, Amirthalingam Ra-manan, and Subha Fernando. Transformers in single object tracking:An experimental survey. IEEE Access, 2023.[80] Solomon Kullback and Richard A Leibler. On information and suffi-ciency. The annals of mathematical statistics, 22(1):79–86, 1951.[81] Junseok Kwon. Robust visual tracking based on variational auto-encoding markov chain monte carlo. Information Sciences, 512:1308–1323, 2020.[82] Hei Law and Jia Deng. Cornernet: Detecting objects as paired key-points. In Proceedings of the European conference on computer vision(ECCV), pages 734–750, 2018.[83] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basedlearning applied to document recognition. Proceedings of the IEEE,86(11):2278–2324, 1998.[84] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Ab-delrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettle-moyer. Bart: Denoising sequence-to-sequence pre-training for naturallanguage generation, translation, and comprehension. In Proceedingsof the 58th Annual Meeting of the Association for Computational Lin-guistics, pages 7871–7880, 2020.[85] Annan Li, Min Lin, Yi Wu, Ming-Hsuan Yang, and Shuicheng Yan.Nus-pro: A new visual tracking challenge. IEEE Transactions on Pat-tern Analysis and Machine Intelligence, 38(2):335–349, 2016.92Bibliography[86] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and JunjieYan. Siamrpn++: Evolution of siamese visual tracking with very deepnetworks. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), June 2019.[87] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High perfor-mance visual tracking with siamese region proposal network. In 2018IEEE/CVF Conference on Computer Vision and Pattern Recognition,pages 8971–8980, 2018.[88] Hanxi Li, Xinyu Wang, Fumin Shen, Yi Li, Fatih Porikli, and Ming-wen Wang. Real-time deep tracking via corrective domain adapta-tion. IEEE Transactions on Circuits and Systems for Video Technol-ogy, 29(9):2600–2612, 2019.[89] Jianan Li, Xiaodan Liang, Yunchao Wei, Tingfa Xu, Jiashi Feng, andShuicheng Yan. Perceptual generative adversarial networks for smallobject detection. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR), July 2017.[90] Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Ray-mond Fu. Tell me where to look: Guided attention inference network.2018 IEEE/CVF Conference on Computer Vision and Pattern Recog-nition, pages 9215–9223, 2018.[91] Peixia Li, Boyu Chen, Wanli Ouyang, Dong Wang, Xiaoyun Yang,and Huchuan Lu. Gradnet: Gradient-guided network for visual objecttracking. In Proceedings of the IEEE/CVF International Conferenceon Computer Vision (ICCV), October 2019.[92] X. Li, Y. Huang, Z. He, Y. Wang, H. Lu, and M. Yang. Citetracker:Correlating image and text for visual tracking. In 2023 IEEE/CVFInternational Conference on Computer Vision (ICCV), pages 9940–9949, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society.[93] Xin Li, Wenjie Pei, Yaowei Wang, Zhenyu He, Huchuan Lu, and Ming-Hsuan Yang. Self-supervised tracking via target-aware data synthesis.IEEE Transactions on Neural Networks and Learning Systems, pages1–12, 2022.[94] Xueting Li, Sifei Liu, Shalini De Mellow, Xiaolong Wang, Jan Kautz,and Ming-Hsuan Yang. Joint-task self-supervised learning for temporalcorrespondence. In NeurIPS, 2019.93Bibliography[95] Yihao Li, Jun Yu, Zhongpeng Cai, and Yuwen Pan. Cross-modaltarget retrieval for tracking by natural language. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) Workshops, pages 4931–4940, June 2022.[96] Zhenxin Li, Xuande Zhang, Long Xu, and Weiqiang Zhang. Fastand robust visual tracking with few-iteration meta-learning. Sensors,22(15), 2022.[97] Zhenyang Li, Ran Tao, Efstratios Gavves, Cees G. M. Snoek, andArnold W.M. Smeulders. Tracking by natural language specification.In Proceedings of the IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR), July 2017.[98] Zhizhong Li and Derek Hoiem. Learning without forgetting.IEEE transactions on pattern analysis and machine intelligence,40(12):2935–2947, 2017.[99] Liting Lin, Heng Fan, Zhipeng Zhang, Yaowei Wang, Yong Xu, andHaibin Ling. Tracking meets lora: Faster training, larger model,stronger performance, 2024.[100] Liting Lin, Heng Fan, Zhipeng Zhang, Yong Xu, and Haibin Ling.Swintrack: A simple and strong baseline for transformer tracking.In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, andA. Oh, editors, Advances in Neural Information Processing Systems,volume 35, pages 16743–16754. Curran Associates, Inc., 2022.[101] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar. Focal loss fordense object detection. In 2017 IEEE International Conference onComputer Vision (ICCV), pages 2999–3007, Los Alamitos, CA, USA,oct 2017. IEEE Computer Society.[102] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, PietroPerona, Deva Ramanan, Piotr Dolla´r, and C. Lawrence Zitnick. Mi-crosoft coco: Common objects in context. In David Fleet, Tomas Pa-jdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision –ECCV 2014, pages 740–755, Cham, 2014. Springer International Pub-lishing.[103] Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, and Anto-nio Torralba. Diverse image generation via self-conditioned gans. In94BibliographyProceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), June 2020.[104] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo.Swin transformer: Hierarchical vision transformer using shifted win-dows. In 2021 IEEE/CVF International Conference on Computer Vi-sion (ICCV), pages 9992–10002, Los Alamitos, CA, USA, oct 2021.IEEE Computer Society.[105] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regulariza-tion. arXiv preprint arXiv:1711.05101, 2017.[106] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regulariza-tion. In International Conference on Learning Representations, 2019.[107] Alan Lukezic, Ugur Kart, Jani Ka¨pyla¨, Ahmed Durmush, Joni-Kristian Kamarainen, Jiri Matas, and Matej Kristan. Cdtb: A colorand depth visual object tracking dataset and benchmark. In 2019IEEE/CVF International Conference on Computer Vision (ICCV),pages 10012–10021, 2019.[108] Run Luo, Zikai Song, Lintao Ma, Jinlin Wei, Wei Yang, and MinYang. Diffusiontrack: Diffusion model for multi-object tracking. arXivpreprint arXiv:2308.09905, 2023.[109] Chao Ma, Jia-Bin Huang, Xiaokang Yang, and Ming-Hsuan Yang. Hi-erarchical convolutional features for visual tracking. In 2015 IEEEInternational Conference on Computer Vision (ICCV), pages 3074–3082, 2015.[110] Christoph Mayer, Martin Danelljan, Danda Pani Paudel, and LucVan Gool. Learning target candidate association to keep track of whatnot to track. In 2021 IEEE/CVF International Conference on Com-puter Vision (ICCV), pages 13424–13434, 2021.[111] Anton Milan, Laura Leal-Taixe´, Ian D. Reid, Stefan Roth, and KonradSchindler. Mot16: A benchmark for multi-object tracking. ArXiv,abs/1603.00831, 2016.[112] Matthias Mueller, Neil Smith, and Bernard Ghanem. A benchmarkand simulator for uav tracking. In Computer Vision–ECCV 2016: 14thEuropean Conference, Amsterdam, The Netherlands, October 11–14,2016, Proceedings, Part I 14, pages 445–461. Springer, 2016.95Bibliography[113] Matthias Mueller, Neil Smith, and Bernard Ghanem. A benchmarkand simulator for uav tracking. In Bastian Leibe, Jiri Matas, NicuSebe, and MaxWelling, editors, Computer Vision – ECCV 2016, pages445–461, Cham, 2016. Springer International Publishing.[114] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, andBernard Ghanem. Trackingnet: A large-scale dataset and benchmarkfor object tracking in the wild. In Vittorio Ferrari, Martial Hebert,Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision –ECCV 2018, pages 310–327, Cham, 2018. Springer International Pub-lishing.[115] Siva Karthik Mustikovela, Shalini De Mello, Aayush Prakash, UmarIqbal, Sifei Liu, Thu Nguyen-Phuoc, Carsten Rother, and Jan Kautz.Self-supervised object detection via generative image synthesis. InProceedings of the IEEE/CVF International Conference on ComputerVision (ICCV), pages 8609–8618, October 2021.[116] H. Nam and B. Han. Learning multi-domain convolutional neuralnetworks for visual tracking. In 2016 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), pages 4293–4302, Los Alami-tos, CA, USA, jun 2016. IEEE Computer Society.[117] Jifeng Ning, Jimei Yang, Shaojie Jiang, Lei Zhang, and Ming-HsuanYang. Object tracking via dual linear structured svm and explicitfeature map. pages 4266–4274, 06 2016.[118] Xiaoying Pan, Minrui Yuan, Tianxin Zhang, and Hao Wang.Deformable attention object tracking network based on cross-correlation. Journal of Visual Communication and Image Representa-tion, 98:104039, 2024.[119] Dong Woo Park, Junseok Kwon, and Kyoung Mu Lee. Robust visualtracking using autoregressive hidden markov model. In 2012 IEEEConference on Computer Vision and Pattern Recognition, pages 1964–1971, 2012.[120] Eunbyung Park and Alexander C. Berg. Meta-tracker: Fast and robustonline adaptation for visual object trackers. In Proceedings of theEuropean Conference on Computer Vision (ECCV), September 2018.96Bibliography[121] Matthieu Paul, Martin Danelljan, Christoph Mayer, and LucVan Gool. Robust visual tracking by segmentation. In European Con-ference on Computer Vision, pages 571–588. Springer, 2022.[122] Duo Peng, Ping Hu, Qiuhong Ke, and Jun Liu. Diffusion-based imagetranslation with label guidance for domain adaptive semantic segmen-tation. In Proceedings of the IEEE/CVF International Conference onComputer Vision (ICCV), pages 808–820, October 2023.[123] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, andA. Sorkine-Hornung. A benchmark dataset and evaluation method-ology for video object segmentation. In 2016 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), pages 724–732,2016.[124] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating di-verse high-fidelity images with vq-vae-2. Advances in neural informa-tion processing systems, 32, 2019.[125] E. Real, J. Shlens, S. Mazzocchi, X. Pan, and V. Vanhoucke. Youtube-boundingboxes: A large high-precision human-annotated data set forobject detection in video. In 2017 IEEE Conference on Computer Vi-sion and Pattern Recognition (CVPR), pages 7464–7473, Los Alami-tos, CA, USA, jul 2017. IEEE Computer Society.[126] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn:Towards real-time object detection with region proposal networks. InC. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, edi-tors, Advances in Neural Information Processing Systems, volume 28.Curran Associates, Inc., 2015.[127] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian,Ian Reid, and Silvio Savarese. Generalized intersection over union.June 2019.[128] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, SanjeevSatheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla,Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet LargeScale Visual Recognition Challenge. International Journal of Com-puter Vision (IJCV), 115(3):211–252, 2015.[129] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakr-ishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual97Bibliographyexplanations from deep networks via gradient-based localization. In2017 IEEE International Conference on Computer Vision (ICCV),pages 618–626, 2017.[130] Jonathan R Shewchuk. An introduction to the conjugate gradientmethod without the agonizing pain. Technical report, USA, 1994.[131] Xingjian SHI, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kinWong, and Wang-chun WOO. Convolutional lstm network: A ma-chine learning approach for precipitation nowcasting. In C. Cortes,N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advancesin Neural Information Processing Systems, volume 28. Curran Asso-ciates, Inc., 2015.[132] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.Indoor segmentation and support inference from rgbd images. In An-drew Fitzgibbon, Svetlana Lazebnik, Pietro Perona, Yoichi Sato, andCordelia Schmid, editors, Computer Vision – ECCV 2012, pages 746–760, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.[133] Karen Simonyan and Andrew Zisserman. Very deep convolutional net-works for large-scale image recognition. In International Conferenceon Learning Representations, 2015.[134] Chon Hou Sio, Yu-Jen Ma, Hong-Han Shuai, Jun-Cheng Chen, andWen-Huang Cheng. S2siamfc: Self-supervised fully convolutionalsiamese network for visual tracking. In Proceedings of the 28th ACMInternational Conference on Multimedia, MM ’20, page 1948–1957,New York, NY, USA, 2020. Association for Computing Machinery.[135] Arnold W. M. Smeulders, Dung M. Chu, Rita Cucchiara, SimoneCalderara, Afshin Dehghan, and Mubarak Shah. Visual tracking: Anexperimental survey. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 36(7):1442–1468, 2014.[136] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and SuryaGanguli. Deep unsupervised learning using nonequilibrium thermo-dynamics. In Francis Bach and David Blei, editors, Proceedings ofthe 32nd International Conference on Machine Learning, volume 37of Proceedings of Machine Learning Research, pages 2256–2265, Lille,France, 07–09 Jul 2015. PMLR.98Bibliography[137] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structuredoutput representation using deep conditional generative models. InC. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, edi-tors, Advances in Neural Information Processing Systems, volume 28.Curran Associates, Inc., 2015.[138] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffu-sion implicit models. In 9th International Conference on LearningRepresentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.OpenReview.net, 2021.[139] Yibing Song, Chao Ma, Xiaohe Wu, Lijun Gong, Linchao Bao, Wang-meng Zuo, Chunhua Shen, Rynson Lau, and Ming-Hsuan Yang. Vital:Visual tracking via adversarial learning. pages 8990–8999, 06 2018.[140] Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi super-vised semantic segmentation using generative adversarial network. InProceedings of the IEEE International Conference on Computer Vision(ICCV), Oct 2017.[141] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever,and Ruslan Salakhutdinov. Dropout: A simple way to prevent neu-ral networks from overfitting. Journal of Machine Learning Research,15(56):1929–1958, 2014.[142] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In 2019IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), pages 5686–5696, 2019.[143] K.-K. Sung and T. Poggio. Example-based learning for view-basedhuman face detection. IEEE Transactions on Pattern Analysis andMachine Intelligence, 20(1):39–51, 1998.[144] Ariawan Suwendi and Jan P. Allebach. Nearest-neighbor and bilinearresampling factor estimation to detect blockiness or blurriness of animage. Journal of Electronic Imaging, 17(2):023005, 2008.[145] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Re-thinking the inception architecture for computer vision. In 2016 IEEEConference on Computer Vision and Pattern Recognition (CVPR),pages 2818–2826, Los Alamitos, CA, USA, jun 2016. IEEE ComputerSociety.99Bibliography[146] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A.Alemi. Inception-v4, inception-resnet and the impact of residual con-nections on learning. In Proceedings of the Thirty-First AAAI Confer-ence on Artificial Intelligence, AAAI’17, page 4278–4284. AAAI Press,2017.[147] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, ScottReed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, andAndrew Rabinovich. Going deeper with convolutions. In 2015 IEEEConference on Computer Vision and Pattern Recognition (CVPR),pages 1–9, 2015.[148] Ran Tao, Efstratios Gavves, and Arnold W. M. Smeulders. Siameseinstance search for tracking. In 2016 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), pages 1420–1429, 2016.[149] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convo-lutional one-stage object detection. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision (ICCV), October 2019.[150] Jakub Tomczak and Max Welling. Vae with a vampprior. In AmosStorkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics,volume 84 of Proceedings of Machine Learning Research, pages 1214–1223. PMLR, 09–11 Apr 2018.[151] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LlionJones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attentionis all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach,R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances inNeural Information Processing Systems, volume 30. Curran Associates,Inc., 2017.[152] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten,Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe.Mots: Multi-object tracking and segmentation. In 2019 IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR),pages 7934–7943, 2019.[153] Paul Voigtlaender, Jonathon Luiten, Philip H.S. Torr, and BastianLeibe. Siam r-cnn: Visual tracking by re-detection. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recogni-tion (CVPR), June 2020.100Bibliography[154] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadar-rama, and Kevin Murphy. Tracking emerges by colorizing videos. InProceedings of the European Conference on Computer Vision (ECCV),September 2018.[155] G. Wang, C. Luo, X. Sun, Z. Xiong, and W. Zeng. Tracking by instancedetection: A meta-learning approach. In 2020 IEEE/CVF Conferenceon Computer Vision and Pattern Recognition (CVPR), pages 6287–6296, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society.[156] N. Wang, Y. Song, C. Ma, W. Zhou, W. Liu, and H. Li. Unsuperviseddeep tracking. In 2019 IEEE/CVF Conference on Computer Visionand Pattern Recognition (CVPR), pages 1308–1317, Los Alamitos, CA,USA, jun 2019. IEEE Computer Society.[157] N. Wang, W. Zhou, J. Wang, and H. Li. Transformer meets tracker:Exploiting temporal context for robust visual tracking. In 2021IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), pages 1571–1580, Los Alamitos, CA, USA, jun 2021. IEEEComputer Society.[158] X. Wang, K. Zhao, R. Zhang, S. Ding, Y. Wang, and W. Shen. Con-trastmask: Contrastive learning to segment every thing. In 2022IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), pages 11594–11603, Los Alamitos, CA, USA, jun 2022. IEEEComputer Society.[159] Xiao Wang, Chenglong Li, Bin Luo, and Jin Tang. Sint++: Robustvisual tracking via adversarial positive instance generation. In 2018IEEE/CVF Conference on Computer Vision and Pattern Recognition,2018.[160] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, AbelGonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generat-ing images from limited data. In Proceedings of the European Confer-ence on Computer Vision (ECCV), September 2018.[161] Zheng’ao Wang, Zikun Zhou, Fanglin Chen, Jun Xu, Wenjie Pei, andGuangming Lu. Robust tracking via fully exploring background priorknowledge. IEEE Transactions on Circuits and Systems for VideoTechnology, pages 1–1, 2023.101Bibliography[162] Zhongdao Wang, Hengshuang Zhao, Ya-Li Li, Shengjin Wang, PhilipTorr, and Luca Bertinetto. Do different tracking tasks require differentappearance models? In M. Ranzato, A. Beygelzimer, Y. Dauphin,P.S. Liang, and J. Wortman Vaughan, editors, Advances in NeuralInformation Processing Systems, volume 34, pages 726–738. CurranAssociates, Inc., 2021.[163] X. Wei, Y. Bai, Y. Zheng, D. Shi, and Y. Gong. Autoregressive visualtracking. In 2023 IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), pages 9697–9706, Los Alamitos, CA,USA, jun 2023. IEEE Computer Society.[164] Q. Wu, T. Yang, Z. Liu, B. Wu, Y. Shan, and A. B. Chan. Drop-mae: Masked autoencoders with spatial-attention dropout for track-ing tasks. In 2023 IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), pages 14561–14571, Los Alamitos, CA,USA, jun 2023. IEEE Computer Society.[165] Qiangqiang Wu and Antoni B. Chan. Meta-graph adaptation for visualobject tracking. In 2021 IEEE International Conference on Multimediaand Expo (ICME), pages 1–6, 2021.[166] Qiangqiang Wu, Jia Wan, and Antoni B. Chan. Progressive unsuper-vised learning for visual object tracking. In 2021 IEEE/CVF Con-ference on Computer Vision and Pattern Recognition (CVPR), pages2992–3001, 2021.[167] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Online object tracking:A benchmark. In 2013 IEEE Conference on Computer Vision andPattern Recognition, pages 2411–2418, 2013.[168] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking bench-mark. IEEE Transactions on Pattern Analysis and Machine Intelli-gence, 37(9):1834–1848, 2015.[169] Ying Wu and Thomas S. Huang. Self-supervised learning for visualtracking and recognition of human hand. In Henry A. Kautz andBruce W. Porter, editors, Proceedings of the Seventeenth NationalConference on Artificial Intelligence and Twelfth Conference on onInnovative Applications of Artificial Intelligence, July 30 - August 3,2000, Austin, Texas, USA, pages 243–248. AAAI Press / The MITPress, 2000.102Bibliography[170] F. Xie, L. Chu, J. Li, Y. Lu, and C. Ma. Videotrack: Learning totrack objects via video transformer. In 2023 IEEE/CVF Conferenceon Computer Vision and Pattern Recognition (CVPR), pages 22826–22835, Los Alamitos, CA, USA, jun 2023. IEEE Computer Society.[171] F. Xie, C. Wang, G. Wang, Y. Cao, W. Yang, and W. Zeng.Correlation-aware deep tracking. In 2022 IEEE/CVF Conference onComputer Vision and Pattern Recognition (CVPR), pages 8741–8750,Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.[172] J. Xu and X. Wang. Rethinking self-supervised correspondence learn-ing: A video frame-level similarity perspective. In 2021 IEEE/CVFInternational Conference on Computer Vision (ICCV), pages 10055–10065, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society.[173] Liang Xu and Ruixin Niu. Semi-supervised visual tracking based onvariational siamese network. In Frederica Darema, Erik Blasch, SaiRavela, and Alex Aved, editors, Dynamic Data Driven ApplicationsSystems, pages 328–336, Cham, 2020. Springer International Publish-ing.[174] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, DingchengYue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang.Youtube-vos: Sequence-to-sequence video object segmentation. InProceedings of the European Conference on Computer Vision (ECCV),September 2018.[175] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu. Siamfc++:Towards robust and accurate visual tracking with target estimationguidelines. Proceedings of the AAAI Conference on Artificial Intelli-gence, 34:12549–12556, 04 2020.[176] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu.Learning spatio-temporal transformer for visual tracking. In Proceed-ings of the IEEE/CVF International Conference on Computer Vision(ICCV), pages 10448–10457, October 2021.[177] T. Yang and A. B. Chan. Visual tracking via dynamic memory net-works. IEEE Transactions on Pattern Analysis & Machine Intelli-gence, 43(01):360–374, jan 2021.103Bibliography[178] Tianyu Yang and Antoni B. Chan. Recurrent filter learning for visualtracking. In 2017 IEEE International Conference on Computer VisionWorkshops (ICCVW), pages 2010–2019, 2017.[179] Xingyi Yang and Xinchao Wang. Diffusion model as representationlearner. In Proceedings of the IEEE/CVF International Conference onComputer Vision (ICCV), pages 18938–18949, October 2023.[180] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, RuslanSalakhutdinov, and Quoc V. Le. XLNet: Generalized AutoregressivePretraining for Language Understanding. Curran Associates Inc., RedHook, NY, USA, 2019.[181] Bowen Yao, Jing Li, Shan Xue, Jia Wu, Huanmei Guan, Jun Chang,and Zhiquan Ding. Garat: Generative adversarial learning for robustand accurate tracking. Neural Networks, 148:206–218, 2022.[182] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen.Joint feature learning and relation modeling for tracking: A one-streamframework. In Shai Avidan, Gabriel Brostow, Moustapha Cisse´, Gio-vanni Maria Farinella, and Tal Hassner, editors, Computer Vision –ECCV 2022, pages 341–357, Cham, 2022. Springer Nature Switzer-land.[183] Alper Yilmaz, Omar Javed, and Mubarak Shah. Object tracking: Asurvey. ACM Comput. Surv., 38(4):13–es, dec 2006.[184] Yingjie Yin, De Xu, Xingang Wang, and Lei Zhang. Adversarial fea-ture sampling learning for efficient visual tracking. IEEE Transactionson Automation Science and Engineering, 17(2):847–857, 2020.[185] Yuan Ying, Pan Kejia, Tang Zhenyu, Huang Lin, and Jia Xinwei.Online object tracking based on generative adversarial networks. In2020 7th International Conference on Information Science and ControlEngineering (ICISCE), pages 934–938, 2020.[186] Weihao Yuan, Michael Wang, and Qifeng Chen. Self-supervised objecttracking with cycle-consistent siamese networks. pages 10351–10358,10 2020.[187] Chongzhen Zhang, Yang Tang, Chaoqiang Zhao, Qiyu Sun, ZhenchengYe, and Ju¨rgen Kurths. Multitask gans for semantic segmentation anddepth completion with cycle consistency. IEEE Transactions on NeuralNetworks and Learning Systems, 32(12):5404–5415, 2021.104Bibliography[188] Huanlong Zhang, Jiamei Liang, Jiapeng Zhang, Tianzhu Zhang, YingziLin, and Yanfeng Wang. Attention-driven memory network for onlinevisual tracking. IEEE Transactions on Neural Networks and LearningSystems, pages 1–14, 2023.[189] Huanlong Zhang, Jiapeng Zhang, Guohao Nie, Jilin Hu, andW.J. (Chris) Zhang. Residual memory inference network for regres-sion tracking with weighted gradient harmonized loss. InformationSciences, 597:105–124, 2022.[190] Hui Zhang, Mu Zhu, Jing Zhang, and Li Zhuo. Long-term visual objecttracking via continual learning. IEEE Access, 7:182548–182558, 2019.[191] Jiacheng Zhang and Yifeng Zhang. Siamese network for object track-ing with diffusion model. ICDIP ’23, New York, NY, USA, 2023.Association for Computing Machinery.[192] Yucheng Zhang, Tian Wang, Kexin Liu, Baochang Zhang, and LeiChen. Recent advances of single-object tracking methods: A briefsurvey. Neurocomputing, 455:1–11, 2021.[193] Z. Zhang, Y. Liu, X. Wang, B. Li, and W. Hu. Learn to match: Auto-matic matching network design for visual tracking. In 2021 IEEE/CVFInternational Conference on Computer Vision (ICCV), pages 13319–13328, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society.[194] Zhipeng Zhang and Houwen Peng. Deeper and wider siamese networksfor real-time visual tracking. In Proceedings of the IEEE/CVF con-ference on computer vision and pattern recognition, pages 4591–4600,2019.[195] Fei Zhao, Jinqiao Wang, Yi Wu, and Ming Tang. Adversarial deeptracking. IEEE Transactions on Circuits and Systems for Video Tech-nology, 29(7):1998–2011, 2019.[196] H. Zhao, D. Wang, and H. Lu. Representation learning for visualobject tracking by masked appearance transfer. In 2023 IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR),pages 18696–18705, Los Alamitos, CA, USA, jun 2023. IEEE Com-puter Society.[197] Xuechen Zhao, Yaoming Liu, and Guang Han. Cooperative use of re-current neural network and siamese region proposal network for robustvisual tracking. IEEE Access, 9:57704–57715, 2021.105Bibliography[198] Linyu Zheng, Yingying Chen, Ming Tang, Jinqiao Wang, and HanqingLu. Siamese deformable cross-correlation network for real-time visualtracking. Neurocomputing, 401:36–47, 2020.[199] Yaozong Zheng, Bineng Zhong, Qihua Liang, Zhiyi Mo, ShengpingZhang, and Xianxian Li. Odtrack: Online dense temporal token learn-ing for visual tracking. In AAAI, 2024.[200] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learningdeep features for discriminative localization. In 2016 IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), pages 2921–2929, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.[201] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual ground-ing and tracking with natural language specification. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recogni-tion (CVPR), pages 23151–23160, June 2023.[202] Tianfei Zhou, Fatih Porikli, David J. Crandall, Luc Van Gool, andWenguan Wang. A survey on deep learning technique for video seg-mentation. IEEE Transactions on Pattern Analysis and Machine In-telligence, 45(6):7099–7122, 2023.[203] Jiajun Zhu, Chao Ma, Shuai Jia, and Shugong Xu. Contrastive cycleconsistency learning for unsupervised visual tracking. In Huimin Ma,Liang Wang, Changshui Zhang, Fei Wu, Tieniu Tan, Yaonan Wang,Jianhuang Lai, and Yao Zhao, editors, Pattern Recognition and Com-puter Vision, pages 564–576, Cham, 2021. Springer International Pub-lishing.[204] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu,and Xinggang Wang. Vision mamba: Efficient visual representationlearning with bidirectional state space model, 2024.[205] Wenjun Zhu, Zuyi Wang, Li Xu, and Jun Meng. Exploiting tempo-ral coherence for self-supervised visual tracking by using vision trans-former. Knowledge-Based Systems, 251:109318, 2022.[206] Wenjun Zhu, Li Xu, and Jun Meng. Consistency-based self-supervised visual tracking by using query-communication transformer.Knowledge-Based Systems, 278:110849, 2023.106[207] X. Zhu, H. Hu, S. Lin, and J. Dai. Deformable convnets v2: Moredeformable, better results. In 2019 IEEE/CVF Conference on Com-puter Vision and Pattern Recognition (CVPR), pages 9300–9308, LosAlamitos, CA, USA, jun 2019. IEEE Computer Society.[208] C. Lawrence Zitnick and Piotr Dolla´r. Edge boxes: Locating objectproposals from edges. In David Fleet, Tomas Pajdla, Bernt Schiele,and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages391–405, Cham, 2014. Springer International Publishing.107Appendix AImplementation andReproducibility DetailsAll experiments in this work use PyTorch 1.13 and Python 3.9. Training,testing, and inference on all studies conducted were performed using twoNVIDIA A6000 GPUs.108
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Shehata, Mohamed S.
Abdelpakey, Mohamed H.
Beyond traditional single object tracking
Text