Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Gong, Wenyi
Master of Applied Science - MASc
University of British Columbia
Many modern Vision Transformer (ViT) backbones adopt spatial architectural designs, such as window attention from Swin and ViTDet, and 2D positional embeddings which includes decomposed relative positional embeddings in SAM and RoPE in DINOv3. Such architectures impose new challenges on token reduction, as the vast majority of existing methods fail to preserve the spatial structure these architectures depend on. In this thesis, we introduce a simple yet effective token merging method that maintains spatial integrity, enabling seamless compatibility with spatial architectures. We reconcile two seemingly conflicting requirements: (1) exploiting the uneven information distribution across the spatial layout, and (2) preserving the spatial structure post-merging. To enable effective token reduction for spatial architectures, we propose three key innovations: - a 2D reduction strategy to enforce structured token layouts, - a spatial-aware merging algorithm that maintains relative token positions, and - a novel max-magnitude-per-dimension token representation that preserves salient features. Our method demonstrates strong performance both off-the-shelf and with fine-tuning, achieving state-of-the-art results on spatial and non-spatial architectures across various vision tasks. Specifically, we achieve 1.25× speedup on SAM-H with only 0.7% mIOU drop evaluated on COCO off-the-shelf, and 1.15× speedup on DeiT-B with no top-1 accuracy drop on ImageNet within just one epoch of fine-tuning.
CubistMerge: Spatial-Preserving Token Merging ForDiverse ViT BackbonesbyWenyi Gonga thesis submitted in partial fulfillmentof the requirements for the degree ofMaster of Applied Scienceinthe faculty of graduate and postdoctoral studies(Electrical and Computer Engineering)The University of British Columbia(Vancouver)October 2025© Wenyi Gong, 2025The following individuals certify that they have read, and recommend to the Facultyof Graduate and Postdoctoral Studies for acceptance, the thesis entitled:CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Back-bonessubmitted by Wenyi Gong in partial fulfillment of the requirements for the degreeof Master of Applied Science in Electrical and Computer Engineering.Examining Committee:Mieszko Lis, Associate Professor, Electrical and Computer Engineering, UBCSupervisorRenjie Liao, Assistant Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberPrashant Nair, Assistant Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberiiAbstractMany modern Vision Transformer (ViT) backbones adopt spatial architecturaldesigns, such as window attention from Swin and ViTDet, and 2D positionalembeddings which includes decomposed relative positional embeddings in SAMand RoPE in DINOv3. Such architectures impose new challenges on token reduction,as the vast majority of existing methods fail to preserve the spatial structure thesearchitectures depend on.In this thesis, we introduce a simple yet effective token merging method thatmaintains spatial integrity, enabling seamless compatibility with spatial architectures.We reconcile two seemingly conflicting requirements: (1) exploiting the uneveninformation distribution across the spatial layout, and (2) preserving the spatialstructure post-merging.To enable effective token reduction for spatial architectures, we propose threekey innovations:• a 2D reduction strategy to enforce structured token layouts,• a spatial-aware merging algorithm that maintains relative token positions, and• a novel max-magnitude-per-dimension token representation that preservessalient features.Our method demonstrates strong performance both off-the-shelf and with fine-tuning, achieving state-of-the-art results on spatial and non-spatial architecturesacross various vision tasks. Specifically, we achieve 1.25× speedup on SAM-Hwith only 0.7% mIOU drop evaluated on COCO off-the-shelf, and 1.15× speedupon DeiT-B with no top-1 accuracy drop on ImageNet within just one epoch offine-tuning.iiiLay SummaryDespite being powerful AI models for image processing, Vision Transformersrequire enormous computational resources, making them expensive and limitingtheir practical deployment.This research develops CubistMerge, a technique that accelerates these modelswhile maintaining accuracy. The key challenge is that modern vision models rely onspatial structure, which is the arrangement and relationships between image pieces,to understand images properly. Previous acceleration methods either destroyed thisspatial structure or failed to reduce redundant information effectively.CubistMerge intelligently combines similar image pieces (tokens) while preserv-ing spatial relationships. It uses a two-step 2D reduction process, an algorithm thatmerges image patches while maintaining relative positions, and a novel representationmethod that preserves important features.The technique achieved notable speedups with minimal accuracy loss: 1.25×faster image segmentation with only 0.7% accuracy loss, and 1.15× faster classifica-tion with no accuracy loss, enabling practical deployment of advanced AI visionsystems.ivPrefaceThis research work is original and was conducted independently by the author WenyiGong under the supervision of Prof. Mieszko Lis. A version of this work was postedto arXiv.Generative AI (Claude Sonnet 4.0) was used to polish writing in this thesis.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Token Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . 21.1.1 Challenges from Modern ViT Architectural Trends . . . . 21.1.2 Limitation of Existing Spatial-Aware Token Reduction . . 51.2 Our Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 51.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . 6vi2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 Basic Transformer Architecture . . . . . . . . . . . . . . . . . . . 72.2 Vanilla Vision Transformers . . . . . . . . . . . . . . . . . . . . 92.3 Adapting Vision Transformers to Different Tasks . . . . . . . . . 102.3.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . 112.3.2 Dense Prediction . . . . . . . . . . . . . . . . . . . . . . 112.4 Spatial Architectures for Vision Transformers . . . . . . . . . . . 122.4.1 Window Attention . . . . . . . . . . . . . . . . . . . . . 122.4.2 2D Positional Embeddings . . . . . . . . . . . . . . . . . 143 Prior Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.1 Token Pruning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.2 Training-based Token Reduction . . . . . . . . . . . . . . . . . . 173.3 Graph-based Token Merging . . . . . . . . . . . . . . . . . . . . 183.4 Clustering-based Token Merging . . . . . . . . . . . . . . . . . . 193.5 Task-specific Token Reduction . . . . . . . . . . . . . . . . . . . 203.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 CubistMerge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.1 2D Reduction Strategy . . . . . . . . . . . . . . . . . . . . . . . 224.1.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 234.1.2 Window Attention Adoption . . . . . . . . . . . . . . . . 234.2 Spatial-Aware Token Matching . . . . . . . . . . . . . . . . . . . 244.2.1 Graph Construction . . . . . . . . . . . . . . . . . . . . . 244.2.2 Edge Selection . . . . . . . . . . . . . . . . . . . . . . . 264.2.3 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . 274.3 Max-Magnitude-Per-Dimension Token Representation . . . . . . 274.3.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 284.3.2 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . 295 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 325.1.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.1.2 Tasks and Metrics . . . . . . . . . . . . . . . . . . . . . . 33vii5.1.3 Configurations . . . . . . . . . . . . . . . . . . . . . . . 345.2 Experiments on Spatial Architectures . . . . . . . . . . . . . . . 355.2.1 Prior Work Comparison . . . . . . . . . . . . . . . . . . 355.2.2 Model Sweep . . . . . . . . . . . . . . . . . . . . . . . . 365.3 Experiments on Non-Spatial Architectures . . . . . . . . . . . . . 425.3.1 Prior Work Comparison . . . . . . . . . . . . . . . . . . 425.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.4 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53viiiList of TablesTable 4.1 Computational complexity of graph construction algorithms fortoken matching. . . . . . . . . . . . . . . . . . . . . . . . . . 25Table 4.2 Ablation study comparing graph construction and edge selectionmethods for CubistMerge. Results show ΔmIOU and speeduprelative to the baseline model without token reduction. Ourchosen design is highlighted in gray. . . . . . . . . . . . . . . 28Table 4.3 Ablation study comparing token representation methods forCubistMerge. Results show ΔmIOU and speedup relative to thebaseline model without token reduction. Our chosen design ishighlighted in gray. . . . . . . . . . . . . . . . . . . . . . . . 29Table 5.1 Image classification results on spatial architectures, with 𝑟ℎ =𝑟𝑤 = 1. Fine-tuned results (highlighted in cyan) are included forMViTv2-B within 3 epochs of training. . . . . . . . . . . . . . 38Table 5.2 Layer sensitivity analysis of object detection on DINOv3-ViT7Bwith 𝑟ℎ = 𝑟𝑤 = 0.1 . . . . . . . . . . . . . . . . . . . . . . . . 40Table 5.3 Segmentation results on Segmenter(ViT-S), comparing CuMeagainst ALGM on Cityscapes and Pascal Context . . . . . . . . 42Table 5.4 Fine-tuned results on DeiT-B, within 5 epochs. . . . . . . . . . 43ixList of FiguresFigure 1.1 Comparison against existing token reduction works. Most ap-proaches disrupt spatial layouts (exemplified by ToMe [3]) whileExpedite [27] preserves structure but fails to exploit informa-tion density unevenness across regions. Our method maintainsspatial coherence while selectively merging redundant tokens. 4Figure 2.1 Transformer encoder [49] with multi-head self-attention (MHSA),feed-forward network (FFN) and positional encoding. . . . . . 8Figure 2.2 Vision Transformer architecture [11] for image classification.An input image is divided into fixed-size patches with a standardTransformer encoder. A learnable [class] token aggregates globalinformation for final classification through an MLP head. . . . 10Figure 2.3 Vision Transformer for dense prediction tasks. DETR [5] servesas an example showing how the encoder produces dense repre-sentations while the decoder uses queries to predict outputs. . 12Figure 4.1 2D token reduction with spatial-aware merging: (1) original14×14 tokens, (2) select horizontal tokens to merge, (3) mergehorizontally to 14×12 tokens, (4) select vertical tokens to merge,(5) merge vertically to 12×12 tokens. . . . . . . . . . . . . . . 24Figure 4.2 Path graph illustration showing tokens arranged in a linear se-quence, where each token connects only to its adjacent neighbors,forming the basis for spatial-aware token merging. . . . . . . 25xFigure 4.3 Illustration of edge selection algorithms. Arrows on selectededges (in orange) indicate the direction of token merging, point-ing from the source token to the destination token. The numberson edges represent execution order of merging required by de-pendencies. (a) Path graph with naive edge selection, (b) Pathgraph with naive edge selection, optimized with reduction treeto 𝑂 (log 𝑁), (c) Path graph with bipartite edge selection toeliminate dependencies by ensuring each source token (in red)can only merge to one destination (in blue), (d) Bipartite SoftMatching [3]. . . . . . . . . . . . . . . . . . . . . . . . . . . 30Figure 5.1 Image classification results on spatial architectures, varying𝑟ℎ = 𝑟𝑤 = 1, 2, 3 with 𝑙 = 10 on MViTv2-B, 𝑙 = 20 onDINOv3-ViT7B and MViTv2-L. . . . . . . . . . . . . . . . . 37Figure 5.2 Object detection results. For DINOv3-ViT7B, we vary 𝑟ℎ =𝑟𝑤 = 0.1, 0.15, 0.2 at 𝑙 = 20. For ViTDet, we vary 𝑟ℎ = 𝑟𝑤 =4, 8, 12 at 𝑙 = 0 for CuMe and 𝑙 = 2 for Expedite. . . . . . . . 39Figure 5.3 Instance segmentation results on SAM. . . . . . . . . . . . . 45Figure 5.4 Panoptic segmentation performance on Mask2Former(Swin-L),varying 𝑟ℎ = 𝑟𝑤 = 1, 2, 3 . . . . . . . . . . . . . . . . . . . . 46Figure 5.5 Instance segmentation performance on Mask2Former(Swin-L),varying 𝑟ℎ = 𝑟𝑤 = 1, 2, 3 . . . . . . . . . . . . . . . . . . . . 46Figure 5.6 Instance segmentation results on SAM2-L evaluated on COCOand ADE20K datasets, varying 𝑟ℎ = 𝑟𝑤 = 4, 8, 12 with CuMeand ToMe at 𝑙 = 9, and Expedite at 𝑙 = 15. . . . . . . . . . . 47Figure 5.7 Image classification results on DeiT-B compared against priortoken reduction methods. . . . . . . . . . . . . . . . . . . . . 48xiList of Symbols𝐾 Key𝑙 Layer to apply token merging𝑄 Query𝑟ℎ Horizontal token reduction rate𝑟𝑤 Vertical token reduction rate𝑉 ValuexiiList of Abbreviations2D 2-DimensionalAI Artificial IntelligenceAP Average PrecisionCOCO Common Objects in ContextCNN Convolutional Neural NetworkCuMe CubistMergeDeiT Data-efficient Image Transformers [47]DETR Detection Transformer [5]DINO Self-distillation with no labelsFFN Feed-Forward NetworksFLOPs Floating Point OperationsFPS Frames Per SecondGPU Graphics Processing UnitMHSA Multi-Head Self-AttentionMLP Multi-Layer PerceptronMViT Multiscale Vision TransformersNLP Natural Language ProcessingPE Positional EncodingReLU Rectified Linear UnitRoPE Rotary Positional EmbeddingsSAM Segment Anything Model [19]ToMe Token Merging [3]ViT Vision TransformerViTDet Vision Transformer for Object Detection [25]xiiiAcknowledgmentsI would first like to thank my advisor, Professor Mieszko Lis. His guidance hasshaped not only this thesis, but my professional path and personal growth. He hasshown patience and encouragement that sustained me through the most difficultmoments, both in the work behind these pages and in life beyond them. He believedin my potential when I struggled to see it myself. I am grateful for his unwaveringsupport in ways that extend far beyond what these pages can capture.I am grateful to my committee members, Professor Prashant Nair and ProfessorRenjie Liao, for their support. Professor Nair is one of the kindest people I havemet at UBC, and I am fortunate to have crossed paths with him. I am grateful toProfessor Liao for bringing his machine learning expertise to my committee.I am grateful to my labmates for creating an environment that was both intel-lectually stimulating and warmly supportive. They are exceptional people in everysense. I especially thank Dingqing for his technical assistance on this work.On a more personal note, I owe gratitude to my anonymous victim. He has beena constant source of support. He kept me pampered and well-fed, and took wonderfulcare of Mushu. His patience and support made it all more bearable.Last but not least, I must thank the furry companions who shaped this journey.Whisky, who’s as smart as she’s adorable, was my greatest motivation to come tocampus. She will forever hold a special place in my heart. And Mushu, my greatestgift in life, the little creature who made me understand what unconditional love trulymeans, and whose presence brought immeasurable joy to my life.xivTo Whisky and MushuxvChapter 1IntroductionTransformers have emerged as the foundational architecture in modern machinelearning, achieving state-of-the-art performance across diverse modalities andapplications. These models operate by processing sequences of discrete elementscalled tokens through self-attention mechanisms, where each token can attend toall other tokens in the sequence to compute contextualized representations [49].Transformers quickly became the leading architecture for language tasks [4, 10],where tokens represent words or subwords, by enabling parallel computation andmodeling long-range dependencies effectively.This architecture was then adapted to computer vision through Vision Transform-ers (ViTs), which divide images into patches and treat each patch as a token [11]. Thetransformer architecture enables novel self-supervised pre-training strategies such asmasked autoencoders [13] and knowledge distillation [6, 38, 45]. Due to these pre-training capabilities, combined with their architectural simplicity and ability to scaleeffectively with data and model size, Vision Transformers have become the leadingarchitecture across various vision applications including classification [11, 13, 47],object detection [5, 25, 43], semantic segmentation [8, 19, 46, 54, 55], video seg-mentation [12, 42], as well as multimodal applications that combine vision andlanguage [40].However, the substantial memory and computational demands of Vision Trans-formers pose major challenges, especially with the growing model size of recentlarge-scale models [45]. The quadratic complexity of self-attention with respect to1the number of input tokens, combined with increasingly large parameter counts,creates significant computational overhead. This limits the deployment of ViTs inresource-constrained environments and real-world applications where efficiency iscritical.Numerous approaches have emerged to address these computational bottlenecks,including efficient architectures that redesign fundamental components, such asmore computationally efficient attention mechanisms [2, 17, 31] and attention headreduction [34, 36, 50].Among these various efficiency strategies, token reduction [20, 28, 41] standsout as especially attractive since it requires no architectural modifications and canwork without retraining [3, 27]. This practical advantage is important given theenormous computational costs and limited data availability that make retraininglarge-scale foundation models prohibitively expensive.1.1 Token ReductionToken reduction exploits a unique characteristic of the transformer architecture: itsability to process variable-length input sequences without structural constraints,enabling dynamic token reduction during processing. This input-adaptive approachis particularly suitable for vision tasks where visual content exhibits varying levelsof complexity across different spatial regions. These methods have demonstratedsuccess in reducing computational costs while maintaining reasonable performanceacross both inference [20, 28, 41] and training phases [3, 27].Despite their proven effectiveness on vanilla ViT backbone [11], token reductionmethods face new challenges with the recent evolution of Vision Transformers.1.1.1 Challenges from Modern ViT Architectural TrendsLarge-Scale ModelsModern vision transformers face significant scalability challenges due to theirincreasing model size and pretraining requirements. Training these large-scalemodels demands substantial computational resources, exemplified by DINOv3ViT-7B/16 which requires 256 H100 GPUs for pretraining. Additionally, these2models are inherently data-hungry, as demonstrated by MAE [13] and DINOv3 [45],requiring massive datasets to achieve their performance gains through scaling upto 7B parameters. Moreover, many of these foundation models, including DINOv3ViT-7B/16, are trained on private datasets that are not publicly accessible, furtherlimiting retraining feasibility.Many token reduction techniques [20, 21, 28–30, 32, 33, 41, 52] require fullretraining, which becomes impractical for such large-scale models due to enormouscomputational demands and limited data availability. The extensive pretrainingefforts invested in these foundation models make retraining-based token reductionapproaches prohibitively expensive, creating a pressing need for training-freeor lightweight fine-tuning solutions that can leverage these powerful pretrainedrepresentations with minimal modification.Spatial ArchitecturesRecent Vision Transformers have increasingly adopted spatial-aware architecturaldesigns that explicitly leverage 2D spatial relationships for improved efficiencyand performance. These spatial architectures have proven effective and are nowemployed by many influential models [19, 25, 31, 45], representing a significantevolution from the vanilla ViT [11].One prominent component is window attention [25, 31], which restricts self-attention to local spatial windows for linear complexity by partitioning the inputfeature map into non-overlapping windows. This approach reduces computationalcomplexity from quadratic to linear with respect to input size while maintaining theability to model local spatial relationships. Window attention requires consistenttoken counts across all windows for efficient batched computation.Another key spatial component is 2D positional embeddings [14, 26] that encodespatial relationships along height and width dimensions rather than treating tokensas a flat sequence. Unlike positional embeddings in vanilla ViT [11] which areapplied before propagating through the layers, 2D positional embeddings are appliedat every attention layer and depend on structured 2D token arrangements to computespatial relationships correctly.However, the vast majority of existing token reduction methods face fundamental3(a) Original(14×14 tokens)(b) ToMe(12×12 tokens)(c) Expedite(12×12 tokens)(d) Ours(12×12 tokens)Figure 1.1: Comparison against existing token reduction works. Most ap-proaches disrupt spatial layouts (exemplified by ToMe [3]) while Ex-pedite [27] preserves structure but fails to exploit information densityunevenness across regions. Our method maintains spatial coherence whileselectively merging redundant tokens.incompatibilities with these spatial architectures. Most existing techniques [3, 7,18, 20, 22, 28, 32, 37, 41, 48] produce unstructured token layouts that break thespatial coherence required by these designs, as visualized in Figure 1.1(b). Theseunstructured reduction strategies disrupt both spatial architectural components:they produce uneven token distributions across windows that compromise thebatched computation efficiency of window attention, while simultaneously breakingthe regular 2D spatial arrangement that 2D positional embeddings depend on forcomputing spatial relationships. This results in significant performance drops onmodels with 2D positional embeddings (see Section 5.2). While one could conceiveof workarounds to force compatibility, such as padding windows to equalize tokencounts after reduction or dynamically recomputing 2D positional embeddings ateach layer to match the transformed token layout, these ad hoc approaches wouldintroduce substantial computational overhead that negates the efficiency gains thattoken reduction aims to achieve.41.1.2 Limitation of Existing Spatial-Aware Token ReductionSpatial-Aware but not Spatial-PreservingDespite some works adopting spatial-aware strategies [16, 37], these approachesfocus on restricting token merging among spatially near tokens, but fail to addressthe fundamental issue: the resulting token distribution is irregular and cannot bereshaped into structured and spatially-coherent layouts, making them incompatiblewith spatial architectures.ExpediteAs the only existing token reduction method that maintains spatial integrity, Expe-dite [27] offers a training-free solution for large-scale models. However, Expeditesuffers from a critical limitation: it modifies every token through clustering, eventhe semantically distinct tokens.Expedite preserves spatial structure by using pooling across the structuredfeature map to initialize cluster centroids. However, this approach conflicts with afundamental principle that token merging techniques typically exploit: informationdensity is uneven across feature maps, meaning representative tokens should not beuniformly distributed across spatial regions (see 1.1(c)). By forcing distinct tokensinto clusters with dissimilar tokens, this uniform initialization approach causesinformation loss, thereby resulting in significant performance degradation especiallywhen applied to early layers. This creates a remaining gap for methods that preservespatial structure without compromising semantic information.1.2 Our ContributionsTo address the above challenges and learning from the weaknesses of existing spatial-preserving approach [27], we reconcile two seemingly conflicting requirements:(1) preserving spatial structure of merged tokens, while (2) exploiting the uneveninformation distribution across the spatial layout.We introduce CubistMerge, a spatial-aware token merging algorithm that selec-tively joins redundant tokens using an information-preserving representation, whileleaving distinct tokens untouched. Our approach maintains structured spatial token5layouts in the resulting tokens after merging, enabling compatibility with spatialarchitectures. CubistMerge can operate as an off-the-shelf solution and also showsstrong fine-tuning performance with a small number of epochs.The key contributions we make in this thesis are:• a 2D token reduction strategy that maintains consistent token counts per rowand column;• a locality-aware graph construction for identifying redundant tokens; and• a max-magnitude-element token representation that preserves salient featureswithout requiring layer-wise attention rescaling.We demonstrate generalizability through comprehensive evaluation acrossseveral influential ViT backbones on diverse tasks including classification, detection,and segmentation. On spatial architectures, we consistently outperform Expedite [27]across every task and model. To compare against existing methods more broadly,we also conduct experiments on vanilla ViT backbones, achieving state-of-the-artresults, both off-the-shelf and with fine-tuning, on DeiT-B compared against 5 othertoken reduction techniques. Notably, we achieve no accuracy loss on ImageNetat 1.15× speedup within just one epoch of fine-tuning. Even compared againstspecialized methods like ALGM [37] which targets segmentation, we achieve similarperformance while maintaining broader applicability.1.3 Thesis OrganizationThis thesis is structured as follows. Chapter 2 introduces Vision Transformer foun-dations with focus on spatial architectural components that pose compatibilitychallenges for existing efficiency techniques. Chapter 3 surveys existing token reduc-tion approaches and their limitations with modern spatial architectures. Chapter 4presents CubistMerge along with comprehensive ablation experiments validatingeach design choice. Chapter 5 provides empirical validation across diverse vi-sion tasks, evaluating spatial architectures extensively while including non-spatialarchitectures for broader comparison against existing methods.6Chapter 2BackgroundThis chapter provides the foundational background necessary for understandingVision Transformer architectures and their evolution. We begin by examining the basictransformer architecture that forms the foundation of all vision transformers. We thenexplore vanilla Vision Transformers [11], which serve as the foundational architecturefor all subsequent ViT variants, and how Vision Transformers are adapted todifferent vision tasks, highlighting the fundamental differences between classificationand dense prediction paradigms in terms of their architectural requirements andoutput utilization. Finally, we discuss spatial architectures, which represent a recentarchitectural trend in Vision Transformers that explicitly leverages 2D spatialstructure to improve both efficiency and performance, and explain how these differfrom vanilla ViT.2.1 Basic Transformer ArchitectureThe Transformer [49] processes sequences of tokens through self-attention mecha-nisms. Originally designed for natural language processing, each token can attend toall other tokens in the sequence to compute contextualized representations.We focus on the encoder architecture since this forms the foundation for VisionTransformers. We explain three key components of the transformer encoder: multi-head self-attention and feed-forward networks form the core of each transformerblock, while positional encoding provides essential sequence order information.7Figure 2.1: Transformer encoder [49] with multi-head self-attention (MHSA),feed-forward network (FFN) and positional encoding.Multi-Head Self-AttentionMulti-head self-attention enables each token to directly attend to all other tokens inthe sequence, capturing diverse relationship types across multiple parallel attentionheads.The attention mechanism computes attention weights between all token pairs inthe sequence. The core scaled dot-product attention function takes queries, keys,and values as input:Attention(𝑄, 𝐾,𝑉) = softmax(𝑄𝐾𝑇√𝑑𝑘)𝑉 (2.1)The scaling factor 1√𝑑𝑘prevents dot products from becoming too large.Multi-head attention applies this attention function multiple times in parallelwith different learned projections, then concatenates and projects the results. Thisallows the model to jointly attend to information from different representationsubspaces at different positions.8Feed-Forward NetworksFeed-forward networks apply the same learned non-linear transformation indepen-dently to each token position, providing increased representational capacity afterattention has mixed information across the sequence.Each token is processed independently through a position-wise feed-forwardnetwork consisting of two linear transformations with ReLU activation:FFN(𝑥) = max(0, 𝑥𝑊1 + 𝑏1)𝑊2 + 𝑏2 (2.2)While the linear transformations are the same across different positions, theyuse different parameters from layer to layer.Positional EncodingPositional encodings inject sequence order information into token representations,compensating for the transformer’s inherent lack of positional awareness due tothe permutation-invariant nature of self-attention. The original Transformer usessinusoidal positional encodings:PE (pos,2𝑖) = sin(pos/100002𝑖/𝑑model) (2.3)PE (pos,2𝑖+1) = cos(pos/100002𝑖/𝑑model) (2.4)where pos is the position and 𝑖 is the dimension of the positional encoding. Thissinusoidal approach allows the model to learn relative positional relationships andpotentially extrapolate to longer sequences than seen during training.2.2 Vanilla Vision TransformersThe original Vision Transformer (ViT) [11] follows the original Transformer [49]as closely as possible, enabling scalable NLP transformer architectures and theirefficient implementations to be used almost out of the box for vision tasks.To handle 2D images, ViT reshapes an input image 𝑥 ∈ R𝐻×𝑊×𝐶 into a sequenceof flattened 2D patches 𝑥𝑝 ∈ R𝑁×(𝑃2 ·𝐶 ) , where (𝐻,𝑊) is the resolution of the9Figure 2.2: Vision Transformer architecture [11] for image classification. Aninput image is divided into fixed-size patches with a standard Transformerencoder. A learnable [class] token aggregates global information for finalclassification through an MLP head.original image, 𝐶 is the number of channels, (𝑃, 𝑃) is the resolution of each imagepatch, and 𝑁 = 𝐻𝑊/𝑃2 is the resulting number of patches, which also serves as theeffective input sequence length for the vision transformer. Each patch is flattened andmapped to 𝐷 dimensions with a trainable linear projection, creating patch tokensthat serve as the core representational elements operated on by the transformer.The vision transformer uses constant latent vector size 𝐷 through all of its layers,ensuring each token maintains this dimensionality throughout processing.Position embeddings [11] are added to the patch embeddings to retain positionalinformation. Like the original Transformer, ViT uses 1-dimensional positionalembeddings where patches are treated as a sequence in raster order. However,unlike the fixed sinusoidal encodings used in the original Transformer, ViT adoptslearnable positional embeddings that are optimized during training. These parametricembeddings are added once at the beginning, immediately after the linear projectionof patches, before the sequence is fed to all transformer encoder layers.2.3 Adapting Vision Transformers to Different TasksWhile the foundational ViT architecture provides a general framework for process-ing visual information, its adoption across different vision tasks requires distinctapproaches due to the varied nature of these tasks. Classification tasks seek to assign10a single label to an entire image and require only global image understanding. Incontrast, dense prediction tasks such as object detection and segmentation mustgenerate spatially-structured outputs that correspond to specific regions or pixels inthe input image. These fundamental differences in task requirements lead to differentarchitectural adaptations and output utilization strategies.2.3.1 ClassificationThe original ViT, designed for image classification (Figure 2.2), prepends a learnable[𝑐𝑙𝑠] token to the patch sequence. This special token serves as a class representationthat aggregates information from all patch tokens through the self-attention mecha-nism. After processing through all transformer layers, this [𝑐𝑙𝑠] token contains theaccumulated information needed for classification, and is then fed to a classificationhead to produce the final prediction. This classification paradigm, where only the[𝑐𝑙𝑠] token output is required for the final prediction, has been adopted by subsequentclassification-focused Vision Transformers such as DeiT [47] and ViT-MAE [13].2.3.2 Dense PredictionFor dense prediction tasks such as object detection and segmentation, VisionTransformers adopt the original vanilla ViT architecture as the encoder component,which is also often referred to as the backbone. Unlike classification, these tasksrequire an encoder-decoder architecture where the encoder must output dense tokenrepresentations rather than just the [𝑐𝑙𝑠] token.The ViT encoder processes image patches and produces a sequence of patch to-kens that preserve spatial correspondence with the original image regions. This denserepresentation is essential because dense prediction tasks require location-specificinformation to generate spatially-aligned outputs. The decoder then takes these densepatch representations along with task-specific learned queries or embeddings toproduce the final predictions.In object detection models like DETR [5] (Figure 2.3), the decoder uses learnedobject queries that attend to the dense encoder output to predict bounding boxes andclasses. For segmentation tasks, approaches like Segmenter [46] employ decodersthat process learned class embeddings together with the dense patch representations11Figure 2.3: Vision Transformer for dense prediction tasks. DETR [5] servesas an example showing how the encoder produces dense representationswhile the decoder uses queries to predict outputs.to generate segmentation masks. The key distinction from classification is that allpatch token information from the encoder must be preserved and made available tothe decoder for location-aware reasoning, rather than being aggregated into a singleextracted representation.2.4 Spatial Architectures for Vision TransformersWhile vanilla ViTs treat image patch tokens as a flat sequence of tokens, spatialarchitectures explicitly leverage the inherent 2D structure of visual data to improveefficiency and performance.Spatial architectures introduce two key components that fundamentally changehow transformers process visual information: window-based attention mechanismsthat constrain computation to local neighborhoods, and 2D positional embeddingsthat encode spatial relationships more effectively than flat positional encodings.2.4.1 Window AttentionWindow attention mechanisms partition the input feature map into non-overlappinglocal windows and restrict self-attention computation within each window. Byrestricting attention computation to local windows of fixed size 𝑀×𝑀 , this approachreduces attention complexity from quadratic 𝑂((𝐻𝑊)2) to linear 𝑂 (𝐻𝑊) with12respect to the number of tokens, making it scalable for high-resolution images. Whilethis local attention design limits direct global attention, modern architectures enableglobal context modeling through mechanisms such as cross-window connections orstrategically placed global attention layers.Shifted Window AttentionThe Swin Transformer [31] proposes a hierarchical transformer architecture withshifted windows. The key innovation lies in limiting self-attention computation tonon-overlapping local windows while enabling cross-window connections through ashifted windowing scheme.Swin alternates between two partitioning configurations in consecutive trans-former blocks to enable information exchange across windows. Regular windowpartitioning is used in the first module, evenly partitioning the feature map intonon-overlapping windows. The next module adopts a windowing configuration thatis shifted from the preceding layer by displacing the windows by (⌊𝑀/2⌋, ⌊𝑀/2⌋)pixels from the regularly partitioned windows, where 𝑀 is the window size.This shifted windowing approach introduces connections between neighboringnon-overlapping windows from the previous layer while maintaining efficientcomputation. The design achieves linear computational complexity with respect toimage size and has proven effective across vision tasks including image classification,object detection, and semantic segmentation.Window-Global AttentionSeveral architectures have adopted simpler window attention schemes that maintainfixed window partitions without shifting, instead relying on occasional globalattention layers to enable cross-window communication.MViTv2 [26] introduces a hybrid window attention design that computes localattention within windows in most layers, but applies global attention in the last blocksof the final stages that feed into downstream detection heads. This approach ensuresthat dense feature representations provided to downstream components containglobal information while maintaining computational efficiency. Unlike Swin’s shiftedwindow approach, this hybrid design achieves cross-window connectivity through13strategic placement of global attention layers rather than window shifting, anddemonstrates superior performance on both image classification and object detectiontasks.ViTDet [25] further demonstrates that fixed window attention suffices for denseprediction tasks when combined with sparse cross-window propagation blocks.Rather than concentrating global attention at the final stages, ViTDet distributes cross-window propagation layers throughout the network at regular intervals. The ViTDetbackbone was subsequently adopted in Segment Anything [19], demonstrating itseffectiveness for large-scale vision applications.2.4.2 2D Positional EmbeddingsSpatial architectures employ specialized positional embedding schemes that explicitlyencode 2D spatial relationships rather than treating image patches as a flat sequence.Unlike vanilla ViT, which adds absolute positional embeddings only once before thefirst transformer layer, spatial architectures inject 2D positional embeddings at everyattention layer to provide consistent spatial context throughout the network. Thislayer-wise application enables the model to maintain and refine spatial understandingat different hierarchical levels.Decomposed Relative Positional EmbeddingsDecomposed relative positional embeddings, introduced in MViTv2 [26], encodespatial relationships based on relative distances between tokens rather than absolutepositions. This shift-invariant approach ensures that the interaction modeling betweentwo patches depends only on their relative positions, not their absolute locations inthe image.Rather than computing pairwise relative positions directly, which would scaleas 𝑂 (𝐻 ×𝑊), this approach decomposes 2D relative positions into separate heightand width components. This decomposition reduces parameter complexity from𝑂 (𝐻 ×𝑊) to 𝑂 (𝐻 +𝑊), providing significant computational savings for high-resolution feature maps.These positional embeddings are learned parameters that are specific to eachattention layer, allowing individual layers to develop their own spatial reasoning14patterns adapted to different hierarchical levels of feature processing. The layer-specific embeddings are added to attention scores before the softmax operationand are particularly effective for dense prediction tasks requiring fine-grainedspatial understanding. ViTDet [25] and Segment Anything [19] demonstrate theeffectiveness of this approach, achieving strong performance on detection andsegmentation tasks.Rotary Positional EmbeddingsRotary Positional Embeddings (RoPE) [14] represent an alternative approach toencoding spatial relationships in vision transformers. Originally developed forlanguage models, RoPE injects positional information directly into the query andkey vectors, rather than adding positional biases to attention scores after query-keymultiplication.For 2D vision applications, RoPE is extended through axial frequency im-plementation that handles spatial coordinates separately. The method divides theembedding dimensions into two parts, applying rotation operations for x and ydimensions independently, preserving relative distance information across bothspatial dimensions. Similar to decomposed relative positional embeddings, thisapproach processes height and width dimensions separately to efficiently encode 2Dspatial relationships.RoPE’s effectiveness in vision applications is demonstrated by models likeDINOv3 [45], which achieves strong self-supervised visual representation learning.The method shows particularly impressive extrapolation performance, maintainingprecision when increasing image resolution at inference time while providing perfor-mance improvements across image classification, object detection, and semanticsegmentation tasks.15Chapter 3Prior WorksThis chapter surveys existing token reduction techniques and examines their funda-mental limitations in addressing the challenges posed by modern Vision Transformerarchitectures. While token reduction has emerged as an attractive efficiency strategydue to its potential to work without architectural modifications, existing approachesface critical incompatibilities with two key trends in modern Vision Transformers: thescale of contemporary foundation models and the prevalence of spatial architectures.The widespread adoption of spatial architectures [19, 25, 26, 31] that employwindow attention and 2D positional embeddings requires token reduction methodsto preserve structured spatial layouts, which is a requirement that most existingapproaches fail to satisfy. Simultaneously, the extensive pretraining efforts invested inlarge-scale models like DINOv3 [45] make retraining-based approaches prohibitivelyexpensive, creating a pressing need for training-free solutions.3.1 Token PruningEarly token reduction methods primarily focused on token pruning [20, 28, 41]. Theseapproaches train lightweight policy networks or selection modules to dynamicallyidentify and remove redundant tokens during inference. Motivated by classificationtasks where only the [𝑐𝑙𝑠] token output is required for final prediction, these methodsshare the observation that vision transformers exhibit sparse attention patterns whereonly a subset of tokens contribute meaningfully to the classification result.16DynamicViT [41] introduces a lightweight prediction module that estimatestoken importance scores. The method uses attention masking to differentiably prunetokens by blocking their interactions with other tokens during training. EViT [28]identifies attentive tokens using class token attention and fuses inattentive tokens.SPViT [20] employs a dynamic attention-based token selector and introduces softpruning, which integrates less informative tokens into a package token rather thandiscarding them completely.While effective for early classification models, these approaches suffer fromcritical limitations: (1) they cannot recover discarded or fused tokens, making themincompatible with dense prediction tasks that require all patch token informationto be preserved [25, 26, 43], and (2) they introduce extra parameters, necessitatingretraining of additional modules alongside the backbone model.3.2 Training-based Token ReductionBeyond these early classification-focused pruning methods, many recent tokenreduction approaches target broader applications but still require extensive retrain-ing [21, 30, 32, 33, 52].BAT [32], GTP-ViT [52] and METR [29] focus on classification tasks withoutdemonstrating effectiveness for dense prediction. BAT [32] considers both tokenimportance and global diversity and decouples tokens based on class token attention.GTP-ViT [52] introduces Graph-based Token Propagation that propagates informa-tion from less significant tokens to spatially and semantically connected importanttokens. METR [29] combines multi-exit architecture with token reduction.Revisiting Token Pruning for Object Detection and Instance Segmentation [30],DTEM [21], and CTS [33] extends their applications to dense prediction tasksbut add additional parameters and therefore requires full retraining. RevisitingToken Pruning for Object Detection and Instance Segmentation extends pruning todense prediction by preserving discarded tokens in feature maps, enabling tokenreactivation. DTEM uses a decoupled embedding module to learn dedicated featuresfor token merging. CTS targets semantic segmentation by enabling patches with thesame semantic class to share tokens, employing a policy network to predict semanticsimilarity between patches.17However, these training-based approaches present significant challenges forlarge-scale models due to high computational resource demands and limited dataavailability, making them impractical for modern foundation models that requireextensive pretraining investments.3.3 Graph-based Token MergingToken Merging (ToMe) [3] represents a significant milestone that combines tokensrather than discarding them, addressing the token recovery limitation of pruningmethods by preserving information through weighted averaging of similar tokens,allowing full token features to be recovered for downstream processing. It alsoaddresses the retraining challenges of large-scale models by showing promisingresults with or without retraining.ToMe introduces a bipartite soft matching algorithm that partitions tokens intotwo sets and draws edges between similar tokens based on key similarity fromattention mechanisms. By restricting each token to nominate only one edge to theopposite set, this bipartite structure enables highly parallelizable graph matching.To address the issue that merged tokens lose attention influence compared to theirconstituent tokens, the method employs proportional attention that adds log 𝑠 toattention scores, where 𝑠 represents the number of tokens each merged token contains,effectively scaling the attention as if multiple copies of the original key were present.ToMe’s foundational bipartite matching framework has inspired numeroussubsequent works that build upon its core algorithm through targeted improvements.These ToMe-based adaptations can be categorized into several research directions:adaptive merging strategies that dynamically adjust reduction rates, importance-aware approaches that selectively protect informative tokens, hybrid methods thatcombine multiple reduction techniques, and spatial-aware variants that attempt toincorporate spatial locality constraints.Adaptive ToMe variants focus on improving the flexibility of token reductionby developing learnable or dynamic merging rates [7, 37]. ALGM [37] introducesadaptive token merging that automatically determines similarity thresholds basedon image complexity, allowing for different reduction rates depending on thecontent rather than using fixed token counts across all images. DiffRate [7] makes18compression ratios differentiable, allowing different layers to automatically learnoptimal layer-wise compression rates through gradient-based optimization.Importance-aware ToMe extensions address the original method’s limitationof treating all tokens equally by incorporating importance scores to avoid mergingsemantically critical tokens [22, 32, 48]. MCTF [22] introduces multi-criteria tokenfusion that considers similarity and informativeness simultaneously. BAT [32] em-phasizes preserving global token diversity alongside token importance. PiToMe [48]introduces an energy score metric that identifies large clusters of similar tokens ascandidates for merging.Hybrid ToMe approaches explore combining token merging with complementaryreduction strategies, particularly integrating ToMe’s merging operations with tokenpruning techniques [18]. ToFu [18] combines token pruning and merging by utilizinga hybrid strategy that transitions from pruning in early layers to merging in deeperlayers.Spatial-aware ToMe variants attempt to address spatial locality by constrainingtoken matching to local regions [16, 37]. ToSA [16] incorporates spatial informationalongside semantic similarity for token matching. ALGM [37] implements a two-stage local-then-global approach that first merges similar tokens within small localwindows in early layers. However, these approaches focus only on matching spatiallynear tokens while neglecting spatial coherence in the resulting merged tokens.Despite these diverse extensions and improvements, all ToMe-based approachesfundamentally inherit the original method’s core limitation: they produce unstruc-tured token arrangements that are inherently incompatible with spatial architecturesrequiring regular 2D token layouts. Even spatial-aware methods fail to preserve spa-tial coherence in the resulting tokens after merging, as the merged token distributionremains irregular and cannot maintain the structured 2D layouts essential for spatialarchitectures.3.4 Clustering-based Token MergingExpedite [27] represents the only existing method that preserves spatial structureby employing a clustering-based approach with k-means clustering on superpixelsinitialized through adaptive average pooling, producing structured 2D layouts19compatible with spatial architectures. However, Expedite suffers from a limitation:it modifies every token during the clustering process by reassigning all tokensto superpixels via weighted averaging. By forcing semantically unique tokensinto clusters with dissimilar tokens, the method causes irreversible informationdegradation. This causes significant performance drops, especially when applied toearly layers (see Section 5.2).AiluRus [24] also noted this weakness and built upon Expedite’s clusteringapproach, addressing this limitation by identifying cluster centers based on semanticimportance rather than spatial organization. However, this improvement consequentlyfails to maintain the regular 2D spatial structure required by spatial architectures.3.5 Task-specific Token ReductionRecent token reduction techniques have increasingly targeted complex tasks, de-veloping specialized approaches for specific modalities and applications whileincorporating task-specific constraints and requirements.Video understanding methods focus on addressing the temporal redundancyinherent in video data, where consecutive frames often contain similar visualcontent [9, 23, 44].Segmentation-focused techniques exploit the spatial coherence principle thatneighboring pixels typically belong to the same object or semantic class, enablingtoken sharing among spatially near patches [33, 37].Vision-language token reduction methods target the computational overheadcaused by the substantial increase in token count when combining visual and textualmodalities [1, 15, 53]. These methods address the extended input sequences inLarge Language Model decoders while preserving cross-modal semantic alignmentessential for vision-language understanding tasks.While these methods have shown success in their specialized domains, they donot address the fundamental spatial compatibility challenge we tackle: maintainingstructured token layouts essential for spatial architectures. A gap remains forgeneral-purpose token reduction methods that can work effectively with the growingprevalence of spatial-aware Vision Transformer designs.203.6 SummaryOur survey reveals a critical limitation across existing token reduction approaches:the vast majority of methods fail to preserve the structured 2D spatial layouts requiredby modern spatial architectures.While Expedite successfully addresses this challenge by maintaining spatialstructure compatibility and requiring no retraining, it suffers from a critical weaknessin that it fails to exploit the uneven information density inherent in visual representa-tions, leading to performance degradation. This creates a clear gap for methods thatcan reconcile spatial structure preservation with selective token reduction based oninformation density.21Chapter 4CubistMergeExisting token reduction methods face a fundamental dilemma: they either fail topreserve spatial structure [3] or fail to exploit uneven information density across thespatial layout [27] (see Figure 1.1). To address this, we employ three key innovations:1. 2D token reduction strategy: Enforces structured spatial layout by reducingtokens sequentially in each dimension2. Spatial-aware token merging: Selectively targets redundant tokens whilepreserving relative spatial relationships3. Max-magnitude-per-dimension token representation: Preserves salientfeatures without requiring layer-wise attention rescalingThrough these innovations, we maintain a structured 2D token layout postmerging and preserve relative spatial relationships among tokens, ensuring that theinherent spatial coherence of the input tokens is retained while achieving efficienttoken reduction.4.1 2D Reduction StrategyWe observe that most existing token reduction methods such as ToMe [3] break2D spatial coherence due to uneven token counts across rows and columns (seeFigure 1.1(b)). To address this, we reduce tokens in each dimension sequentially, toensure consistent token counts per row and column.224.1.1 AlgorithmTo operate on 𝐻 ×𝑊 tokens representing a 2D spatial layout, the 2D reductionperforms two sequential phases:1. Horizontal Reduction: Reduce 𝑟𝑤 tokens from each row, resulting in 𝐻 ×(𝑊 − 𝑟𝑤) tokens.2. Vertical Reduction: Reduce 𝑟ℎ tokens from each column, resulting in (𝐻 −𝑟ℎ) × (𝑊 − 𝑟𝑤) tokens.This 2D reduction strategy maintains consistent token counts per row and columnpost reduction, ensuring that tokens can be reorganized into a structured 2D layoutof (𝐻 − 𝑟ℎ) × (𝑊 − 𝑟𝑤) tokens (see Figure 4.1).Implementation DetailsTo realize this conceptual framework in practice, we present the concrete algorithmwith explicit tensor operations in Algorithm 1:Algorithm 1: 2D Token Reduction AlgorithmInput: 𝑋 ∈ R𝑁×𝐻×𝑊×𝐷Output: 𝑋 ′′ ∈ R𝑁×(𝐻−𝑟ℎ )×(𝑊−𝑟𝑤 )×𝐷// Horizontal Reduction𝑋 = 𝑋.transform( [𝑁, 𝐻,𝑊, 𝐷] → [𝑁 × 𝐻,𝑊, 𝐷])𝑋 ′ = BatchTokenMerge(𝑋, 𝑟𝑤)// Vertical Reduction𝑋 ′ = 𝑋 ′ .transform( [𝑁 × 𝐻, (𝑊 − 𝑟𝑤), 𝐷] → [𝑁 × (𝑊 − 𝑟𝑤), 𝐻, 𝐷])𝑋 ′′ = BatchTokenMerge(𝑋 ′, 𝑟ℎ)return 𝑋 ′′ .transform( [𝑁 × (𝑊 − 𝑟𝑤), (𝐻 − 𝑟ℎ), 𝐷] → [𝑁, (𝐻 − 𝑟ℎ), (𝑊 − 𝑟𝑤), 𝐷])The BatchTokenMerge operations in both phases use our spatial-aware tokenmatching algorithm described in Section 4.2.4.1.2 Window Attention AdoptionTo better adapt to window attention, we perform 2D reduction independently withineach window, restricting token merging among tokens within the same window.Specifically, given a feature map partitioned into non-overlapping 𝑀 × 𝑀 windows,we apply our 2D reduction algorithm to each window’s 𝑀×𝑀 token set independently,reducing tokens to (𝑀 − 𝑟ℎ) × (𝑀 − 𝑟𝑤) within each window. To preserve window23Figure 4.1: 2D token reduction with spatial-aware merging: (1) original 14×14tokens, (2) select horizontal tokens to merge, (3) merge horizontally to14×12 tokens, (4) select vertical tokens to merge, (5) merge vertically to12×12 tokens.boundaries in subsequent layers, we correspondingly adjust the window size to(𝑀 − 𝑟ℎ) × (𝑀 − 𝑟𝑤), ensuring that tokens originally grouped within the samewindow remain together throughout the network. This window-constrained approachmaintains the locality assumptions critical for windowed attention efficiency whileenabling token reduction.Both phases use our spatial-aware token matching algorithm described inSection 4.2.4.2 Spatial-Aware Token MatchingWe employ a graph-based token reduction approach that allows us to selectivelyidentify and merge redundant tokens based on their similarity, while leaving distincttokens untouched to preserve their unique visual information.To further preserve spatial coherence, we propose a novel spatial-aware tokenmatching algorithm that further preserves relative token relationships. It has 2 keydesigns: graph construction, which defines the allowable merge candidates betweenspatially related tokens, and edge selection, which determines the specific subset ofthese candidate token pairs to merge.4.2.1 Graph ConstructionSince our 2D reduction operates on each row and column independently, tokenswithin each subset naturally form a linear arrangement based on their spatial24positions.Motivated by this, we use a path graph to further preserve spatial coherence andcomputational simplicity, where each token only connects to its adjacent neighborswithin the same row or column. This design ensures that merged tokens maintainthe original relative spatial positions of their constituent tokens, which is critical for2D positional embeddings [14, 26].Figure 4.2: Path graph illustration showing tokens arranged in a linear sequence,where each token connects only to its adjacent neighbors, forming thebasis for spatial-aware token merging.Reduced ComplexityAdditionally, path graphs minimize the number of edges by connecting only adjacenttokens, reducing the complexity of computing similarity scores from 𝑂(𝑁2)to𝑂 (𝑁) compared to global token merging algorithms such as ToMe [3] that computesall pairwise token similarities. Table 4.1 summarizes the computational complexitycomparison.Method ComplexityBipartite soft matching [3] 𝑂 (𝑁2)Path graph matching (CubistMerge) 𝑂 (𝑁)Table 4.1: Computational complexity of graph construction algorithms fortoken matching.While this computational advantage may not be substantial for current typicalsequence lengths, it may become increasingly valuable as the field moves towardlonger token sequences. Recent work [51] argues that ViT’s compressive encodinginappropriately treats information-rich visual data like text, causing unnecessaryinformation loss, and demonstrates the potential for scaling token counts, makingour linear complexity advantage increasingly pronounced.254.2.2 Edge SelectionHaving constructed the path graph, we need to determine which edges to select fortoken merging. Each edge in our path graph represents a potential merge operationbetween two spatially adjacent tokens.We select only edges representing high token similarity for merging, basedon cosine similarity computed between token embeddings. By selecting high-similarity edges, we merge only redundant tokens while leaving semantically distincttokens unaltered. This selective approach ensures that unique visual information ispreserved throughout the network, while redundant spatial information is efficientlyconsolidated.Naive ApproachThe naive optimal approach would be to select the top-k most similar edges fromthe path graph for merging. However, this can create processing dependencieswhich limit parallelization. Specifically, when a token can be both the source anddestination of selected edges, it cannot merge as a source token until it has firstcompleted its role as a destination token in another merge operation.These dependencies force sequential execution, resulting in 𝑂 (𝑁) time com-plexity with the naive implementation (shown in Figure 4.3(a)). Reduction treeapproaches can theoretically improve this to 𝑂 (log 𝑁) by organizing merging oper-ations, as shown in Figure 4.3(b): during the first step, tokens b→c and d→e canbe merged simultaneously, then in the second step, c→d is redirected to directlymerging c→e, achieving the same final result while reducing dependency depth.However, such tree-structured computations are poorly supported by frameworkslike PyTorch and GPUs. Moreover, even with logarithmic time complexity, theinherent sequential dependencies still prevent effective parallelization across thetoken merging process.Node BipartitionTo enable better parallelization, we adopt ToMe’s [3] node bipartition approach, whichalternates token role assignments so that adjacent tokens have complementary roles(source and destination). Each source token then nominates its most similar adjacent26neighbor as its merge destination. We select the top-k edges from these nominations,which are guaranteed to be dependency-free and can execute simultaneously usingthe scatter-reduce workflow that is highly parallelizable and well-supported bycommon frameworks [39]. Figure 4.3(c) illustrates this bipartite approach on pathgraph.However, while this design choice prioritizes efficiency, it does not guaranteeselection of strictly the most similar edges, as shown in Figure 4.3(a) vs. (c).4.2.3 Ablation StudiesWe conduct ablation experiments to validate our design choice of graph construction,and to evaluate the trade-off between parallelization efficiency and edge selectionoptimality of naive vs. bipartite edge selection.We conducted ablation experiments on 500 randomly selected COCO trainingimages with token merging methods applied off-the-shelf on SAM-H and SAM-Bacross different token reduction rates and application depths. As shown in Table 4.2,comparing our bipartite approach (“bipartite, local”) against naive top-k edgeselection (“naive, local”) reveals minimal performance differences, and the bipartiteapproach achieves the same or better mIOU in 5 out of 8 experimental settings,while the parallelization enables higher speedups compared to the naive approach.Based on this, we adopt bipartite edge selection with path graph as our design.Additionally, we evaluate against global bipartite matching (“bipartite, global”)from ToMe [3] (illustrated in Figure 4.3(d)). This comparison validates that ourspatially-constrained approach outperform the conventional global matching. SeeFigure 4.3 for illustrations of all described edge selection algorithms.4.3 Max-Magnitude-Per-Dimension Token RepresentationAnother key contribution lies in how we represent merged tokens. The mostcommonly used token representation is weighted averaging introduced in ToMe [3],creating merged representations that are insufficient to attract appropriate attentionfor the multiple tokens they represent. ToMe addresses this using proportionalattention scaling, which can introduce computational overhead of up to 3% ofruntime for DeiT-B and complicating adoption in models with optimized attention27Depth 0 1/4 1/2𝑟ℎ = 𝑟𝑤 4 8 4Method ΔmIOU Speedup ΔmIOU Speedup ΔmIOU Speedup ΔmIOU Speedupbipartite, local -2.23% 1.68 -3.61% 2.05 -1.47% 1.47 -0.72% 1.31SAM-H naive, local -2.17% 1.64 -3.53% 1.99 -1.57% 1.44 -0.72% 1.25bipartite, global -2.47% 1.63 -3.84% 2.03 -1.63% 1.46 -0.77% 1.29bipartite, local -1.49% 1.69 -2.48% 1.94 -1.15% 1.44 -0.47% 1.27SAM-B naive, local -1.60% 1.55 -2.42% 1.70 -1.16% 1.36 -0.51% 1.15bipartite, global -1.62% 1.69 -2.59% 1.95 -1.16% 1.43 -0.51% 1.27Table 4.2: Ablation study comparing graph construction and edge selectionmethods for CubistMerge. Results show ΔmIOU and speedup relativeto the baseline model without token reduction. Our chosen design ishighlighted in gray.implementations.To address this issue, we observe that (i) high-magnitude values in tokenembeddings naturally attract more attention, reflecting more salient features; and(ii) averaging among multiple tokens reduces those highest-magnitude values. Insteadof averaging, therefore, we perform max-magnitude-per-dimension operations acrosstokens being merged, preserving the most prominent values from each dimension.This eliminates the need for token size tracking and layer-by-layer attention scalingwhile ensuring merged tokens remain representative.4.3.1 AlgorithmFormally, for a set of tokens {𝑡1, 𝑡2, . . . , 𝑡𝑛} to be merged, where each token 𝑡 𝑗 ∈ R𝑑 ,the merged token 𝑡𝑚 is computed as𝑡𝑚 [𝑖] = 𝑡𝑐 [𝑖] where 𝑐 = argmax𝑗∈{1,2,...,𝑛}( |𝑡 𝑗 [𝑖] |)for each dimension 𝑖 ∈ {0, 1, 2, . . . , 𝑑}. This operation selects, for each dimensionindependently, the value from whichever token has the maximum absolute value inthat dimension, effectively preserving the most salient feature across all candidatetokens while maintaining both magnitude and sign information.284.3.2 Ablation StudiesWe conduct ablation experiments to validate the effectiveness of our proposed Max-Magnitude-Per-Dimension token representation. Using the same experimental setupas described in Section 4.2, we compare our approach against two other methods: (1)weighted average, the most commonly used token representation method introducedin ToMe [3], and (2) max-magnitude-vector, which selects the token with the highestL1 norm, serves to validate whether our method’s superior performance stems fromthe per-dimension selection principle or merely from choosing more values fromtokens that happen to be more informative.As shown in Table 4.3, per-dimension consistently outperforms both methodsacross different settings in both accuracy and speedup. This validates that our max-magnitude-per-dimension approach is genuinely superior to the standard averagingmethod and not merely benefiting from selecting tokens that are more suitable forpreservation during merging.Depth 0 1/4 1/2𝑟ℎ = 𝑟𝑤 4 8 4Method ΔmIOU Speedup ΔmIOU Speedup ΔmIOU Speedup ΔmIOU SpeedupMax-Per-Dim -2.23% 1.68 -3.61% 2.05 -1.47% 1.47 -0.73% 1.31SAM-H Max-Vector -2.54% 1.68 -3.84% 2.04 -1.66% 1.47 -0.93% 1.30Weighted Average -2.48% 1.63 -3.74% 2.00 -1.63% 1.44 -0.83% 1.28Max-Per-Dim -1.49% 1.69 -2.48% 1.94 -1.15% 1.44 -0.47% 1.27SAM-B Max-Vector -1.67% 1.68 -2.92% 1.92 -1.29% 1.45 -0.60% 1.27Weighted Average -1.57% 1.55 -2.92% 1.83 -1.29% 1.38 -0.44% 1.24Table 4.3: Ablation study comparing token representation methods for Cubist-Merge. Results show ΔmIOU and speedup relative to the baseline modelwithout token reduction. Our chosen design is highlighted in gray.29(a) naive,local③②①(b) naive,local (log 𝑁)f ed cb a①①②(c) bipartite,local①①①(d) bipartite,global①①①Figure 4.3: Illustration of edge selection algorithms. Arrows on selected edges(in orange) indicate the direction of token merging, pointing from thesource token to the destination token. The numbers on edges representexecution order of merging required by dependencies. (a) Path graphwith naive edge selection, (b) Path graph with naive edge selection,optimized with reduction tree to 𝑂 (log 𝑁), (c) Path graph with bipartiteedge selection to eliminate dependencies by ensuring each source token(in red) can only merge to one destination (in blue), (d) Bipartite SoftMatching [3].30Chapter 5ExperimentsWe conduct comprehensive experiments across various architectures and vision tasksto evaluate the effectiveness of CubistMerge. Our experimental design addressesthree primary research questions:• Does our method effectively preserve spatial relationships across diversespatial architectures?• How does our approach generalize across different vision tasks?• How does our method’s quality-efficiency tradeoff (performance drop vs.computational speedup) compare against existing token reduction methods?To answer the first research question, we include spatial architectures withdiverse spatial components in our evaluation. We include models that use bothshifting [31] and non-shifting [25] window attention. For 2D positional embeddings,we include models that use decomposed relative positional embeddings [26] andRoPE [14]. This diversity evaluates our method’s compatibility across the spectrumof modern spatial architectural designs.To address generalizability, we evaluate across diverse vision tasks includingimage classification, object detection, instance segmentation and panoptic segmen-tation.To assess the competitiveness of our method, we conduct comprehensivecomparisons against Expedite [27], the only prior method capable of preserving31spatial structure, across every spatial architecture experiment. To enable even broadercomparative evaluation against a wider range of existing methods, we extend ourevaluation to non-spatial architectures, where more existing methods are compatible.5.1 Experiment SetupThis section outlines the experimental setup and configuration details that remainconsistent across all our evaluations. We establish standardized protocols for perfor-mance measurement, including the computational metrics, task-specific evaluationcriteria, and datasets used to assess model effectiveness. Additionally, we define thekey parameters for applying CubistMerge: layer selection and reduction rates.5.1.1 DatasetsImageNet is a large-scale visual recognition dataset widely used as the standardbenchmark for image classification tasks, containing 1,000 object classes in theImageNet-1K variant. We use ImageNet-1K for image classification evaluation.COCO serves as a standard benchmark for computer vision tasks such as objectdetection and segmentation. It contains bounding boxes and segmentation masksencoded in RLE (run length encoding) for 80 specific “things” (individual objectslike a person or a car) and 91 “stuff” categories (amorphous regions like grassor sky). We use COCO for object detection, instance segmentation, and panopticsegmentation evaluation.ADE20K is a scene parsing dataset with dense pixel-level annotations for 150semantic categories including objects, parts, and scene elements across diverseindoor and outdoor scenes. We use ADE20K for semantic and instance segmentationevaluation.Cityscapes is an urban scene understanding dataset featuring high-resolutionimages from 50 cities with fine pixel-level annotations for 19 semantic classesin autonomous driving scenarios. We use Cityscapes for semantic segmentationevaluation.Pascal Context is a semantic segmentation dataset providing dense annotationsfor over 400 semantic categories with fine-grained object and scene understanding.We use Pascal Context for semantic segmentation evaluation.325.1.2 Tasks and MetricsThis subsection describes the computer vision tasks we evaluate across to demonstratethe generalizability of CubistMerge and the computational efficiency metrics we useto assess performance improvements.Tasks and Quality MetricsFor each computer vision task below, we describe the problem formulation and thecorresponding evaluation metrics used to assess model performance.Image Classification involves assigning a single category label to an entireimage from a predefined set of classes. We evaluate classification performance usingTop-1 accuracy, which measures the percentage of images where the model’s highestconfidence prediction matches the ground truth label, and Top-5 accuracy, whichmeasures the percentage of images where the ground truth label appears among themodel’s top five predictions.Object Detection simultaneously localizes and classifies multiple objects withinan image by predicting bounding boxes and their corresponding class labels. Weevaluate detection performance using Average Precision (AP), which summarizesthe precision-recall curve by computing the area under the curve, providing a singlemetric that captures both localization accuracy and classification performance acrossdifferent confidence thresholds.Instance Segmentation extends object detection by providing pixel-level masksfor each detected object instance, distinguishing between different instances of thesame class. We evaluate instance segmentation using mean Intersection over Union(mIOU), which measures the average overlap between predicted and ground truthmasks across all object instances, computed as the ratio of intersection area to unionarea for each mask.Panoptic Segmentation unifies semantic and instance segmentation by providingdense pixel-level predictions that assign each pixel to either a specific object instance(“things”) or an amorphous region (“stuff”). We evaluate panoptic segmentationusing Panoptic Quality (PQ), which combines segmentation quality and recognitionquality by measuring both the accuracy of pixel-level predictions and the correctnessof instance-level assignments.33Computational Efficiency MetricsIn addition to task-specific performance metrics, we comprehensively evaluatecomputational efficiency through both runtime-based speedup measurements andtheoretical floating-point operation counts.Speedup quantifies the practical acceleration achieved by token reductionmethods during inference. We calculate speedups as the ratio of baseline modelruntime to the accelerated model runtime, measured through wall-clock time onactual hardware. All speedup measurements are conducted on RTX 2080 Ti GPUsto ensure consistent hardware-dependent factors such as memory bandwidth andcompute throughput, except for DINOv3 experiments which require V100 GPUsdue to the RTX 2080 Ti’s insufficient memory capacity for the large model size. Toensure measurement consistency, we conduct all runtime experiments on dedicatedGPUs with no other concurrent processes.Floating Point Operations (FLOPs) provide a hardware-independent theoreti-cal estimate of computational complexity by counting the number of floating-pointarithmetic operations required for model inference. We measure FLOPs using thefvcore library [35], which performs static analysis of the computational graphto count floating-point operations. FLOPs are a standard metric for measuringcomputational complexity across machine learning models, providing a comparisonmetric independent of hardware specifics. However, FLOPs may not fully captureruntime overheads such as data dependency and parallelization that affect actualinference speed.5.1.3 ConfigurationsThis subsection describes the experimental configurations that determine how weapply token merging methods across different models and settings.Training ConfigurationsWe evaluate token reduction methods under two settings: off-the-shelf applicationand fine-tuned adaptation. Off-the-shelf application refers to applying token reductionmethods directly to pretrained models without any additional training, providingimmediate acceleration with minimal computational overhead. Fine-tuned adaptation34involves further training the model with the token reduction method applied, allowingthe model to adapt to the reduced token representation.By default, all experiments apply token reduction methods off-the-shelf withoutadditional training, with fine-tuning results specifically noted where applicable.CubistMerge ConfigurationsCubistMerge’s experimental configuration involves two key variables that determinehow the method is applied to each model:• the layer 𝑙 where token reduction is applied• the reduction rate 𝑟ℎ and 𝑟𝑤The reduction rate, specified by parameters 𝑟ℎ and 𝑟𝑤 , denotes how many tokensare reduced from every row and column respectively. Given 𝐻 ×𝑊 input tokens, weuse integer values 𝑟ℎ = 𝑚, 𝑟𝑤 = 𝑛 for models with consistent token counts across allinputs, resulting in (𝐻 − 𝑚) × (𝑊 − 𝑛) tokens. For models where token counts mayvary based on input image size, we use fractional values 𝑟ℎ = 𝑎, 𝑟𝑤 = 𝑏, resulting in(𝐻 − 𝑎 × 𝐻) × (𝑊 − 𝑏 ×𝑊) tokens.5.2 Experiments on Spatial ArchitecturesWe evaluate CubistMerge on models with spatial architectures that cover diversespatial design choices across classification, object detection, instance segmentationand panoptic segmentation tasks. Our evaluation includes models with differentattention mechanisms: shifted window attention (Mask2Former [8]) and window-global attention (MViTv2 [26], ViTDet [25], SAM [19], SAM2 [42]). For positionalembeddings, we evaluate models using decomposed relative positional embeddings(MViTv2 [26], ViTDet [25], SAM [19]) and RoPE (DINOv3 [45]).5.2.1 Prior Work ComparisonThis subsection describes our approach to fair and comprehensive comparison withexisting token reduction methods. We outline our baseline method selection andlayer selection strategies for fair comparison and assessing layer sensitivity.35Baseline MethodsWe primarily compare against Expedite [27] as the only existing method suitablefor spatial architectures. Additionally, we include ToMe [3] in selected experimentsto demonstrate the performance gap between spatial-preserving and non-spatial-preserving methods. We exclude ToMe results when the performance degradationexceeds 20% as such large drops preclude meaningful comparison.Layer SelectionTo assess layer sensitivity, we conduct experiments with token reduction insertedat different layers within each architecture, examining how performance varieswhen merging is applied at early vs. later layers. Throughout our experiments, wedemonstrate that CubistMerge consistently outperforms existing methods whilemaintaining more consistent performance across different layers.We ensure all of our evaluations include results from Expedite’s optimalconfiguration to guarantee fair comparison. When available, we use the recommendedlayer settings from the original Expedite paper or official repository. Otherwise, wesystematically test Expedite across 4–6 different layers to identify its best-performingconfiguration. We ensure all selected layers maintain reasonable performance withoutsubstantial metric degradation. Additionally, we conduct layer sensitivity analysisusing a default early layer configuration, typically the first layer, or for architectureswith multiple stages, the first layer of the deepest stage.5.2.2 Model SweepThis subsection presents comprehensive evaluation results across diverse spatialarchitectures, demonstrating CubistMerge’s effectiveness across different spatialdesign choices. We organize our evaluation by architecture families, covering modelswith varying attention mechanisms and positional embedding strategies to establishbroad applicability.MViTv2We evaluate MViTv2 [26], which uses decomposed relative positional embeddings,on image classification. Table 5.1 presents layer sensitivity results and includes fine-36Figure 5.1: Image classification results on spatial architectures, varying 𝑟ℎ =𝑟𝑤 = 1, 2, 3 with 𝑙 = 10 on MViTv2-B, 𝑙 = 20 on DINOv3-ViT7B andMViTv2-L.tuning results for MViTv2-B, with fine-tuning limited to 3 epochs. For meaningfulcomparisons, we use 𝑙 = 10 for MViTv2-B and 𝑙 = 20 for MViTv2-L, whereExpedite and ToMe achieve more reasonable performance, and we vary 𝑟ℎ and 𝑟𝑤to produce the results shown in Figure 5.1.37DINOv3-ViT7BMethod Top1(%) Top5(%) Speedup GFLOPsBaseline 88.0 98.4 1.00× 1349.9ToMe 𝑙=10 84.3 97.1 1.12× 1214.0Expedite 𝑙=10 87.1 98.2 1.12× 1215.3CuMe 𝑙=10 87.7 98.2 1.12× 1213.9ToMe 𝑙=20 86.9 98.1 1.07× 1259.3Expedite 𝑙=20 87.7 98.4 1.07× 1259.2CuMe 𝑙=20 87.9 98.4 1.09× 1259.2MViTv2-BMethod Top1(%) Top5(%) Speedup GFLOPs Train Mem (G)Baseline 84.2 96.8 1.00× 10.2 64.1ToMe 𝑙=7 69.8 88.3 1.07× 9.3 -Expedite 𝑙=7 81.4 95.5 1.05× 9.5 -CuMe 𝑙=7 82.6 96.2 1.07× 9.3 -ToMe 𝑙=10 79.8 94.6 1.06× 9.5 -Expedite 𝑙=10 83.6 96.5 1.03× 9.6 -CuMe 𝑙=10 83.8 96.6 1.06× 9.5 -Expedite 𝑙=10 83.8 96.7 1.03× 9.6 61.0CuMe 𝑙=10 84.1 96.7 1.06× 9.5 61.0MViTv2-LMethod Top1(%) Top5(%) Speedup GFLOPsBaseline 85.3 97.1 1.00× 43.9ToMe 𝑙=9 68.5 87.6 1.24× 39.8Expedite 𝑙=9 83.8 96.4 1.20× 40.0CuMe 𝑙=9 84.3 96.6 1.26× 39.8ToMe 𝑙=20 83.9 96.6 1.20× 41.1Expedite 𝑙=20 84.8 97.0 1.19× 41.3CuMe 𝑙=20 85.0 97.0 1.22× 41.1Table 5.1: Image classification results on spatial architectures, with 𝑟ℎ = 𝑟𝑤 =1. Fine-tuned results (highlighted in cyan) are included for MViTv2-Bwithin 3 epochs of training.DINOv3We evaluate DINOv3 with ViT7B backbone [45], which incorporates RoPE for2D positional embeddings, on both image classification and object detection tasks.Classification results are presented in Table 5.1 and Figure 5.1, while object detectionresults are shown in Table 5.2 and Figure 5.2. We experiment with 𝑙 = 10 and38Figure 5.2: Object detection results. For DINOv3-ViT7B, we vary 𝑟ℎ = 𝑟𝑤 =0.1, 0.15, 0.2 at 𝑙 = 20. For ViTDet, we vary 𝑟ℎ = 𝑟𝑤 = 4, 8, 12 at 𝑙 = 0for CuMe and 𝑙 = 2 for Expedite.𝑙 = 20, selecting 𝑙 = 20 where Expedite and ToMe exhibit more reasonableperformance for comprehensive evaluation varying 𝑟ℎ and 𝑟𝑤 . Notably, despite usingthe same pretrained backbone, ToMe and Expedite show significantly degraded layersensitivity for object detection at 𝑙 = 10, while CubistMerge maintains consistentperformance across both tasks.39Model AP SpeedupBaseline 57.4 1.00×ToMe 𝑙=20 52.9 1.08×Expedite 𝑙=20 55.3 1.07×CuMe 𝑙=20 56.9 1.08×ToMe 𝑙=10 31.0 1.13×Expedite 𝑙=10 37.2 1.11×CuMe 𝑙=10 55.5 1.12×Table 5.2: Layer sensitivity analysis of object detection on DINOv3-ViT7Bwith 𝑟ℎ = 𝑟𝑤 = 0.1ViTDetWe further evaluate object detection using ViTDet [25], which employs windowattention and decomposed relative positional embedding [26]. We used the bestperforming backbone (ViT-H) with Mask R-CNN and Cascade Mask R-CNN asbaseline. We apply CubistMerge with 𝑙 = 0 by default. However, Expedite performspoorly at 𝑙 = 0 with over 40 in AP drop, so we use 𝑙 = 2 for Expedite, which weexperimentally determined as the best performing configuration after testing across6 different layer depths. Results are shown in Figure 5.2.SAMWe evaluate instance segmentation on SAM [19], which uses ViTDet backbonearchitecture. Figure 5.3a shows layer sensitivity analysis demonstrating CuMe’ssuperior consistency across layers compared to Expedite. We conduct full evaluationson COCO (Figure 5.3b) and ADE20K (Figure 5.3c) across all model variants usingbounding box prompts, with CuMe applied at 𝑙 = 0 and Expedite at its recommendedlayer1.CuMe maintains stable performance regardless of application depth, showingminimal performance variation across all tested layers. In contrast, Expedite exhibitssignificant layer sensitivity, particularly at early layers such as layer 0, which alignswith observations from prior work that also reported this phenomenon [24].We attribute this sensitivity to Expedite’s clustering algorithm being highly1Expedite only provided recommended settings (𝑙 = 6 and 𝑙 = 16) for SAM-H. We scale therelative depth accordingly for SAM-L and SAM-B.40sensitive to the token representations it operates on. By employing uniform clustercenter initialization and assigning every token to a cluster, the algorithm modifiesall tokens indiscriminately, failing to isolate semantically distinct tokens. In contrast,CuMe selectively merges redundant tokens while leaving semantically distinct onesunaltered, as described in Section 4.2.2.This indiscriminate modification renders Expedite highly sensitive to the specifictoken representations at each layer. As token characteristics evolve across transformerlayers, this sensitivity manifests as varying performance across different applicationdepths, a phenomenon also observed in ablation experiments by AiluRus [24].Further evidence comes from Expedite’s own reported optimal layer selections,which vary substantially across models in both absolute layer indices and proportionaldepth [27].SAM2We extend our evaluation to SAM2 [42], which uses Hiera [43] backbone thatemploys window attention. Expedite generally performs poorly on SAM2 with 20+%mIOU drops; we experiment with Expedite on 6 different layers and reported thebest performance found. To provide additional baselines, we leverage the fact thatSAM2 does not use 2D positional embeddings and has consistent window partitionwithin its deepest stage, allowing ToMe to operate on each window individually toachieve reasonable results. We apply CuMe and ToMe at 𝑙 = 9 (the first layer of thedeepest stage). Figure 5.6 presents instance segmentation results on SAM2-L acrossCOCO and ADE20K datasets.Mask2FormerMask2Former [8] is an architecture capable of addressing any image segmentationtask, with Swin-L [31] being its best performing backbone which we adopt for ourevaluation. We evaluate CuMe against Expedite on both panoptic and instance seg-mentation tasks using the COCO dataset, with results presented in Figures 5.4 and 5.5.We conduct experiments at two layers: the layer from Expedite’s recommendedsettings and at the first layer of Swin-L’s deepest stage (𝑙 = 4).Our method consistently outperforms Expedite and ToMe, often by significant41Method mIOU Speedup GFLOPsCityscapesBaseline 76.54 1.00× 115.97ALGM 75.24 1.75× 61.56CuMe 75.44 1.71× 65.51Pascal ContextBaseline 53.01 1.00× 32.09ALGM 52.97 1.35× 22.28CuMe 52.95 1.27× 23.43Table 5.3: Segmentation results on Segmenter(ViT-S), comparing CuMeagainst ALGM on Cityscapes and Pascal Contextmargins, across all experiments conducted in this section, while showing superiorconsistency across different layers.5.3 Experiments on Non-Spatial ArchitecturesTo enable comparison against a broader range of existing token reduction methods,we extend our evaluation to vanilla Vision Transformers without spatial components.5.3.1 Prior Work ComparisonWe select models based on their established compatibility with existing tokenreduction methods: DeiT [47] for its foundational role and widespread adoption intoken reduction literature, and Segmenter [46] to enable direct comparison withALGM [37], a method specifically designed for segmentation tasks.We evaluate CubistMerge under both off-the-shelf and fine-tuning configurationsto demonstrate versatility across different deployment scenarios. For classifica-tion tasks, we use DeiT-B and compare against five established token reductionmethods: ToMe [3], PiToMe [48], ToFu [18], MCTF [22] and DynamicViT [41].For segmentation tasks, we use Segmenter [46] to enable direct comparison withALGM [37].DeiTWe evaluate on DeiT-B compared against ToMe [3], PiToMe [48], ToFu [18],MCTF [22] and DynamicViT [41]. We evaluate both off-the-shelf and with fine-42Fine-tuned ResultsMethod Top1 Speedup GFLOPsBaseline 81.80 1.00× 17.58ToMe 81.69 1.11× 15.33PiToMe 81.66 1.10× 14.94ToFu 81.59 1.18× 14.89DyViT0.9 81.83 1.10× 15.53CuMe 81.82 1.15× 15.31MCTF 80.96 1.34× 9.93DyViT0.7 81.44 1.33× 11.49CuMe 81.38 1.32× 13.23Table 5.4: Fine-tuned results on DeiT-B, within 5 epochs.tuning limited to 5 epochs. Figure 5.7 shows results using existing methods’recommended merging schedules and CuMe applied at 𝑙 = 1. Fine-tuned results areshown in Table 5.4. CuMe achieves state-of-the-art results with no accuracy loss at1.15× speedup within just one epoch of fine-tuning, while maintaining competitiveperformance across higher speedup ratios.Interestingly, CuMe demonstrates superior speedups despite having slightlyhigher GFLOPs in some settings. We attribute this to computational overheadnot captured by GFLOPs calculations: existing methods require attention scalingand token size tracking during inference, while our max-magnitude-per-dimensionapproach (Section 4.3) eliminates these overheads entirely. This observation issupported by our finding that fvcore GFLOPs measurements remain identicalwhether attention scaling is enabled or disabled, indicating that such runtimeoverheads are not reflected in theoretical GFLOP counts.SegmenterWe evaluate on Segmenter [46] with ViT-S backbone compared against ALGM [37]on Cityscapes and Pascal Context datasets, applying both methods off-the-shelf.We adopt a merging schedule similar to ALGM’s configuration. However, ALGMuses an adaptive method that automatically determines a similarity threshold fortoken merging. We cannot directly adopt this approach because our method requiresmaintaining a structured token layout with consistent token counts per row andcolumn, precluding flexible reduction rates that would break spatial coherence43(discussed further in Section 6.3). Instead, we apply CuMe at the same layers asALGM (𝑙 = 1 and 𝑙 = 5) with the same 𝑟ℎ and 𝑟𝑤 values at both layers. Results areshown in Table 5.3.Results demonstrate that CuMe achieves competitive performance against thisbroader range of existing methods on non-spatial architectures as well.5.4 DiscussionOur experimental evaluation validates two core insights underlying CubistMerge’sdesign. First, maintaining spatially coherent structured layouts is essential for spatialarchitectures, as demonstrated by ToMe’s poor performance in spatial architectureexperiments. Second, selectively exploiting spatial information density unevennesspreserves information better than uniform approaches, evidenced by CubistMerge’sconsistent outperformance of Expedite across all spatial architectures. These findingsconfirm that effective token reduction for spatial architectures requires reconcilingboth spatial structure preservation and selective information density exploitation.44(a) Layer sensitivity analysis by applying CuMe and Expedite at 7 differentlayers of SAM-H evaluated on COCO(b) Full evaluation on COCO, applying CuMe at layer 𝑙 = 0 and Expedite at𝑙 = 6 and 𝑙 = 16(c) Full evaluation on ADE20K, applying CuMe at layer 𝑙 = 0 and Expedite at𝑙 = 6 and 𝑙 = 16Figure 5.3: Instance segmentation results on SAM.45Figure 5.4: Panoptic segmentation performance on Mask2Former(Swin-L),varying 𝑟ℎ = 𝑟𝑤 = 1, 2, 3Figure 5.5: Instance segmentation performance on Mask2Former(Swin-L),varying 𝑟ℎ = 𝑟𝑤 = 1, 2, 346Figure 5.6: Instance segmentation results on SAM2-L evaluated on COCOand ADE20K datasets, varying 𝑟ℎ = 𝑟𝑤 = 4, 8, 12 with CuMe and ToMeat 𝑙 = 9, and Expedite at 𝑙 = 15.47Figure 5.7: Image classification results on DeiT-B compared against priortoken reduction methods.48Chapter 6ConclusionThis chapter concludes the thesis by summarizing the key findings and contributionsof CubistMerge. We begin with a comprehensive summary of our method’s develop-ment and experimental validation across diverse Vision Transformer architecturesand tasks. Following this summary, we discuss the current limitations of our approach.Finally, we outline promising directions for future research that could extend theimpact and applicability of spatial-aware token reduction methods.6.1 SummaryThis thesis introduced CubistMerge, a novel token reduction method that addresses thefundamental limitation of existing approaches in handling spatial vision transformerarchitectures. While previous token reduction techniques [3, 20, 28, 41] primarilyfocused on vanilla Vision Transformers and often disrupted spatial relationshipscritical for architectures with spatial designs [19, 25, 26, 31, 45], we developeda graph-based merging strategy that preserves spatial structure while exploitinguneven information density across the spatial grid.Through comprehensive evaluation across diverse architectures and vision tasks,we demonstrated the effectiveness and broad applicability of our method. Ourexperiments spanned multiple spatial architectural families, including models withdifferent attention mechanisms such as shifted window attention [31] and window-global attention [19, 25, 26, 42], as well as various positional embedding strategies49including decomposed relative positional embeddings [26] and RoPE [45]. Weevaluated performance across four major computer vision tasks: image classification,object detection, instance segmentation, and panoptic segmentation, using standarddatasets including ImageNet, COCO, ADE20K, Cityscapes, and Pascal Context.Our extensive comparisons with existing methods, particularly Expedite [27]as the primary spatial-compatible baseline, revealed several key advantages ofCubistMerge. Compared to Expedite, our method consistently achieved superiorperformance across all tested architectures and tasks while demonstrating betterrobustness to layer selection, maintaining stable performance when applied atdifferent depths within the network. Additionally, our evaluation on non-spatialarchitectures showed that CubistMerge remains competitive with established tokenreduction methods [3, 18, 22, 41, 48], demonstrating its versatility across the fullspectrum of Vision Transformer designs.6.2 DiscussionThe central contribution of this work lies in reconciling two requirements thatprevious approaches failed to achieve simultaneously: preserving spatial structurewhile exploiting information density unevenness. This reconciliation reveals abroader principle applicable beyond token reduction, demonstrating how structuredrepresentations and dynamic, feature-driven adaptation can coexist.Most existing token reduction methods face a fundamental dilemma. Methodslike ToMe [3] achieve strong efficiency gains by freely merging tokens with globalmatching, but this flexibility comes at the cost of spatial coherence, producingirregular token distributions incompatible with spatial architectures. Conversely,Expedite [27] maintains spatial structure through uniform cluster initialization acrossthe entire spatial grid, but this structural rigidity forces distinct tokens into clustersregardless of their semantic distinctiveness, causing information loss in semanticallyunique regions.CubistMerge resolves this dilemma through a key insight: spatial structure anddynamic, feature-driven reduction are not conflicting objectives when reductionis strategically constrained through two complementary mechanisms: enforcingstructured 2D spatial layouts and restricting merging to spatially adjacent neighbors.50Rather than choosing between global flexibility and rigid uniformity, our approachconstrains merging to maintain spatial relationships while preserving the freedom toselectively leave distinct tokens untouched, which enables allocating the reductionbudget to match information density across different regions by applying moreaggressive reduction to areas with lower information density while preserving moretokens from semantically unique regions.This principle of mixing structured and unstructured approaches extends beyondtoken reduction to other efficiency techniques in deep learning. Several directionsmight potentially warrant consideration for future exploration. First, the concept ofselectively reducing tokens along spatial dimensions while maintaining structuralcoherence could prove relevant to token pooling or downsampling operations inmultiscale architectures, where maintaining spatial relationships across differentresolution levels remains important. Second, extending our spatial-aware reductionstrategy beyond 2D spatial dimensions to incorporate temporal dimensions presentsan intriguing possibility for video understanding tasks, where the same tensionbetween preserving temporal structure and exploiting redundancy across framesnaturally arises.6.3 LimitationsWhile CubistMerge demonstrates strong performance across diverse architecturesand tasks, several limitations warrant consideration.First, our structured merging approach imposes constraints on the minimumreduction granularity. Given an 𝐻 ×𝑊 token grid, our sequential 2D reductionrequires removing at least min(𝐻,𝑊) tokens to maintain spatial coherence acrossboth dimensions. This contrasts with more flexible methods like ToMe [3], whichcan apply arbitrary merging schedules without dimensional constraints, potentiallyachieving higher reduction rates in scenarios where spatial structure preservationis not critical. However, our experiments on non-spatial architectures (Section 5.3)demonstrate that this constraint does not significantly impact performance, asCubistMerge performs competitively against ToMe despite ToMe’s schedulingflexibility.Second, our sequential row-wise and column-wise reduction operations require51tokens to be organized in their proper 2D spatial arrangement. Since CubistMergeoperates by reducing tokens within each row and then within each column, tokensmust be spatially arranged to ensure that the correct neighboring tokens are groupedtogether during merging. This necessitates token reorganization even for architectureslike DeiT [47] that do not require spatial structure, introducing computationaloverhead that may be unnecessary for non-spatial models.6.4 Future WorkSeveral promising directions emerge from this work that could extend the impactand applicability of spatial-aware token reduction methods.The most significant opportunities involve exploring CubistMerge for traininglarge-scale models. Due to hardware resource constraints, our current evaluationwas limited to fine-tuning small classification models with minimal training epochs,preventing us from investigating the method’s potential impact on computationallydemanding training scenarios.CubistMerge presents opportunities for reducing the computational burden oftraining large vision models. By reducing memory requirements, our method couldenable training large models on hardware that would otherwise be unable to trainsuch models due to limited memory.This would benefit applying CubistMerge during training of large vision models.Task-specific training of massive pretrained backbones such as DINOv3 [45]represents a relevant scenario: these models are primarily released as backbonenetworks that require extensive additional training for specific applications. Suchtraining demands significantly more time and hardware resources than simplefine-tuning, requiring many more GPU hours, yet is essential for adapting thesepowerful pretrained models to specific downstream tasks. Training from scratch withCubistMerge integrated throughout the complete training process would providesimilar benefits, offering insights into how effective our spatial-aware token reductionapproach is when models must learn from the ground up rather than adapting pre-existing features.52Bibliography[1] S. R. Alvar, G. Singh, M. Akbari, and Y. Zhang. Divprune: Diversity-basedvisual token pruning for large multimodal models. In IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN,USA, June 11-15, 2025, pages 9392–9401, 2025. → page 20[2] D. Bolya, C. Fu, X. Dai, P. Zhang, and J. Hoffman. Hydra attention: Efficientattention with many heads. In L. Karlinsky, T. Michaeli, and K. Nishino,editors, Computer Vision - ECCV 2022 Workshops - Tel Aviv, Israel, October23-27, 2022, Proceedings, Part VII, volume 13807 of Lecture Notes inComputer Science, pages 35–49. Springer, 2022. → page 2[3] D. Bolya, C. Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman. Tokenmerging: Your vit but faster. In The Eleventh International Conference onLearning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.→ pages x, xi, xiii, 2, 4, 18, 22, 25, 26, 27, 29, 30, 36, 42, 49, 50, 51[4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal,A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss,G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu,C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess,J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.Language models are few-shot learners. In H. Larochelle, M. Ranzato,R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural InformationProcessing Systems 33: Annual Conference on Neural Information ProcessingSystems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. → page 1[5] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko.End-to-end object detection with transformers. In A. Vedaldi, H. Bischof,T. Brox, and J. Frahm, editors, Computer Vision - ECCV 2020 - 16thEuropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I,volume 12346 of Lecture Notes in Computer Science, pages 213–229.Springer, 2020. → pages x, xiii, 1, 11, 1253[6] M. Caron, H. Touvron, I. Misra, H. Je´gou, J. Mairal, P. Bojanowski, andA. Joulin. Emerging properties in self-supervised vision transformers. In 2021IEEE/CVF International Conference on Computer Vision, ICCV 2021,Montreal, QC, Canada, October 10-17, 2021, pages 9630–9640. IEEE, 2021.→ page 1[7] M. Chen, W. Shao, P. Xu, M. Lin, K. Zhang, F. Chao, R. Ji, Y. Qiao, and P. Luo.Diffrate : Differentiable compression rate for efficient vision transformers. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris,France, October 1-6, 2023, pages 17118–17128, 2023. → pages 4, 18[8] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar.Masked-attention mask transformer for universal image segmentation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR2022, New Orleans, LA, USA, June 18-24, 2022, pages 1280–1289, 2022. →pages 1, 35, 41[9] J. Choi, S. Lee, J. Chu, M. Choi, and H. J. Kim. vid-tldr: Training free tokenmerging for light-weight video transformer. In IEEE/CVF Conference onComputer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA,June 16-22, 2024, pages 18771–18781, 2024. → page 20[10] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deepbidirectional transformers for language understanding. In J. Burstein,C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of theNorth American Chapter of the Association for Computational Linguistics:Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA,June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186.Association for Computational Linguistics, 2019. → page 1[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai,T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit,and N. Houlsby. An image is worth 16x16 words: Transformers for imagerecognition at scale. In 9th International Conference on LearningRepresentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. →pages x, 1, 2, 3, 7, 9, 10[12] B. Duke, A. Ahmed, C. Wolf, P. Aarabi, and G. W. Taylor. SSTVOS: sparsespatiotemporal transformers for video object segmentation. In IEEEConference on Computer Vision and Pattern Recognition, CVPR 2021, virtual,June 19-25, 2021, pages 5912–5921. Computer Vision Foundation / IEEE,2021. → page 154[13] K. He, X. Chen, S. Xie, Y. Li, P. Dolla´r, and R. B. Girshick. Maskedautoencoders are scalable vision learners. In IEEE/CVF Conference onComputer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA,USA, June 18-24, 2022, pages 15979–15988, 2022. → pages 1, 3, 11[14] B. Heo, S. Park, D. Han, and S. Yun. Rotary position embedding for visiontransformer. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler,and G. Varol, editors, Computer Vision - ECCV 2024 - 18th EuropeanConference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part X,volume 15068 of Lecture Notes in Computer Science, pages 289–305, 2024.→ pages 3, 15, 25, 31[15] T. Hu, L. Li, J. van de Weijer, H. Gao, F. S. Khan, J. Yang, M. Cheng,K. Wang, and Y. Wang. Token merging for training-free semantic binding intext-to-image synthesis. In A. Globersons, L. Mackey, D. Belgrave, A. Fan,U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in NeuralInformation Processing Systems 38: Annual Conference on NeuralInformation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada,December 10 - 15, 2024, 2024. → page 20[16] H. Huang, W. Chai, K. Chen, C. Yang, and J. Hwang. Tosa: Token mergingwith spatial awareness. CoRR, abs/2506.20066, 2025. → pages 5, 19[17] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns:Fast autoregressive transformers with linear attention. In Proceedings of the37th International Conference on Machine Learning, ICML 2020, 13-18 July2020, Virtual Event, volume 119 of Proceedings of Machine LearningResearch, pages 5156–5165. PMLR, 2020. → page 2[18] M. Kim, S. Gao, Y. Hsu, Y. Shen, and H. Jin. Token fusion: Bridging the gapbetween token pruning and token merging. In IEEE/CVF Winter Conferenceon Applications of Computer Vision, WACV 2024, Waikoloa, HI, USA,January 3-8, 2024, pages 1372–1381, 2024. → pages 4, 19, 42, 50[19] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao,S. Whitehead, A. C. Berg, W. Lo, P. Dolla´r, and R. B. Girshick. Segmentanything. In IEEE/CVF International Conference on Computer Vision, ICCV2023, Paris, France, October 1-6, 2023, pages 3992–4003, 2023. → pagesxiii, 1, 3, 14, 15, 16, 35, 40, 49[20] Z. Kong, P. Dong, X. Ma, X. Meng, W. Niu, M. Sun, X. Shen, G. Yuan,B. Ren, H. Tang, M. Qin, and Y. Wang. Spvit: Enabling faster vision55transformers via latency-aware soft token pruning. In S. Avidan, G. J.Brostow, M. Cisse´, G. M. Farinella, and T. Hassner, editors, Computer Vision -ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27,2022, Proceedings, Part XI, volume 13671 of Lecture Notes in ComputerScience, pages 620–640, 2022. → pages 2, 3, 4, 16, 17, 49[21] D. H. Lee and S. Hong. Learning to merge tokens via decoupled embeddingfor efficient vision transformers. In A. Globersons, L. Mackey, D. Belgrave,A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in NeuralInformation Processing Systems 38: Annual Conference on NeuralInformation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada,December 10 - 15, 2024, 2024. → pages 3, 17[22] S. Lee, J. Choi, and H. J. Kim. Multi-criteria token fusion with one-step-aheadattention for efficient vision transformers. In IEEE/CVF Conference onComputer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA,June 16-22, 2024, pages 15741–15750, 2024. → pages 4, 19, 42, 50[23] S. Lee, J. Wang, Z. Zhang, D. Fan, and X. Li. Video token merging for longvideo understanding. In A. Globersons, L. Mackey, D. Belgrave, A. Fan,U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in NeuralInformation Processing Systems 38: Annual Conference on NeuralInformation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada,December 10 - 15, 2024, 2024. → page 20[24] J. Li, Y. Wang, X. Zhang, B. Shi, D. Jiang, C. Li, W. Dai, H. Xiong, andQ. Tian. Ailurus: A scalable vit framework for dense prediction. In A. Oh,T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems 36: Annual Conferenceon Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,LA, USA, December 10 - 16, 2023, 2023. → pages 20, 40, 41[25] Y. Li, H. Mao, R. B. Girshick, and K. He. Exploring plain vision transformerbackbones for object detection. In S. Avidan, G. J. Brostow, M. Cisse´, G. M.Farinella, and T. Hassner, editors, Computer Vision - ECCV 2022 - 17thEuropean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings,Part IX, volume 13669 of Lecture Notes in Computer Science, pages 280–296,2022. → pages xiii, 1, 3, 14, 15, 16, 17, 31, 35, 40, 49[26] Y. Li, C. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer.Mvitv2: Improved multiscale vision transformers for classification anddetection. In IEEE/CVF Conference on Computer Vision and Pattern56Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages4794–4804, 2022. → pages 3, 13, 14, 16, 17, 25, 31, 35, 36, 40, 49, 50[27] W. Liang, Y. Yuan, H. Ding, X. Luo, W. Lin, D. Jia, Z. Zhang, C. Zhang, andH. Hu. Expediting large-scale vision transformer for dense prediction withoutfine-tuning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, andA. Oh, editors, Advances in Neural Information Processing Systems 35:Annual Conference on Neural Information Processing Systems 2022, NeurIPS2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. →pages x, 2, 4, 5, 6, 19, 22, 31, 36, 41, 50[28] Y. Liang, C. Ge, Z. Tong, Y. Song, J. Wang, and P. Xie. Not all patches arewhat you need: Expediting vision transformers via token reorganizations.CoRR, abs/2202.07800, 2022. → pages 2, 3, 4, 16, 17, 49[29] D. Liu, M. Kan, S. Shan, and X. Chen. A simple romance between multi-exitvision transformer and token reduction. In The Twelfth InternationalConference on Learning Representations, ICLR 2024, Vienna, Austria, May7-11, 2024, 2024. → page 17[30] Y. Liu, M. Gehrig, N. Messikommer, M. Cannici, and D. Scaramuzza.Revisiting token pruning for object detection and instance segmentation. InIEEE/CVF Winter Conference on Applications of Computer Vision, WACV2024, Waikoloa, HI, USA, January 3-8, 2024, pages 2646–2656, 2024. →pages 3, 17[31] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swintransformer: Hierarchical vision transformer using shifted windows. In 2021IEEE/CVF International Conference on Computer Vision, ICCV 2021,Montreal, QC, Canada, October 10-17, 2021, pages 9992–10002, 2021. →pages 2, 3, 13, 16, 31, 41, 49[32] S. Long, Z. Zhao, J. Pi, S. Wang, and J. Wang. Beyond attentive tokens:Incorporating token importance and diversity for efficient vision transformers.In IEEE/CVF Conference on Computer Vision and Pattern Recognition,CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 10334–10343,2023. → pages 3, 4, 17, 19[33] C. Lu, D. de Geus, and G. Dubbelman. Content-aware token sharing forefficient semantic segmentation with vision transformers. In IEEE/CVFConference on Computer Vision and Pattern Recognition, CVPR 2023,Vancouver, BC, Canada, June 17-24, 2023, pages 23631–23640, 2023. →pages 3, 17, 2057[34] L. Meng, H. Li, B. Chen, S. Lan, Z. Wu, Y. Jiang, and S. Lim. Adavit:Adaptive vision transformers for efficient image recognition. In IEEE/CVFConference on Computer Vision and Pattern Recognition, CVPR 2022, NewOrleans, LA, USA, June 18-24, 2022, pages 12299–12308. IEEE, 2022. →page 2[35] Meta Research. fvcore, 2023. https://github.com/facebookresearch/fvcore. →page 34[36] P. Michel, O. Levy, and G. Neubig. Are sixteen heads really better than one?In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alche´-Buc, E. B. Fox,and R. Garnett, editors, Advances in Neural Information Processing Systems32: Annual Conference on Neural Information Processing Systems 2019,NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages14014–14024, 2019. → page 2[37] N. Norouzi, S. Orlova, D. de Geus, and G. Dubbelman. ALGM: adaptivelocal-then-global token merging for efficient semantic segmentation with plainvision transformers. In IEEE/CVF Conference on Computer Vision andPattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages15773–15782, 2024. → pages 4, 5, 6, 18, 19, 20, 42, 43[38] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov,P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas,W. Galuba, R. Howes, P. Huang, S. Li, I. Misra, M. Rabbat, V. Sharma,G. Synnaeve, H. Xu, H. Je´gou, J. Mairal, P. Labatut, A. Joulin, andP. Bojanowski. Dinov2: Learning robust visual features without supervision.Trans. Mach. Learn. Res., 2024, 2024. → page 1[39] PyTorch Contributors. torch.scatter reduce — PyTorch 2.8 Documentation.PyTorch Project. URL https://docs.pytorch.org/docs/stable/generated/torch.Tensor.scatter reduce .html. →page 27[40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry,A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learningtransferable visual models from natural language supervision. In M. Meilaand T. Zhang, editors, Proceedings of the 38th International Conference onMachine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139of Proceedings of Machine Learning Research, pages 8748–8763. PMLR,2021. → page 158[41] Y. Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C. Hsieh. Dynamicvit: Efficientvision transformers with dynamic token sparsification. pages 13937–13949,2021. → pages 2, 3, 4, 16, 17, 42, 49, 50[42] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Ra¨dle,C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C. Wu,R. B. Girshick, P. Dolla´r, and C. Feichtenhofer. SAM 2: Segment anything inimages and videos. In The Thirteenth International Conference on LearningRepresentations, ICLR 2025, Singapore, April 24-28, 2025, 2025. → pages1, 35, 41, 49[43] C. Ryali, Y. Hu, D. Bolya, C. Wei, H. Fan, P. Huang, V. Aggarwal,A. Chowdhury, O. Poursaeed, J. Hoffman, J. Malik, Y. Li, andC. Feichtenhofer. Hiera: A hierarchical vision transformer without thebells-and-whistles. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt,S. Sabato, and J. Scarlett, editors, International Conference on MachineLearning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202of Proceedings of Machine Learning Research, pages 29441–29454, 2023. →pages 1, 17, 41[44] L. Shen, T. Hao, T. He, S. Zhao, Y. Zhang, P. Liu, Y. Bao, and G. Ding.Tempme: Video temporal token merging for efficient text-video retrieval. InThe Thirteenth International Conference on Learning Representations, ICLR2025, Singapore, April 24-28, 2025, 2025. → page 20[45] O. Sime´oni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose,V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haziza,L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts,A. Vedaldi, J. Tolan, J. Brandt, C. Couprie, J. Mairal, H. Je´gou, P. Labatut, andP. Bojanowski. Dinov3, 2025. → pages 1, 3, 15, 16, 35, 38, 49, 50, 52[46] R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer forsemantic segmentation. In 2021 IEEE/CVF International Conference onComputer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021,pages 7242–7252, 2021. → pages 1, 11, 42, 43[47] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Je´gou.Training data-efficient image transformers & distillation through attention. InM. Meila and T. Zhang, editors, Proceedings of the 38th InternationalConference on Machine Learning, ICML 2021, 18-24 July 2021, VirtualEvent, volume 139 of Proceedings of Machine Learning Research, pages10347–10357, 2021. → pages xiii, 1, 11, 42, 5259[48] C. Tran, D. M. H. Nguyen, M. Nguyen, T. Nguyen, N. Le, P. Xie, D. Sonntag,J. Y. Zou, B. Nguyen, and M. Niepert. Accelerating transformers withspectrum-preserving token merging. In A. Globersons, L. Mackey,D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems 38: Annual Conferenceon Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver,BC, Canada, December 10 - 15, 2024, 2024. → pages 4, 19, 42, 50[49] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. vonLuxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, andR. Garnett, editors, Advances in Neural Information Processing Systems 30:Annual Conference on Neural Information Processing Systems 2017,December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017. →pages x, 1, 7, 8, 9[50] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. Analyzingmulti-head self-attention: Specialized heads do the heavy lifting, the rest canbe pruned. In A. Korhonen, D. R. Traum, and L. Ma`rquez, editors,Proceedings of the 57th Conference of the Association for ComputationalLinguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1:Long Papers, pages 5797–5808. Association for Computational Linguistics,2019. → page 2[51] F. Wang, Y. Yu, G. Wei, W. Shao, Y. Zhou, A. L. Yuille, and C. Xie. Scalinglaws in patchification: An image is worth 50,176 tokens and more. CoRR,abs/2502.03738, 2025. → page 25[52] X. Xu, S. Wang, Y. Chen, Y. Zheng, Z. Wei, and J. Liu. Gtp-vit: Efficientvision transformers via graph-based token propagation. In IEEE/CVF WinterConference on Applications of Computer Vision, WACV 2024, Waikoloa, HI,USA, January 3-8, 2024, pages 86–95, 2024. → pages 3, 17[53] X. Ye, Y. Gan, Y. Ge, X. Zhang, and Y. Tang. Atp-llava: Adaptive tokenpruning for large vision language models. In IEEE/CVF Conference onComputer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA,June 11-15, 2025, pages 24972–24982, 2025. → page 20[54] B. Zhang, L. Liu, M. H. Phan, Z. Tian, C. Shen, and Y. Liu. Segvit v2:Exploring efficient and continual semantic segmentation with plain visiontransformers. Int. J. Comput. Vis., 132(4):1126–1147, 2024. → page 160[55] L. Zhang, J. Lu, S. Zheng, X. Zhao, X. Zhu, Y. Fu, T. Xiang, J. Feng, andP. H. S. Torr. Vision transformers: From semantic segmentation to denseprediction. Int. J. Comput. Vis., 132(12):6142–6162, 2024. → page 161
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Lis, Mieszko
CubistMerge : spatial-preserving token merging for diverse ViT backbones
Text