CubistMerge : spatial-preserving token merging for diverse ViT backbones
department or school: Applied Science, Faculty of
department or school: Electrical and Computer Engineering, Department of
data provider: DSpace
degree campus: UBCV
creator: Gong, Wenyi
issued: 2025-10-16T19:40:40Z
issued: 2025
related degree: Master of Applied Science - MASc
degree grantor: University of British Columbia
description: Many modern Vision Transformer (ViT) backbones adopt spatial architectural designs, such as window attention from Swin and ViTDet, and 2D positional embeddings which includes decomposed relative positional embeddings in SAM and RoPE in DINOv3. Such architectures impose new challenges on token reduction, as the vast majority of existing methods fail to preserve the spatial structure these architectures depend on. In this thesis, we introduce a simple yet effective token merging method that maintains spatial integrity, enabling seamless compatibility with spatial architectures. We reconcile two seemingly conflicting requirements: (1) exploiting the uneven information distribution across the spatial layout, and (2) preserving the spatial structure post-merging. To enable effective token reduction for spatial architectures, we propose three key innovations: - a 2D reduction strategy to enforce structured token layouts, - a spatial-aware merging algorithm that maintains relative token positions, and - a novel max-magnitude-per-dimension token representation that preserves salient features. Our method demonstrates strong performance both off-the-shelf and with fine-tuning, achieving state-of-the-art results on spatial and non-spatial architectures across various vision tasks. Specifically, we achieve 1.25× speedup on SAM-H with only 0.7% mIOU drop evaluated on COCO off-the-shelf, and 1.15× speedup on DeiT-B with no top-1 accuracy drop on ImageNet within just one epoch of fine-tuning.
aggregated CHO: https://circle.library.ubc.ca/rest/handle/2429/92643
note: CubistMerge: Spatial-Preserving Token Merging ForDiverse ViT BackbonesbyWenyi Gonga thesis submitted in partial fulfillmentof the requirements for the degree ofMaster of Applied Scienceinthe faculty of graduate and postdoctoral studies(Electrical and Computer Engineering)The University of British Columbia(Vancouver)October 2025© Wenyi Gong, 2025The following individuals certify that they have read, and recommend to the Facultyof Graduate and Postdoctoral Studies for acceptance, the thesis entitled:CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Back-bonessubmitted by Wenyi Gong in partial fulfillment of the requirements for the degreeof Master of Applied Science in Electrical and Computer Engineering.Examining Committee:Mieszko Lis, Associate Professor, Electrical and Computer Engineering, UBCSupervisorRenjie Liao, Assistant Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberPrashant Nair, Assistant Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberiiAbstractMany modern Vision Transformer (ViT) backbones adopt spatial architecturaldesigns, such as window attention from Swin and ViTDet, and 2D positionalembeddings which includes decomposed relative positional embeddings in SAMand RoPE in DINOv3. Such architectures impose new challenges on token reduction,as the vast majority of existing methods fail to preserve the spatial structure thesearchitectures depend on.In this thesis, we introduce a simple yet effective token merging method thatmaintains spatial integrity, enabling seamless compatibility with spatial architectures.We reconcile two seemingly conflicting requirements: (1) exploiting the uneveninformation distribution across the spatial layout, and (2) preserving the spatialstructure post-merging.To enable effective token reduction for spatial architectures, we propose threekey innovations:• a 2D reduction strategy to enforce structured token layouts,• a spatial-aware merging algorithm that …
has type: Thesis/Dissertation
date issued: 2025-11
is shown at: 10.14288/1.0450473
language: eng
degree discipline: Electrical and Computer Engineering
provider: Vancouver : University of British Columbia Library
publisher: University of British Columbia
rights: Attribution-NonCommercial-NoDerivatives 4.0 International
rights URI: http://creativecommons.org/licenses/by-nc-nd/4.0/
scholar level: Graduate
contributor: Lis, Mieszko
type: Text
identifier URI: http://hdl.handle.net/2429/92643