Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Sun, Weiwei
Doctor of Philosophy - PhD
University of British Columbia
We are interested in processing point clouds -- a set of unordered points -- specifically in Euclidean space, such as 3D point cloud acquired from a range sensor (LiDAR) or 4D correspondence cloud in stereo matching task. Point clouds play an increasingly essential role in many tasks due to prevalence they hold. However, it is notoriously challenging to process point clouds with deep neural networks because of their irregular data structure, the difficulty in encoding contextual information from nearby points, and the large compute requirement that is typically required. This thesis addresses these challenges by enforcing intermediate features or model parameters to carry specific meanings such as attention and poses, leading to explicit representation. The meanings of explicit representation allow for traditional ways of manipulating features in order to solve target tasks. We refer to these architectures with explicit representations as explicit models. Explicit models largely improve performances without massively scaling up training data or model size because the explicit representation directly injects the prior knowledge needed by target tasks into neural networks without any learning. We explore explicit models for point cloud learning to perform robust estimation, stereo matching, segmentation, reconstruction and neural rendering. The thesis is organized into four chapters: 1, ACNe: An optimization-inspired network architecture that allows learning with point clouds contaminated with an abundance of outliers. 2, Canonical Capsules: An equivariant latent representation that consists of pose and pose-invariant features, enabling point cloud auto-encoding in unaligned datasets. 3, NeuralBF: A novel 3D instance proposal generation inspired by traditional bilateral filtering for top-down instance segmentation for 3D point clouds. 4, PointNeRF++: A multi-scale, point-based NeRF architecture, allowing seamless integration of point-based representation with Neural Radiance Fields. Across these four chapters, we show that explicit models largely improve point cloud learning, inspiring more future research in this domain. We conclude with a discussion about future works, practical tips on how to form an explicit model, and its role in the era of large foundation models.
Exploring Explicit Models forGeometric Point Cloud LearningbyWeiwei SunA THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)October 2024© Weiwei Sun 2024The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:Exploring Explicit Models for Geometric Point Cloud Learningsubmitted by Weiwei Sun in partial fulfillment of the requirements for thedegree of Doctor of Philosophy in Computer Science.Examining Committee:Kwang Moo Yi, Assistant Professor, Department of Computer Science, TheUniversity of British ColumbiaSupervisorAndrea Tagliasacchi, Associate Professor, Department of Computer Science,Simon Fraser UniversityCo-SupervisorIan Mitchell, Professor, Department of Computer Science, The Universityof British ColumbiaUniversity ExaminerNicholas Coops, Professor, Department of Forest Resources Management,The University of British ColumbiaUniversity ExaminerAndrew Markham, Professor, Department of Computer Science, Universityof OxfordExternal ExaminerAdditional Supervisory Committee Members:Helge Rhodin, Assistant Professor, Department of Computer Science, TheUniversity of British ColumbiaSupervisory Committee MemberiiAbstractWe are interested in processing point clouds – a set of unordered points –specifically in Euclidean space, such as 3D point cloud acquired from a rangesensor (LiDAR) or 4D correspondence cloud in stereo matching task. Pointclouds play an increasingly essential role in many tasks due to prevalencethey hold. However, it is notoriously challenging to process point cloudswith deep neural networks because of their irregular data structure, thedifficulty in encoding contextual information from nearby points, and thelarge compute requirement that is typically required.This thesis addresses these challenges by enforcing intermediate features ormodel parameters to carry specific meanings such as attention and poses,leading to explicit representation. The meanings of explicit representationallow for traditional ways of manipulating features in order to solve tar-get tasks. We refer to these architectures with explicit representations asexplicit models. Explicit models largely improve performances without mas-sively scaling up training data or model size because the explicit representa-tion directly injects the prior knowledge needed by target tasks into neuralnetworks without any learning. We explore explicit models for point cloudlearning to perform robust estimation, stereo matching, segmentation, re-construction and neural rendering. The thesis is organized into four chap-ters: 1, ACNe: An optimization-inspired network architecture that allowslearning with point clouds contaminated with an abundance of outliers. 2,Canonical Capsules: An equivariant latent representation that consistsof pose and pose-invariant features, enabling point cloud auto-encoding inunaligned datasets. 3, NeuralBF: A novel 3D instance proposal generationinspired by traditional bilateral filtering for top-down instance segmenta-tion for 3D point clouds. 4, PointNeRF++: A multi-scale, point-basedNeRF architecture, allowing seamless integration of point-based representa-tion with Neural Radiance Fields.Across these four chapters, we show that explicit models largely improvepoint cloud learning, inspiring more future research in this domain. WeiiiAbstractconclude with a discussion about future works, practical tips on how toform an explicit model, and its role in the era of large foundation models.ivLay SummaryVarious real-world data is represented in terms of point clouds, for exam-ple, LiDAR scans and even correspondences across images – anything thatallows data to be represented as a point in some space is a point cloud.However, because of the flexibility of what can be a point cloud, processingit is nontrivial. Many studies treat point clouds like 3D images by usingcubes, similar to Minecraft, but this approach is too abstract and not ef-fective. Deep learning for point clouds remains a challenge. In this thesis,we explore models that use explicit representations with physical meaningas prior knowledge. This helps make deep networks less opaque. We showthat models using this prior knowledge perform better in various real-worldtasks, including stereo matching, point cloud understanding, and neuralrendering. Overall, explicit representation plays a crucial role in improvingdeep learning for point clouds.vPrefaceThe thesis extends four publications as below, with each of them exploringdifferent tasks. These publications are the results of close collaborationsbetween the authors listed under each work. I, as the first author, imple-mented the research ideas and experiments. The last authors Kwang MooYi and Andrea Tagliasacchi are the advisors providing guidance. All authorscontribute to the writing of papers and regular discussions of projects.We list the publications below with details including the publication venue,authors, and authors’ special contributions except for the writing and guid-ance. Note ∗ denotes the equal contribution.1. Chapter 3: “Acne: Attentive Context Normalization for Robust Permutation-equivariant Learning”.Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, and KwangMoo YiProject website: https://acne.github.ioVenue: IEEE Computer Vision and Pattern Recognition Conference.2020Additional detail on the contributions: Kwang Moo Yi helped preparethe experiments for line estimation.2. Chapter 4:“Canonical Capsules: Self-supervised Capsules in CanonicalPose”.Weiwei Sun∗, Andrea Tagliasacchi∗, Boyang Deng, Sara Sabour, SorooshYazdani, Geoffrey E. Hinton, and Kwang Moo YiProject website: https://canonical-capsules.github.ioVenue: Annual Conference on Neural Information Processing Systems(NeurIPS). 2021Additional detail on the contributions: Boyang Deng wrote the code forthe experiments on unsupervised classification; Andrea Tagliasacchi andBoyang Deng performed preliminary experiments before I led the projectviPreface3. Chapter 5: “NeuralBF: Neural Bilateral Filtering for Top-down InstanceSegmentation on Point Clouds”.Weiwei Sun, Daniel Rebain, Renjie Liao, Vladimir Tankovich, SorooshYazdani, Kwang Moo Yi, and Andrea Tagliasacchi.Project website: https://neuralbf.github.ioVenue: IEEE/CVF Winter Conference on Applications of Computer Vi-sion (WACV). 2023Additional detail on the contributions: Daniel Rebain wrote the code forthe 3D demo and visualization; Soroosh Yazdani helped with the exper-imental design.4. Chapter 6: “PointNeRF++: A multi-scale, Point-based Neural RadianceField”.Weiwei Sun, Eduard Trulls, Yang-Che Tseng, Sneha Sambandam, GopalSharma, Andrea Tagliasacchi, and Kwang Moo Yi.Project website: https://pointnerfpp.github.ioVenue: European Conference on Computer Vision (ECCV). 2024Additional detail on the contributions: Yang-Che Tseng ran the GaussianSplatting baseline for the experiments.viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Thesis overview and contributions . . . . . . . . . . . . . . . 52 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 Processing point clouds by preprocessing them into regularstructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1.1 Multi-view-based methods . . . . . . . . . . . . . . . 72.1.2 Voxel-based methods . . . . . . . . . . . . . . . . . . 92.2 Point-based methods . . . . . . . . . . . . . . . . . . . . . . 102.2.1 Permutation equivariant/invariant layers . . . . . . . 102.2.2 Point convolutions . . . . . . . . . . . . . . . . . . . . 112.2.3 Point Transformer . . . . . . . . . . . . . . . . . . . . 122.3 Explicit models for point clouds . . . . . . . . . . . . . . . . 133 Unrolling Optimization as a Deep Network: ACNe . . . . 143.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 143.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . 17viiiTable of Contents3.4 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.4.1 Attentive Context Normalization . . . . . . . . . . . 193.4.2 Network architecture and applications . . . . . . . . . 213.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.5.1 Implementation details . . . . . . . . . . . . . . . . . 243.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 243.6 More Experiments . . . . . . . . . . . . . . . . . . . . . . . . 323.6.1 Visualizing attention – Fig. 3.5 . . . . . . . . . . . . . 323.6.2 Attention on feature maps – Tab. 3.7 . . . . . . . . . 343.6.3 Essential matrix estimation – Tab. 3.8 . . . . . . . . 353.6.4 Number of ACNs within ARB – Tab. 3.9 . . . . . . . 353.6.5 Number of parameters – Tab. 3.10 . . . . . . . . . . . 353.6.6 Timing of *SAC Methods – Tab. 3.11 . . . . . . . . . 363.6.7 Learnt IRLS variant – Fig. 3.6, Tab. 3.12 . . . . . . . 383.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 Explicit Latent Representation: Canonical Capsules . . . . 404.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 404.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.3 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . 424.4 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.4.1 Losses . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.4.2 Network Architectures . . . . . . . . . . . . . . . . . 474.4.3 More Details of Network Architectures . . . . . . . . 484.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.5.1 Experimental setup . . . . . . . . . . . . . . . . . . . 504.5.2 Auto-encoding – Fig. 4.2 and Tab. 4.1 . . . . . . . . . 514.5.3 Canonicalization – Tab. 4.2 . . . . . . . . . . . . . . . 534.5.4 Unsupervised classification – Tab. 4.3 . . . . . . . . . 554.5.5 Ablation study . . . . . . . . . . . . . . . . . . . . . . 564.6 More Experiments . . . . . . . . . . . . . . . . . . . . . . . . 574.6.1 Additional ablation studies . . . . . . . . . . . . . . . 574.6.2 Per-class results for auto-encoding . . . . . . . . . . . 624.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625 Instance Proposal Generation via Bilateral Filtering: Neu-ralBF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 645.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.3 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . 68ixTable of Contents5.4 Method – Fig. 5.2 . . . . . . . . . . . . . . . . . . . . . . . . 695.4.1 Affinity definition . . . . . . . . . . . . . . . . . . . . 695.4.2 Convex parameterization C(x; f) . . . . . . . . . . . . 715.4.3 Neural Bilateral Filter – Fig. 5.4 . . . . . . . . . . . . 735.4.4 Training . . . . . . . . . . . . . . . . . . . . . . . . . 735.4.5 Implementation details . . . . . . . . . . . . . . . . . 745.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 765.5.1 Synthetic dataset . . . . . . . . . . . . . . . . . . . . 765.5.2 Instance segmentation on ScanNetV2 . . . . . . . . . 795.5.3 Ablations . . . . . . . . . . . . . . . . . . . . . . . . . 815.5.4 Visualization of convex hulls in ScanNet – Fig. 5.11 . 825.5.5 More qualitative results in ScanNet – Fig. 5.12 . . . . 835.5.6 Interactive 3D visualization . . . . . . . . . . . . . . . 835.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 866 Multi-scale Point Rendering: PointNeRF++ . . . . . . . . 876.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 876.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 886.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . 906.4 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 926.4.1 Multi-scale aggregation – A . . . . . . . . . . . . . . 926.4.2 Point-based rendering – R . . . . . . . . . . . . . . . 936.4.3 Per-point tri-plane features . . . . . . . . . . . . . . . 956.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 956.5.1 Implementation Details . . . . . . . . . . . . . . . . . 956.5.2 Experimental setup . . . . . . . . . . . . . . . . . . . 976.5.3 KITTI-360 results – Fig. 6.4 and Sec. 6.5.3 . . . . . . 1006.5.4 ScanNet results – Fig. 6.7 and Tab. 6.3 . . . . . . . . 1016.5.5 NeRF Synthetic results – Fig. 6.6 and Tab. 6.1 . . . . 1036.5.6 Ablation study . . . . . . . . . . . . . . . . . . . . . . 1046.5.7 More rendering results . . . . . . . . . . . . . . . . . 1056.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1067 Conclusion and Outlook . . . . . . . . . . . . . . . . . . . . . . 1077.1 Practical tips for the design of explicit representation . . . . 1087.2 Future works . . . . . . . . . . . . . . . . . . . . . . . . . . . 109Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111xList of Tables2.1 A brief overview of previous works in point cloud learn-ing. Note that we list the representative works for brevity.For more detailed previous works closely related to our targettasks, please refer to the related work section in each chapter. 83.1 Robust line fitting – Line fitting results over the test setin terms of the ℓ2 distance between ground-truth and theestimates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.2 2D Point cloud classification – Classification accuracy onMNIST, under different outlier ratios (%). Our method per-forms best in all cases, and the gap becomes wider with moreoutliers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.3 3D Point cloud classification –We replicate the 3D pointclassification experiment on ModelNet40 from [1], with vanillaPointNet. We then add outliers with Gaussian noise. Our ap-proach performs best with and without outliers. . . . . . . . . 263.4 Pose estimation accuracy – mAP at 10◦/20◦ error thresh-old. Similarly to [2], we consider multiple baselines, as wellas pairing different methods with state-of-the-art RANSACvariants. Our method consistently outperforms all others bya large margin, even without an additional robust estimator,in some cases. . . . . . . . . . . . . . . . . . . . . . . . . . . 303.5 Ablation study – mAP at 20◦ with different CNe [2] andACNe (ours) variants on stereo. The labels indicate: L – localattention; G – global attention; S – local attention supervi-sion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.6 With learned local features –mAP at 20◦ with learned lo-cal features and different methods. Our method outperformsother methods and performs best with SuperPoint [3]. . . . . 32xiList of Tables3.7 Applying attention to the feature maps – We com-pare our method (right) to applying attention directly to thefeature maps (left). Note we use attention variants: L (lo-cal), G (global), and L+G (Both). We report mAP at a20◦ error threshold on our validation set – the Saint Peter’sSquare scene. Applying attention on the normalization per-forms largely better than applying attention to the featuremaps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.8 Essential matrix estimation – mAP at a 20◦ error thresh-old when we train the models to estimate the Essential matrixinstead of the Fundamental matrix, for the indoors and out-doors experiments. . . . . . . . . . . . . . . . . . . . . . . . 363.9 Ablation on #ACN – Performance as we vary the numberof ACNs within each ARB – mAP at 20◦ on our validationset – Saint Peter’s Square. . . . . . . . . . . . . . . . . . . . 363.10 Number of parameters – Our method introduces a smalloverhead compared to CNe, and is much smaller than OANet. 373.11 Average elapsed time – Runtime, in milliseconds, for in-dividual steps of each method. We execute the forward passof our networks on a GTX 1080 Ti GPU and the robust es-timator on a Intel(R) Core(TM) i7-8700 CPU. We disablemulti-threading for the CPU timings since not all robust es-timator implementations support multi-threading. Networksare implemented with TensorFlow 1.8.0, except for OANet,which uses PyTorch 1.2.0. . . . . . . . . . . . . . . . . . . . . 373.12 Performance of learnt IRLS – Comparison with CNe andACNe on the line fitting problem, under an outlier ratio of 0.7. 374.1 Auto-encoding / quantitative – Performance in terms ofChamfer distance with 1024 points per point cloud – metricis multiplied by 103 as in [4]. . . . . . . . . . . . . . . . . . . 524.2 Canonicalization – Quantitative evaluation for canonical-ization, where we highlight massively better performance thanthe concurrent work Compass [5]. While pairwise registrationis not our main objective, the global alignment frame createdby our method still allows for effective registration (on par,or better) than the state-of-the-art. . . . . . . . . . . . . . . 554.3 Classification – Top-1 accuracy(%) . . . . . . . . . . . . . . 554.4 Effect of losses – Reconstruction performance, and canon-icalization performance when loss terms are removed. . . . . 57xiiList of Tables4.5 Backbone – Auto-encoding performance (Chamfer distance)when we use various permutation-invariant backbones. . . . 574.6 Number of points P – Auto-encoding performance (Cham-fer distance × 103) as we vary the input point cloud cardi-nality; aligned setup for both training and testing, with allobject categories. . . . . . . . . . . . . . . . . . . . . . . . . 584.7 Ablation study on the number of capsules – We showthe reconstruction performance (Chamfer distance × 103)with varying number of capsules. While they all performbetter than competitors, 10 capsules give best performance.Note the representation power is kept constant as we vary thenumber of capsules. . . . . . . . . . . . . . . . . . . . . . . . 594.8 Number of capsules w/ fixed descriptor dimensionper capsule – Better reconstruction performance is ob-tained as number of capsules are increased, since the networkcapacity expands proportionally. . . . . . . . . . . . . . . . . 594.9 One-shot canonicalization – Reconstruction performanceof our method against a naive one-shot alignment, where anarbitrary point cloud is selected as reference. . . . . . . . . . 604.10 Canonical descriptors – Auto-encoding performance (Cham-fer Distance) of our method with descriptors βk vs. β¯k. Notethe unaligned setup is the one of primary interest. . . . . . . 604.11 The impact of Lcanonical on canonicalization – Bestcanonicalization is achieved when Lcanonical is used; see textfor details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604.12 Large range of random translations – The performancechanges are negligible as the magnitude of random transla-tions increases. . . . . . . . . . . . . . . . . . . . . . . . . . . 624.13 Auto-encoding / per-class quantitative – Performancein terms of Chamfer distance with 1024 points per point cloud– metric is multiplied by 103 as in [4]. . . . . . . . . . . . . . 625.1 Query-conditioned instance proposal generation – werandomly sample a single query for each instance and gener-ate the non-overlapped proposals. We report the mean andstandard deviation of average precision by running the eval-uation pipeline five times. . . . . . . . . . . . . . . . . . . . 76xiiiList of Tables5.2 Quantitative/ScanNetV2 – instance segmentation bench-mark; our method provides the top performance for the top-down category. For looser thresholds our method performsslightly worse, which may be improved with advanced post-processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 806.1 PSNR↑ on NeRF Synthetic [6] – Our method performsbest overall, even on object-centric data with dense pointclouds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1006.2 Results on KITTI-360 [7] – Our method achieves the bestperformance among methods that supervise only with color.It performs on par with those that also rely on semantics. Weprovide results on the (public) validation set and the (hidden)test set as some baselines have results for one, but not theother. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1016.3 Results on two ScanNet scenes [8] as pre-processedby [9] – Our method outperforms all others, especially Point-NeRF, the method most similar to ours, by a large margindemonstrating the effectiveness of our multi-scale approach. 1026.4 Impact of the tri-plane –We evaluate tri-planes vs. MLPs.Using tri-planes for the coarsest scales improves performance,at a comparable computational cost. . . . . . . . . . . . . . . 1026.5 Using a global feature –Wemeasure PSNR↑ on two datasetsfor different variants of our approach. Adding a global voxelat the coarsest scale to the hierarchical structure (right), im-proves performance (left), but is not sufficient by itself (mid-dle). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046.6 Number of scales vs rendering quality – As expected,more levels lead to better PSNR↑. We always use a globalscale—‘0’ corresponds to using only a global scale, ‘1’ thefinest scale plus the global scale, and later adding coarserscale levels, up to our full model (‘4’). . . . . . . . . . . . . . 1046.7 Number of points – We show that our method remainsapplicable to sparser point clouds, with noticeable improve-ment over points-agnostic model even at drastic downsam-pling rates (1%). . . . . . . . . . . . . . . . . . . . . . . . . . 105xivList of Figures1.1 The prevalence of geometric point cloud: Geometricpoint cloud is a commonplace and important data across do-mains. (First row), point clouds are easily captured with therange sensors like depth sensors and LiDAR, enabling a largerange of applications. (Second row), point cloud capturesgeometry information simply with a set of points and is acommonly-adopted 3D representation [9, 10]. (Thrid row),point cloud generalizes into other tasks with any-dimensionalpoints (2D, 4D and 6D). Notably, in matching tasks, we com-pute putative correspondences –i.e., pairs of 2D image pixelsor 3D points, forming the point cloud in 4D (image matching)or 6D (point cloud registration). . . . . . . . . . . . . . . . . 21.2 Explicit models: In general, a deep neural network is ablack box, which consists of a stack of K layers with param-eters of w and the in-between intermediate features f – bothdon’t need to carry any meaning (i.e., non-interpretable). Inthis thesis, we inject explicit meanings into these featureslike pose and attention, which enables the traditional waysof manipulating the learned representation to solve the tar-get tasks including iterative reweighted least square (ACNe –Ch. 3), equivariant capsule(Canonical capsules–Ch. 4), bilat-eral filtering (NeuralBF–Ch. 5), and points-based representa-tion (PointNeRF++ – Ch. 6). . . . . . . . . . . . . . . . . . 32.1 Three input representations for point cloud learning.We review methods with three input representations includ-ing multi-view, voxel, and points. As detailed in Sec. 2.2,through permutation-equivalence, point-based methods (alsothe focus of this thesis) have no information loss because theydo not need preprocessing steps while multi-view-based [11]and voxel-based methods [12] do. . . . . . . . . . . . . . . . 9xvList of Figures3.1 Robust neural line fitting – We learn to fit lines with out-liers (80%) via our ACNe, as well as CNe [2]. We visualize theground truth and the network estimates. We color-codethe weights learned by the k-th residual layer of ACNe andused to normalize the feature maps – notice that our method,which mimics Iterative Re-weighted Least Squares (IRLS),learns to progressively focus its attention on the inliers. Thisallows ACNe to find the correct solution where CNe fails. . . 163.2 ACNe architecture – (Left) Our permutation-equivariantnetwork receives an input tensor P of size N×D, which isprocessed by a series of K Attentive Residual Blocks (ARB).The output of the network is a tensor O size N×C, which isthen converted to a representation appropriate for the taskat hand. Note that the first perceptron F inφ changes the di-mensionality from P of size N×D (input dimensions) to fea-tures f of size N×C. (Middle) Within each residual path ofthe ARB, we manipulate the feature map with perceptronsFφ with parameters φ, followed by Attentive Context Nor-malization (ACN) – we repeat this structure twice. (Right)An ACN module computes local/global attention with twotrainable networks, combines them via element-wise multipli-cation, and normalizes the feature maps with said weights –the NACN block – followed by Group Normalization. Notethat all features are all processed in the same way, individu-ally, and the ACN block is the only place where they interactwith each other – this architecture guarantees permutation-equivariance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.3 Classification – We add salt-and-pepper noise to MNISTimages, and then convert the digits to an unstructured pointcloud. The % reports the outlier-to-inlier ratio. . . . . . . . 223.4 Wide-baseline stereo – We show the results of differentmatching algorithms on the dataset of [2]. We draw the in-liers produced by them, in green if the match is below theepipolar distance threshold (in red otherwise). Note that thismay include some false positives, as epipolar constraints mappoints to lines – perfect ground truth would require densepixel-to-pixel correspondences. . . . . . . . . . . . . . . . . . 27xviList of Figures3.5 Attention in wide-baseline stereo. An illustration ofACNe on an example image pair. The top row shows thematches, and the bottom row shows a representation of thefeature maps obtained via t-SNE. The left column displaysACN weights, color-coded by magnitude (highest in red, low-est in blue), and the right column the ground truth matchlabels (inliers in red, outliers in blue), computed by thresh-olding the symmetric epipolar distance (Sec. 4.3). We drawmatches with negative labels with thinner lines. . . . . . . . 333.6 Output weight computed by the 2-layer MLP for agiven residual input – Computed for the residual-to-weight variant of ACNe. . . . . . . . . . . . . . . . . . . . . 384.1 Framework – We learn a capsule encoder for 3D pointclouds by relating the decomposition result of two randomrigid transformations Ta and Tb, of a given point cloud, i.e.,a Siamese training setup. We learn the parameters of anencoder E , a per-capsule decoder Dk, as well as a networkthat represents a learnt canonical frame K. For illustrativepurposes, we shade-out the outputs that do not flow forward,and with Σ summarize the aggregations in (4.2). . . . . . . . 444.2 Auto-encoding / qualitative – Example decompositionand reconstruction results setup using Canonical Capsules onseveral unaligned point cloud instances from the test set. Wecolor each Canonical Capsule with a unique colour, and sim-ilarly color “patches” from the reconstruction heads of 3D-PointCapsNet [13] and AtlasNetV2 [4]. Canonical Capsulesprovide semantically consistent decomposition that is alignedin canonical frame, leading to improved reconstruction qual-ity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534.3 Auto-encoding / qualitative – Example decompositionresults using Canonical Capsules on the test set, with thealigned setup; we color each decomposition (capsule) with aunique color. For 3D-PointCapsNet [13] and AtlasNetV2 [4],rather than capsules, these correspond to “patches” in thereconstruction network. Our method clearly provides the bestqualitative results. . . . . . . . . . . . . . . . . . . . . . . . . 58xviiList of Figures4.4 Random sampling of rotations – (a) Sampling Eulerangles uniformly results in a non-uniform coverage of SO(3);note we visualize only one-eighth of a sphere for ease of visu-alization. (b) Non uniform sampling results in auto-encodingerror to be biased w.r.t rotations (we use a cold-warm col-ormap to visualize the Chamfer distance error). (c) By prop-erly sampling rotations [14], this bias can be alleviated. . . . 615.1 Teaser – (top) Given a query point, we define the corre-sponding instance as the element-wise product of feature (pointswith similar class) and spatial (points with similar localiza-tion) affinities; this leads to formulating instance segmen-tation as a neural bilateral filter. (bottom) Our techniquecan be applied to large-scale instance segmentation of real-world ScanNet scenes, leading to the best-performance-in-class amongst top-down methods. . . . . . . . . . . . . . . . 665.2 Overview – (a) Given a query (⊙), we regress the boundinghull of the corresponding instance; (b) Together with seman-tic segmentation, this defines an affinity function on the entirepoint cloud; (c) This affinity can be threshold to generate acandidate instance proposal; (d) Instance proposals are thengrouped by non-maximal suppression to generate the scene’sinstance segmentation. . . . . . . . . . . . . . . . . . . . . . 695.3 Spatial similarity – The semantic feature is uninformativein separating the two instances: (left) an isotropic affinitykernel w.r.t. the query point would mistakenly assign pointson the left instance to the right one, regardless of bandwidthchoice; (right) a non-isotropic kernel does not suffer this short-coming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705.4 Neural Bilateral Filter (Section 5.4.3)– Given a query(position and feature) we iteratively apply the learned filtersto advect the query point position and features. Ultimately,the final attention A(T )(q) is used for downstream tasks. . . 725.5 Architecture – The point cloud is processed by a backboneto produce fq=f (0)q , which is then processed by a neural bilat-eral filter by our kernels K∗. We supervise sq via ground truthsemantic classification labels, where S1 is a 2-layer MLP, andS2 is a linear layer. . . . . . . . . . . . . . . . . . . . . . . . 74xviiiList of Figures5.6 Qualitative/Synthetic – Our method generates nearlyperfect query-conditioned instance proposals while the base-line is limited by the noisy bounding box. Note the red largepoints are the sampled queries. We color the points detectedby different instance proposals. And the black points are thebackground points or the points detected by more than twoinstance proposals. . . . . . . . . . . . . . . . . . . . . . . . 785.7 Visualizing the spatial kernels – Our method learns theeffective convex hulls that act as the tight bounding box of thetarget instance. For each convex hull, the magenta lines arethe learned half-planes. The red polygon is the intersectionbetween half-planes. Points are colored with spatial similaritywhere red means larger similarity while blue means smallersimilarity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 795.8 Qualitative/ScanNet – Instance segmentation results ontest set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 815.9 Ablation: number of iterations T – (top) We report theaverage and standard deviation of mAP by repeating the ex-periment 5 times. (bottom) Our algorithm shifts the queries(red points) to the centroid after a small number of iterations. 825.10 Ablation: losses – We report the average and standarddeviation of mAP by repeating the experiment 5 times. . . . 835.11 Visualization of convex hulls on ScanNet – Note wevisualize only some of the hulls to reduce visual density. Wevisualize the wireframe of the hulls for better visibility. Andwe color the points according to the instance prediction. . . 845.12 More qualitative results on ScanNet – Instance segmen-tation results in the validation set. . . . . . . . . . . . . . . . 856.1 Teaser – We introduce a novel volume-rendering frameworkto effectively leverage point clouds for Neural Radiance Fields.Our formulation aggregates points over multiple scales—includinga global scale governing the entire scene, equivalent to thestandard, point-agnostic NeRF. Our solution leads to muchbetter novel-view synthesis in challenging real-world situa-tions with sparse or incomplete point clouds. Here, we showexample renderings from the KITTI-360 test set. . . . . . . . 88xixList of Figures6.2 Overview – Given an input point cloud, we aggregate it overmulti-scale voxel grids (Sec. 6.4.1). For clarity, we draw thevoxel grids in 2D. We then perform volume rendering basedon points, relying on feature vectors stored thereon, whichwe aggregate across multiple scales (Sec. 6.4.2). Importantly,when aggregating across scales, we only take into account‘valid’ scales, i.e., those with nearby points—indicated withsolid blue lines and illustrated as the two overlaid scales inthe middle—naturally dealing with incomplete/sparse pointclouds. The coarsest scale (the top row in the figure) is asingle, global voxel, equivalent to standard NeRF—i.e., it isnot point-based. . . . . . . . . . . . . . . . . . . . . . . . . . 916.3 Increasing coverage with multiple scales –We illustrateour sparse, hierarchical representation at three granularitylevels, including a single, global voxel (left). We also showthree query points, with their respective neighborhoods (dot-ted circles) at each scale level—color-coded in blue if theyhave neighbouring features, and in orange if they do not.Our multi-scale approach naturally fills in empty regions, re-moving the need for failure-prone region-growing heuristics[9]. Drawn in 2D, for clarity. . . . . . . . . . . . . . . . . . . 936.4 Examples on KITTI-360 – We show novel-view render-ings obtained with our method, 3D Gaussian splatting [15](pink colored, 1-4 rows ) and PointNeRF [9] (green colored,4-8 row) on a challenging outdoors dataset, using the samepoint clouds as input. Our approach provides sharper render-ings with more details, and better coverage in areas withoutpoints, where Gaussian Splatting and PointNeRF producehighly salient artifacts highlighted with red boxes. . . . . . . 966.5 Rendering across scales – We study the behavior of ourhierarchical approach by rendering an image adding one scalelevel at a time, from the global scale to the finest one. Asexpected, the coarse scales are responsible for filling in emptyregions (highlighted with red boxes) in the point cloud, dif-ferent from the well-covered regions (highlighted with greenboxes) that can be modeled via fine scales. . . . . . . . . . . 97xxList of Figures6.6 Examples on NeRF Synthetic – Our multi-scale approachconsistently fills in the holes in the input point cloud. Point-NeRF relies on ‘pruning’ and ‘growing’ heuristics, which canfail where the point cloud is not sufficiently dense, as shownhere. While Gaussian Splatting also previous failed to fill inholes with their densification heuristics (Fig. 6.4 and Fig. 6.7),for NeRF Synthetic these heuristics work well. . . . . . . . 996.7 Examples on ScanNet – PointNeRF fails to reconstruct thescene on regions where the point cloud is empty. Both ourmethod and Gaussian Splatting are able to fill them in, butour approach produces cleaner results, with fewer artifacts.This is especially noticeable for Scene-101 (top and bottomrows), where the mesh has large holes where PointNeRF failsto render meaningful pixels, even with their ‘growing’ heuris-tic that is aimed towards filling such gaps. . . . . . . . . . . 1037.1 A surprising failure case in ChatGPT [16]: given theprompt ( “ We know small explicit models are awesomein some aspects. But it’s really the time to embracelarge scale foundation model"), ChatGPT (version of GPT-4o in Jun. 2024) – i.e., one of world’s largest language model,while generating the images of high image quality, has thegrammar errors in the image (multiple words are misspelled).And we observe this grammar error consistently with differentrandom seeds. . . . . . . . . . . . . . . . . . . . . . . . . . . 109xxiAcknowledgementsI would like to express my heartfelt gratitude to my supervisor, Dr. KwangMoo Yi, and my co-supervisor, Dr. Andrea Tagliasacchi, for their unwa-vering support and guidance throughout this journey. Their patience andencouragement have been invaluable, and this thesis would not have beenpossible without them. I also extend my sincere thanks to my committeemember, Dr. Helge Rhodin, for his insightful feedback.I am grateful to my collaborators, co-authors and friends, including SorooshYazdani, Eduard Trulls, Geoffrey Hinton, Daniel Rebain, Amin Banitalebi,Renjie Liao, Vladimir Tankovich, Boyang Deng, Sara Sabour, GengchenMai, Ni Lao, Wei Jiang, Yuhe Jin, Gopal Sharma, Shakiba Kheradmand,Daniel Rebain, Kacper Kania, Sneha Sambandam, Eric Hedlin, BaptisteAngles, Erqun Dong, Zhuoting Xie, Matthew Tang, Seungyeon Baek, ZhijieWu, Shabo Xia, Shrisudhan Govindarajan, Ahan Shabanov, Colin Li, andYang-Che Tseng. Your support has enriched my research and made thisjourney more rewarding.Lastly, I would like to thank my family for their love and patience. Myparents, my dog Yoko, and especially my wife, Dr. Xuezhen Ge, have beenmy pillars of strength. Your encouragement and unwavering support havemeant the world to me. I truly could not have accomplished this withoutyou.xxiiChapter 1IntroductionA geometric point cloud is an unordered set of points in Euclidean space.As shown in Fig. 1.1, this form of data is flexible and prevalent acrossvarious domains, ranging from 3D point clouds captured by range sensors(e.g., LiDAR and Depth sensor) to 4D/6D correspondence cloud in 2D/3Dmatching tasks [17, 18]. Furthermore, the continuous spatial coordinatesof each point offer precise geometric information complementary to othermodalities such as color images where geometries – e.g., obstacle distance inphysical space, 3D structure – are unknown yet essential. This makes themincreasingly indispensable across a broad range of applications[19–21].However, point cloud learning with deep neural networks is notoriously chal-lenging because points distribute non-uniformly [21, 22] – the distance be-tween closest point pairs vary, unlike image data with pixels defined in auniform grid where closest pixel pairs always have the same distance [22–25]. This uniformness allows efficient and effective processing of a data pointwith respect to its neighbors via, e.g., convolutions, which is a key elementfor the successful application of neural networks to images [26]. However,because point clouds are not non-uniform, it is difficult to directly applythe successful neural network techniques [27, 28] from the image domainfor point cloud learning. Naturally, despite the sophistication of backbonenetworks in the image domain, there is still ongoing research investigat-ing even basic operations such as how to perform convolution with pointclouds [25, 29]. Furthermore, the high fidelity of point clouds typically re-sults in a large number of points, requiring large computing requirementseven for small scenes, making it difficult to scale.Various methods have been proposed for point cloud learning [30]. Someapproaches follow the success of deep learning in the image domain. Theypreprocess a point cloud into the more regular forms of representation suchas multiview images [11] or voxels [20, 31], then use techniques developedfor regular structures, which are typically well explored. However, these1Chapter 1. IntroductionFigure 1.1: The prevalence of geometric point cloud: Geometric pointcloud is a commonplace and important data across domains. (First row),point clouds are easily captured with the range sensors like depth sensorsand LiDAR, enabling a large range of applications. (Second row), pointcloud captures geometry information simply with a set of points and is acommonly-adopted 3D representation [9, 10]. (Thrid row), point cloudgeneralizes into other tasks with any-dimensional points (2D, 4D and 6D).Notably, in matching tasks, we compute putative correspondences –i.e., pairsof 2D image pixels or 3D points, forming the point cloud in 4D (imagematching) or 6D (point cloud registration).methods require specific preprocessing steps like voxelization and render-ing, which compromises generality as preprocessing methods vary betweendatasets. More importantly, because preprocessing methods convert con-tinuous point coordinates into discrete images or voxels, they can lead toinformation loss.Another stream of work is point-based networks [22, 23]. They directlyconsume raw input point clouds and process them. They do not need anypreprocessing steps and thus have no information loss. While voxelization-2Chapter 1. IntroductionFigure 1.2: Explicit models: In general, a deep neural network is a blackbox, which consists of a stack of K layers with parameters of w and thein-between intermediate features f – both don’t need to carry any meaning(i.e., non-interpretable). In this thesis, we inject explicit meanings into thesefeatures like pose and attention, which enables the traditional ways of ma-nipulating the learned representation to solve the target tasks including iter-ative reweighted least square (ACNe – Ch. 3), equivariant capsule(Canonicalcapsules–Ch. 4), bilateral filtering (NeuralBF–Ch. 5), and points-based rep-resentation (PointNeRF++ – Ch. 6).based methods are currently prevalent in large-scale point cloud learning dueto their better efficiency, for small-scale tasks, point-based networks achievestate-of-the-art, for example, for small-scale tasks such as object classifica-tion [32] and semantic segmentation of indoor datasets [8, 33]. However, dueto the limited scalability – i.e., difficulty in training point-based networksin large-scale point clouds, the network design remains an open researchproblem [25, 29, 34].3Chapter 1. IntroductionIn this thesis, we focus on enforcing explicit representations into point-basednetworks, which we refer to as the explicit model. We define explicit repre-sentations as intermediate neural features that can be easily interpreted intotask-relevant information, such as how much one should attend to a datapoint – attention – and a general trend of a cluster of point clouds – theirposes. Having explicit representations allows traditional methods for solv-ing target tasks such as bilateral filtering [35] to be easily integrated into alearning-based framework. By integrating traditional methods, it serves asprior knowledge, which a learning-based model does not need to learn. Thiscan lead to better data efficiency (smaller dataset required for the task),parameter efficiency (a more compact model), and computation efficiency(faster inference) for point cloud learning. We argue that this is particularlyessential to point cloud learning where the training of a large network onlarge-scale point cloud data is still difficult.Specifically, we enforce a variety of explicit representations in different tasks.As shown Fig. 1.2, a deep neural network typically is a stack of layers pa-rameterized via weights and intermediate features. As a black box, theweights and intermediate features of each layer are not constrained to havespecific meanings. We inject explicit meanings by utilizing explicit repre-sentations as the weights and intermedia features of layers. We investi-gate four different tasks and propose four different architectures includingACNe[17], Canonical Capsules [36], NeuralBF[37] and PointNeRF++[21].In the ACNe[17], to deal with outliers we unroll robust optimization intoa stack of layers where explicit attention, serving as point confidence, is it-eratively refined. In Canonical capsules[36], for self-supervised learning ofpoint cloud representations, we propose a capsule-based point cloud autoen-coding framework where each capsule explicitly models pose and descriptors.In the NeuralBF [37], to perform instance segmentation with point clouds,we explicitly integrate bilateral filtering [35] into a neural network. Lastly,in the PointNeRF++[21], for novel-view rendering of scenes, we propose anexplicit architecture that combine point-based representation with NeuralRadiance Fields (NeRF) [6]).In each task, these explicit representations serve as valuable prior knowl-edge, resulting in improved performance and generalization. We empiricallyvalidate the benefits of the corresponding explicit models for each task andconclude that explicit representation consistently outperforms their black-box counterparts when data is limited. Most notably, resulting explicitrepresentations from our method are highly interpretable, inspiring futureresearch in this domain.41.1. Thesis overview and contributions1.1 Thesis overview and contributionsWe first briefly review work on processing point clouds in Chapter 2. Wethen present the four main contributions of the thesis, each investigating adifferent task.Chapter 3 – ACNe[17]: Many problems in computer vision require deal-ing with sparse, unordered data in the form of point clouds. Permutation-equivariant networks have become a popular solution – they operate onindividual data points with simple perceptrons and extract contextual in-formation with global pooling. This can be achieved with a simple normal-ization of the feature maps, a global operation that is unaffected by theorder. In this chapter, we propose Attentive Context Normalization (ACN),a simple yet effective technique to build permutation-equivariant networksrobust to outliers. Specifically, we show how to normalize the feature mapswith weights that are estimated within the network, excluding outliers fromthis normalization. We use this mechanism to leverage two types of atten-tion: local and global – by combining them, our method is able to find theessential data points in high-dimensional space to solve a given task. Wedemonstrate through extensive experiments that our approach, which wecall Attentive Context Networks (ACNe), provides a large leap in perfor-mance compared to the state-of-the-art on camera pose estimation, robustfitting, and point cloud classification under noise and outliers.Chapter 4 – Canonical Capsules [36]: We propose a self-supervised cap-sule architecture for 3D point clouds. We compute capsule decompositionsof objects through permutation-equivariant attention, and self-supervise theprocess by training with pairs of randomly rotated objects. Our key idea isto aggregate the attention masks into semantic keypoints, and use these tosupervise a decomposition that satisfies the capsule invariance/equivarianceproperties. This not only enables the training of a semantically consistentdecomposition, but also allows us to learn a canonicalization operation thatenables object-centric reasoning. To train our neural network we requireneither classification labels nor manually-aligned training datasets. Yet, bylearning an object-centric representation in a self-supervised manner, ourmethod outperforms the state-of-the-art on 3D point cloud reconstruction,canonicalization, and unsupervised classification.Chapter 5 NeuralBF [37]: We introduce a method for instance proposalgeneration for 3D point clouds. Existing techniques typically directly regressproposals in a single feed-forward step, leading to inaccurate estimation. We51.1. Thesis overview and contributionsshow that this serves as a critical bottleneck, and propose a method basedon iterative bilateral filtering with learned kernels. Following the spirit ofbilateral filtering, we consider both the deep feature embeddings of eachpoint, as well as their locations in the 3D space. We show via syntheticexperiments that our method brings drastic improvements when generatinginstance proposals for a given point of interest. We further validate ourmethod on the challenging ScanNet benchmark, achieving the best instancesegmentation performance amongst the sub-category of top-down methods.Chapter 6 PointNeRF++ [21]: Point clouds offer an attractive sourceof information to complement images in neural scene representations, espe-cially when few images are available. Neural rendering methods based onpoint clouds do exist, but they do not perform well when the point cloudquality is low—e.g. sparse or incomplete, which is often the case with real-world data. We overcome these problems with a simple representation thataggregates point clouds at multiple scale levels with sparse voxel grids atdifferent resolutions. To deal with point cloud sparsity, we average acrossmultiple scale levels—but only among those that are valid, i.e., that haveenough neighboring points in proximity to the ray of a pixel. To help modelareas without points, we add a global voxel at the coarsest scale, thus uni-fying “classical” and point-based NeRF formulations.After our main contributions, in Chapter 7 we conclude the thesis with adiscussion about the important role of explicit models in the era of largefoundation models, some practical tips about how to enforce explicit repre-sentation, and potential future works.6Chapter 2Related WorksWe briefly review work on neural network architectures for point cloud learn-ing in general. Specifically, we review both point cloud networks that pre-process points into a more standard (structured) representation and thenprocess them and those that directly operate on raw point clouds. And wesummarize three major input representations in Fig. 2.1. For a more de-tailed review of different point cloud networks, we refer readers to a surveypaper on point clouds [30].We also discuss other point cloud networks that share our philosophy of be-ing explicit. For related work that is specific to each task that we investigate,we provide them in their relevant Chapters.2.1 Processing point clouds by preprocessingthem into regular structuresSeveral methods [11, 38, 40, 55] attempt to use Convolutional Neural Net-works (CNN) to tackle point cloud learning, in their attempt to benefit fromtheir success in processing 2D images [27, 28]. They preprocess to convertthe irregular point cloud data into regular structures such as voxels [20, 41]and images [11, 38]. We categorize these prior works into multi-view-basedmethods and voxelization-based methods. A brief categorization of variousprevious works is shown in Tab. 2.1 and in Fig. 2.1.2.1.1 Multi-view-based methodsThese methods typically render a 3D point cloud into multi-view 2D im-ages and directly apply 2D CNNs. MVCNN [11] learns a global feature fora 3D object by aggregating information extracted from multi-view imagesindividually via 2D CNN. While it achieves promising performance in 3Dobject classification, it suffers from information loss when it aggregates datafrom multiple views via e.g., max pooling. Follow-ups [38–40] thus aim to72.1. Processing point clouds by preprocessing them into regular structuresYear Input TasksMVCNN [11] 2015 Multi-views ClassificationGVCNN [38] 2018 Multi-views ClassificationRelationNet [39] 2019 Multi-views ClassificationViewGCN [40] 2020 Multi-views ClassificationVoxNet [41] 2015 Voxel ClassificationORION [12] 2017 Voxel Classification, detectionO-CNN [42] 2017 Voxel ClassificationOctNet [43] 2017 Voxel Classification, segmentationSparseConvNet [31] 2018 Voxel General-purpose backboneMinkowskiNet [20] 2019 Voxel General-purpose backboneOctFormer [34] 2023 Voxel General-purpose backbonePointNet [44] 2016 Point Classification, segmentationPointNet++ [45] 2017 Point General-purpose backboneSONet [46] 2018 Point Classification, segmentationCNe [2] 2018 Point Robust point cloud learningPointConv [47] 2019 Point Classification, segmentationRSConv [48] 2019 Point Classification, segmentationKPConv [24] 2019 Point Classification, segmentationRandLA-Net [49] 2020 Point SegmentationConvPoint [50] 2020 Point Classification, segmentationOANet [51] 2019 Point Robust point cloud learningACNe [17] 2020 Point Robust point cloud learningPointTransformerV1 [52] 2021 Point Classification, segmentationPCT [53] 2021 Point Classification, segmentationPAConv [54] 2021 Point Classification, segmentationKPConvX [25] 2024 Point General-purpose backbonePointTransformerV3 [29] 2024 Point General-purpose backboneTable 2.1: A brief overview of previous works in point cloud learning.Note that we list the representative works for brevity. For more detailedprevious works closely related to our target tasks, please refer to the relatedwork section in each chapter.82.1. Processing point clouds by preprocessing them into regular structuresFigure 2.1: Three input representations for point cloud learning.We review methods with three input representations including multi-view,voxel, and points. As detailed in Sec. 2.2, through permutation-equivalence,point-based methods (also the focus of this thesis) have no information lossbecause they do not need preprocessing steps while multi-view-based [11]and voxel-based methods [12] do.enhance the aggregation module by considering the contextual informationbetween views. For example, ViewGCN [40] utilizes the graph neural net-work to relate views where each 2D view is a node in the graph, leadingto an improved feature representation for a 3D object. Thanks to CNNarchitecture, multi-view-based methods still remain the top performers in3D object classification. However, they lack the ability to provide point-level understanding, for example, segmentation, which greatly limits theirapplicability.2.1.2 Voxel-based methodsSeveral methods have been proposed to first convert point clouds into volu-metric data via voxelization, and then apply a 3D CNN. During voxelization,a voxel is labeled as occupied if there are points residing within the voxel.VoxNet [12, 41] classifies a 3D object using a 3D CNN with the input ofvoxels. The subsequent works improve performance in both efficiency and92.2. Point-based methodsaccuracy with better data structures like OctTree [42, 43]. More recently,some works [20, 31] propose to use sparse CNNs for point cloud learning.They ignore unoccupied voxels when applying convolution and massivelyimprove the computational efficiency of 3D CNNs. Naturally, since its in-troduction, Sparse 3D CNN (e.g., MinkowskiNet [20]) has become a standardmethod when processing large point clouds. Still, note that they still relyon voxelization, and are thus inherently subject to information loss duringthis process.2.2 Point-based methodsUnlike preprocessing-based methods, these methods directly consume rawpoint cloud data. Seminal works, PointNet [23] and PointNet++ [22], haveshown that deep neural networks can achieve promising performances bysimply processing raw points, without any preprocessing steps as requiredby previous voxels-based or multi-view-based methods. Their key recipeis to guarantee the permutation-equivariance at an architectural level, bysharing layers like multilayer perception (MLP) between points. Since then,many variants [24, 25, 29, 46, 56] have been proposed to enhance thesepoint-based methods. They propose various architectural variations for thebuilding blocks of PointNet++, e.g., how to aggregate neighborhood in-formation or to enforce properties such as permutation equivariance andinvariance, that are essential to point cloud learning. We thus discuss themin terms of which building block these methods are enhancing – permuta-tion equivariant layers, permutation invariant layers, point convolution, andpoint transformers.2.2.1 Permutation equivariant/invariant layersA point cloud remains the same after shuffling the order of points. Per-mutation equivariance/invariance is essential in the design of point-basedmethods. Specifically, when the order of input points changes, the order ofthe output point features must change accordingly – i.e., the permutationequivariance, and the output global feature must not change – i.e., permu-tation invariance. In this case, deep networks do not need to remember therandom order of input points.Permutation equivariance.Various modules exist to achieve this goal.PointNet [44] processes each point individually with a shared MLP. So theoutput point features keep the same order of input raw points – the output102.2. Point-based methodswill follow any change in the order of points, leading to the permutation-equivariance. PointNet++ [22] introduces the hierarchical feature learninginto PointNet, following the success of the hierarchical architecture thatCNNs use. In more detail, PointNet++ introduces a group aggregationlayer to mimic downsampling in CNNs and uses interpolation instead ofupsampling. Importantly, both operations are designed to be permutation-equivariant by construction. CNe [2] proposes ‘context normalization’ whichis performed at multiple intermediate layers instead of a global pooling thatis used in PointNet. Our work further introduces attention mechanisms tocontext normalization to gain robustness to outliers. Both CNe and ACNeguarantee the permutation equivariance by construction as their operationsare always either the same for each point or global, both of which are per-mutation equivariant. OANet [51] uses an order-aware module where theinput data is processed according to the order of input values, which makesinput order irrelevant to the processing pipeline. Other layers like pointconvolution and point transformer are also permutation equivariant, whichwe review separately in Sec. 2.2.2 and Sec. 2.2.3.Permutation invariance. Permutation-invariant layers learn to map pointcloud to an invariant feature regardless of the order of the input point cloud– e.g., the global classification of a point cloud should be variant to the inputorder of points. To this end, PointNet[44] utilizes the global max poolingover all points to get the global information. SONet [46] organizes pointsinto a set of ordered nodes. The order of nodes does not change if the orderof points changes, thus keeping the permutation invariance.2.2.2 Point convolutionsPoint convolution [47, 50], inspired by the discrete convolution in the imagedomain, defines a continuous convolution on the point cloud. Specifically,they calculate the output feature g ∈ RC at a D-dimensional point x byconsidering it’s neighboring points xi ∈ N (x) and the associated featuresfi ∈ RC asg =∑xi∈N (x)K(xi − x)fi (2.1)where N (x) defines the local neighborhood of x and K is the kernel functionthat maps relative coordinates between a point and its neighboring pointinto the weight matrix. The weight matrix transforms the input features,which are then summed into the output feature. Note that similarly to112.2. Point-based methodsimage convolution, output features can have a different dimension from theinput feature.Most variants of point convolutions follow this formula with slight differencesin the design of the kernel function. In RSConv [48], the kernel function ad-ditionally consumes the distance and absolute coordinates of neighboringpoints and outputs the channel-wise weight matrix to adjust input features– it is similar to depth-wise image convolution [57] which aims to improvethe efficiency of convolution. PAConv [54] uses a relatively larger MLPas the kernel function and predicts a full weight matrix. Despite betterperformance, PAConv is more computationally intensive than other vari-ants [47, 48, 50]. To improve the efficiency, KPConv [24] interpolates theweight matrix in a set of kernel points defined on grids, which is more effi-cient than the MLP-based kernel function. KPConvX [25] further improvesthe efficiency by predicting the channel-wise weight matrix and thereby cantrain deeper networks. KPConvX is the current state-of-the-art for pointconvolutions.2.2.3 Point TransformerInspired by the success of vision transformers [58], some works propose sev-eral variants of point transformers [29, 34, 52, 53]. They have set the newstate of the art across benchmarks of point cloud learning including semanticsegmentation [8, 33] and instance segmentation [8]. PCT [53] proposes offsetattention based on the vanilla attention layer [59] for point cloud process-ing. Despite the promising performance, PCT considers all points instead ofonly local neighboring points to learn the contextual information for a singlepoint, causing inferior performance in point cloud segmentation where local-ity information is important. To address this issue, PointTransformer [52]and its follow-ups [29] define the attention layer locally to incorporate local-ity information. Unlike typical transformers [59] where the attention layerrelates all the input tokens, each point only relates to neighboring points.Not only does it incorporate more locality information, but it also improvesthe efficiency of the transformer because of a smaller number of points toconsider to extract contextual information for a point. Octformer [34] uti-lizes the z-order pattern of the point cloud to efficiently partition the pointcloud into local patches, further improving the time efficiency of attentionlayers. While these point transformers achieve the best accuracy in segmen-tation [8, 33], voxel-based methods are still preferred in large-scale pointcloud processing due to their efficient data structure.122.3. Explicit models for point clouds2.3 Explicit models for point cloudsWe further review works that share the spirit as our thesis – point cloudlearning with the explicit architectural designs that are specifically designedfor a certain goal. They range across a variety of tasks and methods, but herewe briefly discuss those that show a large improvement. Transformation-equivariant networks such as Vector Neurons [60] and Frame Averaging [61]enforce rotation and translation equivariance into layer designs explicitlysuch that these transformations can be directly extracted and manipulatedwithin neural networks. They can thus generalize to point clouds of unseenposes, whereas vanilla point-based networks such as PointNet[44] would beunable to, without careful data augmentations and large training data. Forthe task of surface reconstruction from point clouds, NKSR [62] proposes adifferentiable kernel solver in the network to optimize the point-wise coef-ficients – i.e., a part of parameters in their signed distance field that out-puts signed distance to the surface from input query coordinates. Theseoptimized coefficients assure the consistency of normal prediction and thequeried signed distance. It achieves the state of the art in surface reconstruc-tion. Similarly to the findings of this thesis, their method shows improvedgeneralization compared to existing works, thanks to explicit modules en-forcing consistency or surfaces and normals, which is universal regardless ofthe data. For weakly-supervised point cloud segmentation, DSL [63] pro-poses to regularize the feature learning explicitly with bipartite matching.This encourages consistency between the semantic prototypes and unsuper-vised low-level segments, which leads to massively better performance overprior works [64, 65].13Chapter 3Unrolling Optimization as aDeep Network: ACNe3.1 Problem StatementThis chapter designs an explicit model (ACNe) for robust point cloud learn-ing, with a wide range of downstream applications in 3D vision. The keychallenge is the large ratio of outliers/noise in the input point cloud. Whileprevious works such as PointNet[44] achieve success in clean point cloudlearning, they’re limited in their inability to deal with noisy point clouds.As shown in our experiments Tab. 3.2, the performance of baselines largelydrops with the increasingly large ratio of outliers.Following the traditional robust estimation algorithm – iterative re-weightedleast square (IRLS)[66], we propose the ACNe, which we base our model onan existing best performer (i.e., CNe [18]) in this task, to explicitly modelpoint confidence inside network via attention. Furthermore, we enforce at-tention learning in each of the stack of layers where we treat each optimiza-tion step as a layer in a deep network, akin to the iterative refined weightsin traditional optimization algorithm. Our method demonstrates large im-provement over baselines across multiple tasks including robust point cloudclassification and outlier filtering in image matching. Most notably, we ob-serve that the simple attention injection is the key to the large improvementof our method, highlighting the potential of explicit representation.3.2 BackgroundSeveral problems in computer vision require processing sparse, unorderedcollections of vectors P = {pn ∈ RD}, commonly called clouds. Examplesinclude pixel locations (D=2), point clouds from depth sensors (D=3), andsparse correspondences across a pair of images (D=4). The latter includes143.2. Backgroundwide-baseline stereo, one of the fundamental problems in computer vision.It lies at the core of Structure-from-Motion (SfM), which, in turn, is thebuilding block of applications such as 3D reconstruction [67], image-basedrendering [68] and time-lapse smoothing [69].Wide-baseline stereo has been traditionally solved by extracting small collec-tions of discrete keypoints [70] and finding correspondences among them withrobust estimators [71], a reliable approach used for well over two decades.This has changed over the past few years, with the arrival of deep learn-ing and an abundance of new dense [72–74] and sparse [2, 75–78] methods.Here, we focus on sparse methods, which have seen many recent devel-opments made possible by the introduction of PointNets [1, 79] – neuralnetworks that rely on multi-layer perceptrons and global pooling to processunordered data in a permutation-equivariant manner – something which isnot feasible with neither convolutional nor fully-connected layers.Networks of this type – hereafter referred to as permutation-equivariant net-works – have pioneered the application of deep learning to point clouds.The original PointNet relied on the concatenation of point-wise (context-agnostic) and global (point-agnostic) features to achieve permutation equiv-ariance. Yi et al. [2] proposed Context Normalization (CN) as a simple,yet effective alternative to global feature pooling: all it requires is a non-parametric normalization of the feature maps to zero mean and unit vari-ance. Contrary to other normalization techniques utilized by neural net-works [80–83], whose primary objective is to improve convergence, contextnormalization is used to generate contextual information while preservingpermutation equivariance. Despite its simplicity, it proved more effectivethan the PointNet approach on wide-baseline stereo, contributing to a rela-tive increase in pose estimation accuracy of 50−100%; see [2, Fig. 5].Note that CN normalizes the feature maps according to first- (mean) andsecond- (variance) order moments. Interestingly, these two quantities canbe expressed as the solution of a least-squares problem:µˆ = argminµ∑n∥pn − µ∥22 (3.1)σˆ = argminσ∑n∥∥∥∥pn − µˆ∥22 − σ◦2∥∥∥22 (3.2)However, it is well known that least-squares optimization is not robust tooutliers [84, Sec. 3], a problem that also afflicts CN. We illustrate this lim-itation in Fig. 3.1, where the toy task is to fit a line to data corrupted153.2. BackgroundFigure 3.1: Robust neural line fitting – We learn to fit lines with outliers(80%) via our ACNe, as well as CNe [2]. We visualize the ground truthand the network estimates. We color-code the weights learned by thek-th residual layer of ACNe and used to normalize the feature maps – noticethat our method, which mimics Iterative Re-weighted Least Squares (IRLS),learns to progressively focus its attention on the inliers. This allows ACNeto find the correct solution where CNe fails.by outliers. Note that this is a critical weakness, as the application CNwas originally devised for, wide-baseline stereo, a problem plagued with out-liers : outlier ratios above 80% are typical in standard public datasets; seeSec. 3.4.2.To address this issue, we take inspiration1 from a classical technique used inrobust optimization: Iteratively Re-weighted Least Squares (IRLS) [66]. Asan example, let us consider the computation of the first-order moment (3.1).Rather than using the square of the residuals, we can optimize with respectto a robust kernel κ that allows for outliers to be ignored:argminµ∑nκ(∥pn − µ∥2), (3.3)which can then be converted back into an iterative least-squares optimization(t indexes iterations):argminµt∑nψ(∥pn − µt−1∥2)−1︸ ︷︷ ︸attention wtn∥pn − µt∥22, (3.4)where ψ(·) is the penalty function associated with the kernel κ(·); see [86,87]. Inspired by this, we design a network that learns to progressively focusits attention on the inliers, operating analogously to ψ(.) over the IRLSiterations.Specifically, we propose to train a perceptron that translates the (intermedi-ate) feature maps into their corresponding attention weights, and normalizesthem accordingly. We denote this approach as Attentive Context Normaliza-tion (ACN), and the networks that rely on this mechanism Attentive Context1See Section 3.6.7. A robust kernel can also be trained with a neural network andenforcing monotonicity; see [85].163.3. Related workNetworks (ACNe). We consider two types of attention, one that operates oneach data point individually (local), and one that estimates the relative im-portance of data points (global), and demonstrate that using them togetheryields the best performance. We also evaluate the effect of supervising thisattention mechanism when possible.We verify the effectiveness of our method on (1) robust line fitting, (2) clas-sification of 2D and 3D point clouds, and (3) wide-baseline stereo on real-world datasets (outdoors and indoors) showing large improvements over thestate of the art. Our work is, to the best our knowledge, the first to applyattentive mechanisms to the normalization of feature maps. One can alsoapply a more common form of attention by operating directly on featuremaps [88, 89], but we demonstrate that this does not perform as effectively.Contributions In summary, in this chapter we:• propose attentive context normalization, a novel architecture for pointcloud processing;• demonstrate its importance in processing outliers;• achieve state-of-the-art performance across multiple tasks including noisypoint cloud classification and wide-baseline stereo matching in real-worlddata.3.3 Related workWe discuss recent works on deep networks operating on point clouds, re-view various normalization methods for deep networks, and briefly discussattention mechanisms.Deep networks for point clouds. Several methods have been proposedto process point cloud data with neural networks. These include graph con-volutional networks [90, 91], VoxelNets [92], tangent convolutions [93], andmany others. A simpler strategy was introduced by PointNets [1, 79], whichhas since become a popular solution due to its simplicity. At their core, theyare convolutional neural networks with 1× 1 kernels and global pooling op-erations. Enhancements to the PointNet architecture include incorporatinglocality information with kernel correlation [94], and contextual informationwith LSTMs [95]. Another relevant work is Deep Sets [56], which derivesneural network parameterizations that guarantee permutation-equivariance.Permutation-equivariant networks for stereo. While PointNets were173.3. Related workoriginally introduced for segmentation and classification of 3D point clouds,Yi et al. [2] demonstrated that they can also be highly effective for robustmatching in stereo, showing a drastic leap in performance against hand-crafted methods [71, 96, 97]. The core ingredient of Yi et al. [2] is ContextNormalization (CN), an alternative to global feature pooling from PointNets.While similar to other normalization techniques for deep networks [80–83],CN has a different role – to aggregate point-wise feature maps and generatecontextual information. Follow-ups to CN include the use of architecturessimilar to Yi et al. [2] to iteratively estimate fundamental matrices [76],novel loss formulations [75], and the modeling of locality [77]. In OANet [51],order-aware filtering was utilized to incorporate context and spatial correla-tion. While all of these works rely on “vanilla” CN, we show how to improveits performance by embedding an attention mechanism therein. Our im-provements are compatible with any of these techniques.Normalization in deep networks. In addition to CN, different strategieshave been proposed to normalize feature maps in a deep network, startingwith the seminal work of Batch Normalization [80], which proposed to nor-malize the feature maps over a mini-batch. Layer Normalization [81] trans-posed this operation by looking at all channels for a single sample in thebatch, whereas Group Normalization [83] applied it over subsets of channels.Further efforts have proposed to normalize the weights instead of the activa-tions [98], or their eigenvalues [99]. The main use of all these normalizationtechniques is to stabilize the optimization process and speed up convergence.By contrast, Instance Normalization [82] proposed to normalize individualimage samples for style transfer, and was improved upon in [100] by align-ing the mean and standard deviation of content and style. Regardless of thespecifics, all of these normalization techniques operate on the entire sample– in other words, they do not consider the presence of outliers or their statis-tics. While this is not critical in image-based pipelines, it can be extremelyharmful for point clouds; see Fig. 3.1.Attentional methods. The core idea behind attention mechanisms is tofocus on the crucial parts of the input. There are different forms of at-tention, and they have been applied to a wide range of machine learningproblems, from natural language processing to images. Vaswani et al. [59]proposed an attentional model for machine translation eschewing recurrentarchitectures. Luong et al. [101] blended two forms of attention on sequen-tial inputs, demonstrating performance improvements in text translation.Xu et al. [102] showed how to employ soft and hard attention to gaze on183.4. Methodsalient objects and generate automated image captions. Local response nor-malization has been used to find salient responses in feature maps [103, 104],and can be interpreted as a form of lateral inhibition [105]. The use of atten-tion in convolutional deep networks was pioneered by Spatial TransformerNetworks [106], which introduced a differentiable sampler that allows forspatial manipulation of the image. In [107], attention is directly appliedto the feature map, given by a PointNet-style network operating on pointclouds. However, this strategy does not work as well as ours for wide-baselinestereo; see Section 3.6.2 for details.3.4 Method3.4.1 Attentive Context NormalizationGiven a feature map f∈RN×C , where N is the number of features (or datapoints at layer zero), C is the number of channels, and each row correspondsto a data point, we recall that Context Normalization [2] is a non-parametricoperation that can be written asNCN(f) = (f − µ(f))⊘ σ(f), (3.5)where µ(f)=E[f ] is the arithmetic mean, σ(f)=√E[(f − E[f ])◦2] is the stan-dard deviation of the features across N , and ⊘ denotes the element-wisedivision. Here we assume a single cloud, but generalizing to multiple clouds(i.e. batch) is straightforward. Note that to preserve the properties of un-structured clouds, the information in the feature maps needs to be nor-malized in a permutation-equivariant way. We extend CN by introducinga weight vector w∈[0, . . . 1]N , and indicate with µw (·) and σw (·) the cor-responding weighted mean and standard deviation. In contrast to ContextNormalization, we compute the weights w with a parametric functionWω(·)with trainable parameters2 ω that takes as input the feature map, and re-turns a unit norm vector of weights:w = η(Wω(f)) , η(x) = x/∥x∥1. (3.6)We then define Attentive Context Normalization asNACN(f ;w) = (f − µw(f))⊘ σw(f). (3.7)2For simplicity, we abuse the notation and drop the layer index from all parameters.All the perceptrons in our work operate individually over each data point with sharedparameters across each layer.193.4. MethodACNAttentive Context Normalization (ACN)Attentive Residual Block (ARB)ARBARBACNeGroup Norm.+ wglobalWlocal!Wglobal!wlocalACN wglobalfFin 'P OFk,0' Fk,1'KN ACN(f;w)f<latexit sha1_base64="bgSiO2IUOR+PFmva6ZDdIym5dAk=">AAAB8XicbVDLSgMxFL1T [binary] ==</latexit>f<latexit sha1_base64="bgSiO2IUOR+PFmva6ZDdIym5dAk=">AAAB8XicbVDLSgMxFL1T [binary] ==</latexit>f<latexit sha1_base64="bgSiO2IUOR+PFmva6ZDdIym5dAk=">AAAB8XicbVDLSgMxFL1T [binary] ==</latexit>f<latexit sha1_base64="bgSiO2IUOR+PFmva6ZDdIym5dAk=">[binary]</latexit>f<latexit sha1_base64="bgSiO2IUOR+PFmva6ZDdIym5dAk=">[binary]</latexit>f<latexit sha1_base64="bgSiO2IUOR+PFmva6ZDdIym5dAk=">[binary]</latexit>Figure 3.2: ACNe architecture – (Left) Our permutation-equivariant net-work receives an input tensor P of size N×D, which is processed by a seriesof K Attentive Residual Blocks (ARB). The output of the network is a ten-sor O size N×C, which is then converted to a representation appropriatefor the task at hand. Note that the first perceptron F inφ changes the di-mensionality from P of size N×D (input dimensions) to features f of sizeN×C. (Middle) Within each residual path of the ARB, we manipulate thefeature map with perceptrons Fφ with parameters φ, followed by AttentiveContext Normalization (ACN) – we repeat this structure twice. (Right) AnACN module computes local/global attention with two trainable networks,combines them via element-wise multiplication, and normalizes the featuremaps with said weights – the NACN block – followed by Group Normaliza-tion. Note that all features are all processed in the same way, individually,and the ACN block is the only place where they interact with each other –this architecture guarantees permutation-equivariance.The purpose of the attention networkWω (·) is to compute a weight functionthat focuses the normalization of the feature maps on a subset of the inputfeatures – the inliers. As a result, the network can learn to effectively clusterthe features, and therefore separate inliers from outliers.There are multiple attention functions that we can design, and multiple waysto combine them into a single attention vectorw. We will now describe thosethat we found effective for finding correspondences in wide-baseline stereo,and how to combine and supervise them effectively.Generating attention. We leverage two different types of attention mech-anisms, local and global:wlocali =W localω (fi) = sigmoid(Wf⊤i + b), (3.8)wglobali =Wglobalω (fi) =exp(Wf⊤i + b)ΣNj=1 exp(Wf⊤j + b) , (3.9)where W and b are the parameters of a perceptron, and fk denotes thefeature vector for data point k – the k-th row of the feature map f . Observe203.4. Methodthat the local attention mechanism (3.8) acts on each feature vector inde-pendently, whereas the global attention mechanism (3.9) relates the featurevector for each data point to the collection through softmax.Blending attention. Note that the product does not change the scale ofthe normalization applied in (3.7). Therefore, to take into account multipletypes of attention simultaneously, we simply merge them together throughelement-wise multiplication. One could use a parametric form of atten-tion blending instead; however, it is non-trivial to combine the weights in apermutation-equivariant way, and we found this simple strategy effective.Supervising attention. In some problems, the class for each data pointis known a priori and explicit supervision can be performed. In this case,adding a supervised loss on the attention signals can be beneficial. Forinstance, when finding good correspondences for stereo we can apply binary-cross entropy using the epipolar distance to generate labels for each putativecorrespondence, as in [2]. Our experiments in Section 3.5.2 show that whilethis type of supervision can provide a small boost in performance (1-2%),our approach performs nearly as well without this supervision.3.4.2 Network architecture and applicationsOur network receives as input P∈RN×D, the tensor representation of P,and produces an output tensor O∈RN×C . Note that as P is unstructured,O must be equivariant with respect to permutations of the N rows of P.This output tensor is then used in different ways according to the task athand. We model our architecture after [2], which we refer to as ContextNetwork (CNe). It features a series of residual blocks [108] with ContextNormalization (CN). Our architecture, which we call Attentive Context Net-work, or ACNe, is pictured in Fig. 3.2. A key distinction is that within eachnormalization block (Fig. 3.2; right) we link the individual outputs of eachperceptron Fφ to our ACN layer. We also replace the Batch Normalizationlayers [80] used in [2] with Group Normalization [83], as we found it performsbetter; see Section 3.5.2 for ablation tests.We demonstrate that ACNe can be used to solve multiple applications, rang-ing from classical problems such as robust line fitting (Section 3.4.2) andpoint cloud classification on MNIST and ModelNet40 (Section 3.4.2), torobust camera pose estimation for wide-baseline stereo (Section 3.4.2).213.4. MethodFigure 3.3: Classification – We add salt-and-pepper noise to MNIST im-ages, and then convert the digits to an unstructured point cloud. The %reports the outlier-to-inlier ratio.Robust line fittingWe consider the problem of fitting a line to a collection of points P∈RN×2that is ridden by noise and outliers; see Fig. 3.1. This problem can be ad-dressed via smooth (IRLS) or combinatorial (RANSAC) optimization – bothmethods can be interpreted in terms of sparse optimization, such that inliersand outliers are clustered separately; see [109]. Let us parameterize a line asthe locus of point [x, y] such that θ · [x, y, 1]=0. We can then score each rowof P (i.e. each 2D point) by passing the output tensor O=ACNe(P) to anadditional weight network – with local and global components – following(3.6), yielding weights w = η(Wω(O)). Given w, and expressing our pointsin homogeneous coordinates as P¯ = [P, 1]∈RN×3, we can compute our co-variance matrix as Cw(P)=P¯⊤diag(w)2P¯∈R3×3. Then, denoting ν0[C] asthe eigenvector of C corresponding to its smallest eigenvalue, ν0 [Cw(P)]is the estimated plane equation that we seek to find. We, therefore, min-imize the difference between this eigenvector and the ground truth, withadditional guidance to wlocal to help convergence:L(ω) = αmin+/−{∥ν0 [Cw(P)]± θ ∥22}+ βE[H(y,wlocal)], (3.10)where E [H(a,b)] is the average binary cross entropy between a and b, yis the ground-truth inlier label, and hyper-parameters α and β control theinfluence of these losses. The min+/− resolves the issue that −θ and θ arethe same line.Point cloud classificationWe can also apply ACNe to point cloud classification rather than reason-ing about individual points. As in the previous application, we consider aset of 2D or 3D locations P∈RN×D as input, where D is the number ofdimensions. In order to classify each point set, we transform the output223.4. Methodtensor O=ACNe(P) into a single vector v=µw (O) and associate it with aground-truth one-hot vector y through softmax. Additional weight networksto generate w are trained for this task. We train with the cross entropy loss.Thus, the loss that we optimize is:L(ω) = H (y, softmax(v)) . (3.11)Wide-baseline stereoIn stereo we are given correspondences as input, which is thus P ∈ RN×4,where N is the number of correspondences and each row contains two pixellocations on different images. In order to remain comparable with tradi-tional methods, we aim to solve for the Fundamental matrix, instead of theEssential matrix, i.e., without assuming known camera intrinsics. Thus,differently from [2, 75, 77], we simply normalize the image coordinateswith the image size instead. This makes our method more broadly ap-plicable, and directly comparable with most robust estimation methods forstereo [71, 96, 110–112].We obtain w from the output tensor O=ACNe(P) via (3.6), as in Sec-tion 3.4.2. The weights w indicate which correspondences are consideredto be inliers and their relative importance. We then apply a weighted vari-ant of the 8-point algorithm [113] to retrieve the Fundamental matrix Fˆ,which parameterizes the relative camera motion between the two cameras.To do so we adopt the differentiable, non-parametric form proposed by [2],and denote this operation as Fˆ=g (X,w). We then train our network toregress the ground-truth Fundamental matrix, as well as providing auxiliaryguidance to wlocal – the final local attention used to construct the outputof the network – with per-correspondence labels obtained by thresholdingover the symmetric epipolar distance [113], as in [2]. In addition, we alsoperform auxiliary supervision on wlocalk – the intermediate local attentionswithin the network – as discussed in Section 3.4.1. Note that this loss is notnecessary, but helps training and provides a small boost in performance;see Section 3.5.2. We do not supervise global attention and leave it for thenetwork to learn. We therefore write:L(ω) = αmin+/−{∥∥∥Fˆ± F∗∥∥∥2F}+ βE[H(y,wlocal)]+ γEk[H(y,wlocalk )], (3.12)where ∥ · ∥F is the Frobenius norm, H is the binary cross entropy, and ydenotes ground truth inlier labels. Again, the hyper-parameters α, β, and233.5. Experimentsγ control the influence of each loss. Similarly to the line-fitting case, themin+/− resolves the issue that −F∗ and F∗ express the same solution.3.5 Experiments3.5.1 Implementation detailsWe employ a K-layer structure (excluding the first linear layer that changesaccording to the number of channels) for ACNe, with K× ARB units, andtwo perceptron layers in each ARB. The number of layers K is set to 3× for2D point cloud classification, 6× for robust line fitting, and 12× for stereo.For 3D point cloud classification, we add ACN normalization to an existingarchitecture. We also use 32 groups for Group Normalization, as suggestedin [83]. Similarly to [2], we use C=128 channels per perceptron.Training setup. For all applications we use the ADAM optimizer [114]with default parameters and a learning rate of 10−3. Except for robust linefitting, we use a validation set to perform early stopping. For robust linefitting, the data is purely synthetic and thus infinite, and we train for 50kiterations. For MNIST, we use 70k samples with a 8:1:1 split for training,validation and testing. For stereo, we use the splits from [51]. For theloss term involving eigen-decomposition (terms multiplied by α in (3.10)and (3.12)), we use α=0.1, following [2]. All other loss terms have a weightof 1, that is, β=1 and γ=1. For stereo, we follow [2] and enable the terminvolving the Fundamental matrix – the first term in (3.12) – after 20kiterations.Robust estimators for stereo inference. As a special case, we evaluatethe possibility of applying standard robust estimators for outlier rejectionsuch as RANSAC after training the model to potentially maximize its per-formance, as previously done in [2, 51, 75]. To do so, we modify our architec-ture by changing the final layer to output only the local attention with theReLU+Tanh activation, as in Yi et al. [2]. We then simply threshold w withzero, select the data points that survive this process as inliers, and feed themto different RANSAC methods to process them further. We compare theseresults with those obtained directly from the weighted 8-point formulation.3.5.2 ResultsWe first consider a toy example on fitting 2D lines with a large ratio ofoutliers. We then apply our method to point cloud classification, follow-243.5. ExperimentsOutlier ratio 60% 70% 80% 85% 90%CNe [2] .00019 .0038 .056 .162 .425ACNe (Ours) 1e-6 .0008 .024 .130 .383Table 3.1: Robust line fitting – Line fitting results over the test set interms of the ℓ2 distance between ground-truth and the estimates.Outlier ratio 0% 10% 20% 30% 40% 50% 60%PointNet [1] 98.1 95.1 93.2 79.5 67.7 70.0 54.8CNe [2] 98.0 95.8 94.0 91.0 90.1 87.7 87.2ACNe (Ours) 98.3 97.2 96.5 95.3 94.7 94.3 93.7Table 3.2: 2D Point cloud classification – Classification accuracy onMNIST, under different outlier ratios (%). Our method performs best in allcases, and the gap becomes wider with more outliers.ing [1, 79], which includes 2D for digit classification on MNIST and 3Dfor object classification on ModelNet40 [115]. These three experiments il-lustrate that our attentional method performs better than vanilla ContextNormalization under the presence of outliers. We then apply our solutionto wide-baseline stereo, and demonstrate that this increase in performanceholds on challenging real-world applications, and against state-of-the-artmethods for robust pose estimation. Finally, we perform an ablation studyand evaluate the effect of supervising the weights used for attention in stereo.Robust line fitting – Fig. 3.1 and Tab. 3.1To generate 2D points on a random line, as well as outliers, we first sample2D points uniformly within the range [−1,+1]. We then select two pointsrandomly and fit a line that goes through them. With probability accord-ing to the desired inlier ratio, we then project each point onto the line toform inliers. We measure the error in terms of the ℓ2 distance betweenthe estimated and ground truth values for the line parameters. The resultsare summarized in Table 3.1, with qualitative examples in Fig. 3.1. ACNeconsistently outperforms CNe [2]. Both methods break down at a 85-90%outlier ratio, while the performance of ACNe degrades more gracefully. Asillustrated in Fig. 3.1, our method learns to progressively focus on the inliersthroughout the different layers of the network and weeds out the outliers.253.5. ExperimentsOutlier ratio 0% 10% 20% 30% 40% 50%PointNet 85.8 81.7 81.7 80.1 78.2 78.5PointNet w/ CN 87.2 84.3 84.5 83.4 81.7 81.6PointNet w/ ACN 87.7 84.6 85.0 84.6 83.3 84.2Table 3.3: 3D Point cloud classification – We replicate the 3D pointclassification experiment on ModelNet40 from [1], with vanilla PointNet.We then add outliers with Gaussian noise. Our approach performs bestwith and without outliers.Classifying digits – Fig. 3.3 and Tab. 3.2We evaluate our approach on handwritten digit classification on MNIST,which consists of 28 × 28 grayscale images. We create a point cloud fromthese images following the procedure of [1]: we threshold each image at128 and use the coordinates – normalized to a unit bounding box – of thesurviving pixel locations as data samples. We subsample 512 points withreplacement, in order to have the same number of points for all trainingexamples. We also add a small Gaussian noise of 0.01 to the pixel coordi-nates after sampling following [1]. Outliers are generated by sampling froma uniform random distribution. We compare our method against vanillaPointNet [1] and CNe [2]. For PointNet, we re-implemented their methodunder our framework to have an identical training setup.Tab. 3.2 summarizes the results in terms of classification accuracy. Ourmethod performs best, with the gap widening as the outlier ratio increases– while CNe shows some robustness to noise, PointNet quickly breaks down.Note that the results for PointNet are slightly different from the ones re-ported in [1], as we use a validation split to perform early stopping. Inaddition, to reduce randomness, we train 10 different models and report theaverage results.Classifying 3D objects – Tab. 3.3We apply our method to the problem of 3D object (point cloud) classifica-tion. We use the ModelNet40 dataset [115], and compare with PointNet [79].Similarly to the MNIST case, we contaminate the dataset with outliers totest the robustness of each method. Specifically, we add a pre-determinedratio of outliers to the point clouds, sampled uniformly within the range[−1, 1]. We also add small Gaussian perturbations to the locations of thepoints, with a standard deviation of 0.01. We then sample 1024 pointsfrom the point cloud to perform classification. Again, to simply test if ACN263.5. Experiments(a) RANSAC [71] (b) CNe [2] (c) ACNe (ours)(d) RANSAC [71] (e) CNe [2] (f) ACNe (ours)Figure 3.4: Wide-baseline stereo – We show the results of different match-ing algorithms on the dataset of [2]. We draw the inliers produced bythem, in green if the match is below the epipolar distance threshold (inred otherwise). Note that this may include some false positives, as epipolarconstraints map points to lines – perfect ground truth would require densepixel-to-pixel correspondences.273.5. Experimentscan improve existing pipelines, we plug our normalization into the vanillaPointNet architecture. Note that the original PointNet includes an affineestimation step which provides a small performance boost – we omit it fromour implementation, in order to isolate the architectural differences betweenthe methods. We report the results in Tab. 3.3. Our method performs best,with the gap becoming wider as outliers become prevalent.Wide-baseline stereo – Fig. 3.4 and Tab. 3.4Wide-baseline stereo is an extremely challenging problem, due to the largenumber of variables to account for – viewpoint, scale, illumination, occlu-sions, and properties of the imaging device – see Fig. 3.4 for some examples.We benchmark our approach on a real-world dataset against multiple state-of-the-art baselines, following the data [51] and protocols provided by [2].Their ground truth camera poses are obtained from Structure-from-Motionwith VisualSfM [116], from large image collections of publicly available,challenging photo-tourism data.We evaluate performance in terms of the reconstructed poses. Since thestereo matching problem is defined only up to a scale factor [113], it is notpossible to compute absolute (metric) errors for translation. Instead, wefollow the methodology of [2] and measure the error between the groundtruth and estimated vectors between both cameras, for both rotation andtranslation, and combine them by taking the maximum of the two. We thenevaluate the accuracy over all image pairs at multiple error thresholds, ac-cumulate it up to a limit (either 10◦ or 20◦), and summarize performanceby its mean – which we call the mean Average Precision (mAP); see [2].This means that methods that perform better at lower error thresholds arerewarded. We use their data mostly as is, using the pre-extracted corre-spondences and splits from OANet [51], but adapt it to the Fundamentalmatrix problem. In contrast to previous works [2, 51], which report resultson the scene the models were trained on, we focus on unknown scenes, inorder to determine each method’s actual performance.As we discussed in Section 3.4.2, both CNe [2] and OANet [51] assumeknown camera intrinsics and estimate the Essential matrix, instead of theFundamental matrix – this is a much easier problem, as the number of freeparameters drops from 7 to 5. However, most research papers on this topicfocus on estimating the Fundamental matrix [110–112, 117, 118], which iswhy we focus on this problem instead. For completeness, we also reportresults for the Essential matrix in the Tab. 3.8, for which we also achieve283.5. Experimentsstate-of-the-art results.In more detail, given an image pair, we extract 2k keypoints for each imagewith SIFT [70]. Matches are then formed from one image to the other, inboth directions. As is typical for image matching, we then filter out non-discriminative correspondences via bi-directional check, enforcing one-to-onematching. For RANSAC variants we found it to be critical to further applyLowe’s ratio test [70] – without it RANSAC variants provide worse results.We apply it with a ratio threshold of 0.8. We do not apply this test forlearned methods, as it throws out too many inliers for learned methods tobring any benefit. Also, when training learned methods, we train withoutbidirectional check to show as many correspondences in the training set aspossible to the network.We consider the following methods: LMedS [119], RANSAC [71, 110], MLE-SAC [96], DegenSAC [111], MAGSAC [112] CNe [2], DFE [76], OANet [51]and ACNe (ours). We consider the pose estimated with the weighted 8-pointalgorithm directly, as well as those combined with a robust outlier rejectionmethod as outlined in Section 3.5.1.Quantitative results. We report quantitative results in Tab. 3.4, for twodifferent error thresholds (10◦/20◦). We make three fundamental observa-tions:(1) Our method consistently outperforms all of the baselines, including CNeand OANet. The difference in performance between ACNe and its closestcompetitor, OANet, is of 14.1/9.8% relative for Outdoors, and 8.6/9.2% rel-ative for Indoors when used without any additional post processing. Thegap for Outdoors is reduced to 1% when they are combined with MAGSAC,but ACNe still outperforms OANet. For Indoors, we observe a drop in per-formance for both OANet and ACNe when combining them with RANSACor MAGSAC. The margin between learned and traditional methods is large,with ACNe performing 30.1/39.6% better relative on Outdoors and 45.5/47.1%better relative on Indoors, compared to the best performing traditional base-line – including a very recent method, MAGSAC.(2) Different from the findings of [2], we observe that RANSAC variantsmay harm performance, particularly with ACNe. This is because throughits global attention – wglobal – ACNe can infer the relative importance ofeach correspondence, which is not easily taken into account when passingsamples to a robust estimator. In this manner, ACNe goes beyond simpleoutlier rejection. The best performance is typically achieved by using ACNe293.5. ExperimentsMethod Outdoors IndoorsTraditional LMedS .296/.383 .142/.235RANSAC .356/.437 .172/.272MLESAC .148/.216 .135/.230DegenSAC .328/.394 .191/.291MAGSAC .385/.457 .185/.282LearnedCNe (weighted-8pt) .323/.469 .189/.331CNe+RANSAC .449/.554 .201/.315CNe+MAGSAC .500/.598 .213/.326DFE (weighted-8pt)3 .319/.470 .167/.294DFE+RANSAC .414/.508 .193/.303DFE+MAGSAC .452/.541 .211/.320OANet (weighted-8pt) .439/.581 .256/.392OANet+RANSAC .482/.592 .211/.331OANet+MAGSAC .514/.615 .230/.346Ours ACNe (weighted-8pt) .501/.638 .278/.428ACNe+RANSAC .478/.590 .209/.329ACNe+MAGSAC .518/.621 .226/.343Table 3.4: Pose estimation accuracy – mAP at 10◦/20◦ error threshold.Similarly to [2], we consider multiple baselines, as well as pairing differentmethods with state-of-the-art RANSAC variants. Our method consistentlyoutperforms all others by a large margin, even without an additional robustestimator, in some cases.303.5. Experimentsat its pure form, directly feeding its weights to the weighted 8-point algo-rithm. Given that all our experiments are on unseen sequences, this furthershows that ACNe generalizes very well, even without being followed by anadditional robust estimator.(3) Contrary to the results of Yi et al. [2] and Zhang et al. [51], we find thattraditional baselines perform better than reported on either work. This isbecause their experimental setup did not consider Lowe’s ratio test, nor thebidirectional check. Without these, the performance of traditional base-lines drops drastically – RANSAC and MAGSAC drop 79.2/73.0% and92.0/85.1% relative performance, respectively, for Outdoors, and 66.9/59.2%and 82.2/74.1% for Indoors.Ablation study – Tab. 3.5. We perform an ablation study to evaluatethe effect of the different types of attention, as well as the supervision onthe local component of the attentive mechanism. We also compare withCNe, as its architecture is the most similar to ours. We use the train andvalidation splits for the Saint Peter’s Square sequence for this study, as it isthe primary sequence used for training in [2] and has many images within theset. (1) We confirm that CNe [2] performs better with Batch Normalization(BN) [80] than with Group Normalization (GN) [83] – we use GN for ACNe,as it seems to perform marginally better with our attention mechanism. (2)We observe that our attentive mechanisms allow ACNe to outperform CNe,and that their combination outperforms their separate use. (3) Applyingsupervision on the weights further boosts performance.With learned features – Tab. 3.6. Finally, we report that our methodalso works well with two state-of-the-art, learned local feature methods –SuperPoint [3] and LF-Net [120]. They are learned end-to-end – their char-acteristics are thus different from those of SIFT keypoints. We test againon Saint Peter’s Square, as our primary focus is to show that it is possi-ble to use other feature types. In Tab. 3.6 we report that both methodsimprove performance over SIFT with OANet and ACNe, but with ratiotest and MAGSAC they perform worse. It is interesting how SuperPoint,without the ratio test, performs better than SIFT with MAGSAC, but theorder is reversed when the ratio test is introduced, highlighting its impor-3DFE, when tested with bidirectional check, gives poor results – .236 for Outdoors(20o). We hypothesize this may be due to the fact that DFE uses additional, “side” infor-mation, whose statistics may vary drastically with the bidirectional check. We thereforedo not use it for this baseline. This problem disappears when additional robust estimatorsare used.313.6. More ExperimentsMethods CNe [2] ACNe (Ours)w/ BN w/ GN L G L+G L+G+SWeighted-8pt .435 .414 .531 .593 .597 .602Table 3.5: Ablation study – mAP at 20◦ with different CNe [2] and ACNe(ours) variants on stereo. The labels indicate: L – local attention; G – globalattention; S – local attention supervision.SIFT SuperPoint LF-NetMAGSAC (w/o ratio test) .146 .205 .134MAGSAC .264 .230 .157OANet (weighted 8pt) .488 .547 .543OANet+MAGSAC .479 .442 .452ACNe (weighted 8pt) .602 .637 .619Table 3.6: With learned local features – mAP at 20◦ with learned localfeatures and different methods. Our method outperforms other methodsand performs best with SuperPoint [3].tance. Regardless of feature type, we demonstrate that our approach pro-vides improved performance over other methods, and that it pairs best withSuperPoint.3.6 More Experiments3.6.1 Visualizing attention – Fig. 3.5We visualize the weights for wide-baseline stereo, along with the “groundtruth” labels on the matches. Since the labels are obtained by thresholdingthe epipolar distance, computed from the ground truth poses, they containa few false positives. This figure shows that ACN learns to focus on inliersby emulating a robust iterative optimization. As our system is trained byFundamental matrix supervision, it was also able to learn to ignore thesefalse positives.323.6. More ExperimentsMatches: ACN Matches: Ground TruthFeature maps (t-SNE): for ACN Feature maps (t-SNE): for GTFigure 3.5: Attention in wide-baseline stereo. An illustration of ACNeon an example image pair. The top row shows the matches, and the bottomrow shows a representation of the feature maps obtained via t-SNE. The leftcolumn displays ACN weights, color-coded by magnitude (highest in red,lowest in blue), and the right column the ground truth match labels (inliersin red, outliers in blue), computed by thresholding the symmetric epipolardistance (Sec. 4.3). We draw matches with negative labels with thinnerlines.333.6. More ExperimentsMethods Attn. on feature map Our methodL G L+G L G L+GWeighted-8pt .410 .260 .408 .531 .593 .597Weighted-8pt (ReLU+Tanh) .427 .347 .369 — — —Table 3.7: Applying attention to the feature maps – We compareour method (right) to applying attention directly to the feature maps (left).Note we use attention variants: L (local), G (global), and L+G (Both).We report mAP at a 20◦ error threshold on our validation set – the SaintPeter’s Square scene. Applying attention on the normalization performslargely better than applying attention to the feature maps.3.6.2 Attention on feature maps – Tab. 3.7We show that applying attention to the normalization of the feature maps(our method) outperforms the more commonly used strategy of applyingattention directly to the feature maps [88, 89]. Tab. 3.7 extends our abla-tion study from Tab. 3.5, demonstrating that our method outperforms thisalternative approach by 29% relative.It is important to note that introducing global attention to the feature mapsresulted in unstable training. To avoid gradient explosion we reduced thelearning rate to one-tenth of the value we typically used. Nonetheless, gra-dients exploded after 224k iterations. We suspect that attention on featuremaps causes the feature maps to become artificially small, resulting in nu-merical instability. In all cases, the performance is slightly worse than whatCNe [2] gives in Tab. 3.5 (.414) showing that attention on feature maps isactually harmful.We also tried modifying the output of the network – i.e., the weights usedby the eight-point algorithm – to use the ReLU+Tanh configuration from[2], which we report in the bottom row of Tab. 3.7. This variant trained ina stable way, but provided sub-par results that are always lower than usinglocal attention only on the normalization. Note that with ReLU+Tanhand local attention only, attention on the feature maps does help – by 1%relative – but the increase in performance is very small compared to whatour method can achieve.343.6. More Experiments3.6.3 Essential matrix estimation – Tab. 3.8Several learned methods [2, 51] focus on estimating the Essential matrixinstead of the Fundamental matrix. The latter is more broadly applicableas it does not need a-priori knowledge about the intrinsics of the camera –hence it is closer to Computer Vision “in the wild”.We now demonstrate that our approach outperforms these methods for Es-sential matrix estimation as well. We report the results in Tab. 3.8, usingthe authors’ original implementations for this comparison. We also reportthe performance of robust estimators such as RANSAC and MAGSAC. ForRANSAC, we rely on findEssentialMat from OpenCV. We found that itis beneficial to use both local and global attention when applying *SAC tothe Essential matrix problem unlike the Fundamental matrix problem, andwe simply threshold w with an optimal threshold (i.e., 10−7) found on thevalidation set, and feed the surviving correspondences to *SAC. We observethat RANSAC improves the performance for ACNe in the outdoor experi-ments. This is due to the reduced complexity of the problem, which assumesknown camera intrinsics. For MAGSAC, we carefully implement the 5-pointalgorithm into their framework for estimating the Essential matrix. Whileit achieves competitive results, MAGSAC is still much worse than RANSACbecause it is originally geared for Fundamental matrix estimation.3.6.4 Number of ACNs within ARB – Tab. 3.9We perform an ablation study to evaluate the impact of the number ofACN blocks within each ARB. Due to the increasing computation overheadand GPU memory limitations, we only report the results of ACNe up tothree ACN blocks for each ARB; see Tab. 3.9. We expect that more ACNblocks would further improve the accuracy, at the cost of an increase inmemory/computation. We use 2 blocks, as it provides a good compromisebetween computational requirements and performance, and also the addi-tional advantage that this makes our results directly comparable to CNe [2].3.6.5 Number of parameters – Tab. 3.10Even though ACNe outperforms CNe with a large gap, the number of param-eters added to CNe is only 6K, which is only ≈1.5% more. The advantagesare more prominent when we compare against OANet, which introduces alarge increase in the number of parameters in the network, while providingworse results.353.6. More ExperimentsMethod Outdoors IndoorsRANSAC .671 .365MAGSAC .415 .204CNe (weighted-8pt) .515 .332CNe+RANSAC .750 .404CNE+MAGSAC .514 .259DFE (weighted-8pt) .573 .352DFE+RANSAC .721 .384DFE+MAGSAC .532 .265OANet (weighted-8pt) .648 .401OANet+RANSAC .776 .419OANet+MAGSAC .547 .271ACNe (weighted-8pt) .706 .429ACNe+RANSAC .780 .418ACNe+MAGSAC .415 .187Table 3.8: Essential matrix estimation – mAP at a 20◦ error thresholdwhen we train the models to estimate the Essential matrix instead of theFundamental matrix, for the indoors and outdoors experiments.#ACN per ARB 1 2 3Weighted-8pt .527 .602 .621Table 3.9: Ablation on #ACN – Performance as we vary the number ofACNs within each ARB – mAP at 20◦ on our validation set – Saint Peter’sSquare.3.6.6 Timing of *SAC Methods – Tab. 3.11We observed that ACNe and CNe share a similar runtime, and are bothmore efficient than OANet, which uses a deep permutation-equivariant net-work that performs an iterative refinement of an initial guess. Additionally,due to the GPU efficiency and low computational complexity, the runtime oflearned methods is negligible compared with traditional robust estimators(*SAC). Furthermore, learning-based methods are capable of facilitatingthe task of a robust estimator by proactively rejecting outliers. For in-stance, we found that ACNe makes RANSAC approximately 12× times andMAGSAC approximately 5× faster, while also massively improving overallperformance; see Tab. 3.4.363.6. More ExperimentsMethods OANet CNe ACNe# of Parameters 2347K 394K 400KTable 3.10: Number of parameters – Our method introduces a smalloverhead compared to CNe, and is much smaller than OANet.Methods Network Robust estimatorRANSAC — 194MAGSAC — 2752CNe 15 —CNe + RANSAC 15 19CNe + MAGSAC 15 523OANet 18 —OANet + RANSAC 18 13OANet + MAGSAC 18 546ACNe 14 —ACNe + RANSAC 14 16ACNe + MAGSAC 14 594Table 3.11: Average elapsed time – Runtime, in milliseconds, for indi-vidual steps of each method. We execute the forward pass of our networkson a GTX 1080 Ti GPU and the robust estimator on a Intel(R) Core(TM)i7-8700 CPU. We disable multi-threading for the CPU timings since not allrobust estimator implementations support multi-threading. Networks areimplemented with TensorFlow 1.8.0, except for OANet, which uses PyTorch1.2.0.Method CNe IRLS ACNeℓ2 error .0038 .0024 .0008Table 3.12: Performance of learnt IRLS – Comparison with CNe andACNe on the line fitting problem, under an outlier ratio of 0.7.373.6. More ExperimentsFigure 3.6: Output weight computed by the 2-layer MLP for a givenresidual input – Computed for the residual-to-weight variant of ACNe.3.6.7 Learnt IRLS variant – Fig. 3.6, Tab. 3.12Our method is inspired by iteratively re-weighted least squares (IRLS), andnot a direct translation. We learn a representations-to-weights mapping,and not a residuals-to-weights mapping as in traditional IRLS; see (3.4). Toensure the validity of our approach, we compare to a variant of our methodthat is more faithful to IRLS. In each (unrolled) iteration, we compute resid-uals by solving for the final objective (e.g. the optimal line parameters) withthe current weights. Then, we represent ψ(.)−1 in (3.4) with a 2-layer MLPthat is shared between iterations. In other words, the MLP serves as alearnt kernel, which is traditionally hand-picked in the IRLS literature. Weevaluate performance on the line fitting example due to its simplicity.As shown in Tab. 3.12, the residuals-to-weights variant performs much worsethan ACNe. This is not surprising, given that the IRLS variant is more383.7. Conclusionrestricted in what in can do, compared to ACNe. However, it is interestingto note that it still performs better than classical CNe. One very interestingaspect is that the learnt ψ(.)−1 has the typical monotonically decreasingproperty of typical (M-Estimator) kernel functions; see Fig. 3.6.3.7 ConclusionWe have proposed Attentive Context Normalization (ACN), and used it tobuild Attentive Context Networks (ACNe) to solve problems on permutation-equivariant data. Our solution is inspired by IRLS, where one iterativelyre-weighs the importance of each sample, via a soft inlier/outlier assignment.We demonstrated that by learning both local and global attention we areable to outperform state-of-the-art solutions on line fitting, classification ofpoint clouds in 2D (digits) and 3D (objects), and challenging wide-baselinestereo problems. Notably, our method thrives under large outlier ratios.For future research directions, we consider incorporating ACN into generalnormalization techniques for deep learning. We believe that this is an inter-esting direction to pursue, as all existing techniques make use of statisticalmoments.39Chapter 4Explicit LatentRepresentation: CanonicalCapsules4.1 Problem StatementIn this chapter, we propose a capsule network for self-supervised point cloudlearning, achieving state-of-the-art performance in downstream tasks includ-ing autoencoding, unsupervised classification, and canonicalization. An im-portant research contribution we address is to highlight the importance ofcanonicalization in 3D object understanding. Specifically, the success ofprevious works such as AtlasNetV2 [4] and 3D-PointCapsNet [13] in thisdomain relies on a built-in bias of 3D dataset that all objects are alignedin a canonical pose. For example, all cars from the ShapeNet dataset [121]head in the same direction, which is unrealistic in real-world data. Whilethese methods achieve promising performance in the aligned dataset, theirperformance massively drops in the unaligned dataset.We solve this problem with the explicit capsule-based presentation. Specifi-cally, our network takes in a point cloud of randomly transformed 3D objectsand outputs a set of capsules, each of which corresponds to one part of theobject. Each capsule consists of a transformation-invariant descriptor andthe transformation-equivariant pose. We then perform the canonicalizationby relating two components of the capsule. Under the mild assumption thatobjects having the similar appearance share the canonical pose, we learn anMLP to map the descriptor to the canonical pose. Through the comparisonbetween the canonical pose and the equivariant pose, we then calculate thecanonicalization matrix. We supervise the entire process via reconstructiontogether with some regularizers to avoid degeneration cases. We note thatthe explicit nature of the capsule is the key to enabling canonicalization and404.2. Backgroundstate-of-the-art performance in downstream tasks.4.2 BackgroundUnderstanding objects is one of the core problems of computer vision [122–124]. While this task has traditionally relied on large annotated datasets [28,125], unsupervised approaches that utilize self-supervision [126] have emergedto remove the need for labels. Recently, researchers have attempted to ex-tend these methods to work on 3D point clouds [127], but the field of unsu-pervised 3D learning remains relatively uncharted. Conversely, researchershave been extensively investigating 3D deep representations for shape auto-encoding4 [128–131], making one wonder whether these discoveries can nowbenefit from unsupervised learning for tasks other than auto-encoding.Importantly, these recent methods for 3D deep representation learning arenot entirely unsupervised. Whether using point clouds [128], meshes [129],or implicits [130], they owe much of their success to the bias within thedataset that was used for training. Specifically, all 3D models in the popu-lar ShapeNet [121] dataset are “object-centric” – they are pre-canonicalizedto a unit bounding box, and, even more importantly, with an orientationthat synchronizes object semantics to Euclidean frame axes (e.g. airplanecockpit is always along +y, car wheels always touch z = 0). Differentiable3D decoders are heavily affected by the consistent alignment of their out-put with an Euclidean frame [131, 132] as local-to-global transformationscannot be easily learnt by fully connected layers. As we will show in Sec-tion 4.5.2, these methods fail in the absence of pre-alignment, even whendata augmentation is used. A concurrent work [5] also recognizes this prob-lem and proposes a separate learnt canonicalizer, which is shown helpful indownstream classification tasks.In this work, we leverage the modeling paradigm of capsule networks [133].In capsule networks, a scene is perceived via its decomposition into parthierarchies, and each part is represented with a (pose, descriptor) pair: (1)The capsule pose specifies the frame of reference of a part, and hence shouldbe transformation equivariant; (2) The capsule descriptor specifies the ap-pearance of a part, and hence should be transformation invariant. Thus,one does not have to worry how the data is oriented or translated, as thesechanges can be encoded within the capsule representation.4Auto-encoding is also at times referred to as “reconstruction” or “shape-space” learn-ing.414.3. Related worksWe introduce Canonical Capsules, a novel capsule architecture to computea K-part decomposition of a point cloud. We train our network by feedingpairs of a randomly rotated/translated copies of the same shape (i.e. siamesetraining) hence removing the requirement of pre-aligned training datasets.We then decompose the point cloud by assigning each point into one of the Kparts via attention, which we aggregate into K keypoints. Equivariance isthen enforced by requiring the two keypoint sets to only differ by the known(relative) transformation (i.e. a form of self-supervision). For invariance, wesimply ask that the descriptors of each keypoint of the two instances match.WithCanonical Capsules, we exploit our decomposition to recover a canon-ical frame that allows unsupervised “object-centric” learning of 3D deep rep-resentations without requiring a semantically aligned dataset. We achievethis task by regressing canonical capsule poses from capsule descriptors viaa deep network, and computing a canonicalizing transformation by solvinga shape-matching problem [134]. This not only allows more effective shapeauto-encoding, but our experiments confirm this results in a latent represen-tation that is more effective in unsupervised classification tasks. Note that,like our decomposition, our canonicalizing transformations are also learnt ina self-supervised fashion, by only training on randomly transformed pointclouds.Contributions In summary, in this chapter we:• propose an architecture for 3D self-supervised learning based on capsuledecomposition;• enable object-centric unsupervised learning by introducing a learned canon-ical frame of reference;• achieve state-of-the-art performance 3D point cloud auto-encoding / re-construction, canonicalization, and unsupervised classification.4.3 Related worksConvolutional Neural Networks lack equivariance to rigid transformations,despite their pivotal role in describing the structure of the 3D scene behind a2D image. One promising approach to overcome this shortcoming is to addequivariance under a group action in each layer [135, 136]. In our work, weremove the need for a global SE(3)-equivariant network by canonicalizingthe input.424.3. Related worksCapsule Networks Capsule Networks [133] have been proposed to over-come this issue towards a relational and hierarchical understanding of nat-ural images. Of particular relevance to our work, are methods that applycapsule networks to 3D input data [13, 137, 138], but note these methods arenot unsupervised, as they either rely on classification supervision [137], oron datasets that present an inductive bias in the form of pre-alignment [13].In this work, we take inspiration from the recent Stacked Capsule Auto-Encoders [139], which shows how capsule-style reasoning can be effectiveas long as the primary capsules can be trained in a self-supervised fash-ion (i.e. via reconstruction losses). The natural question, which we answerin this work, is “How can we engineer networks that generate 3D primarycapsules in an unsupervised fashion?”Deep 3D representations Reconstructing 3D objects requires effectiveinductive biases about 3D vision and 3D geometry. When the input isimages, the core challenge is how to encode 3D projective geometry conceptsinto the model. This can be achieved by explicitly modeling multi-viewgeometry [140], by attempting to learn it [141], or by hybrid solutions [142].But even when input is 3D, there are still challenges. It is still not clearwhich is the 3D representation that is most amenable to deep learning.Researchers proposed the use of meshes [143, 144], voxels [42, 145], surfacepatches [4, 129, 146], and implicit functions [130, 147, 148]. Unfortunately,the importance of geometric structures (i.e. part-to-whole relationships) isoften overlooked. Recent works have tried to close this gap by using partdecomposition consisting of oriented boxes [149], ellipsoids [131, 150], convexpolytopes [151], and grids [152]. However, as previously discussed, most ofthese still heavily rely on a pre-aligned training dataset; our work attemptsto bridge this gap, allowing learning of structured 3D representations withoutrequiring pre-aligned data.Canonicalization One way to circumvent the requirement of pre-aligneddatasets is to rely on methods capable of registering a point cloud into acanonical frame. The recently proposed CaSPR [153] fulfills this premise,but requires ground-truth canonical point clouds in the form of normalizedobject coordinate spaces [154] for supervision. Similarly, [155] regresses eachview’s pose relative to the canonical pose, but still requires weak annotationsin the form of multiple partial views. C3DPO [156] learns a canonical 3Dframe based on 2D input keypoints. In contrast to these methods, our solu-tion is completely self-supervised. The concurrent Compass [5] also learnsto canonicalize in a self-supervised fashion, but as the process is not end-434.4. MethodFigure 4.1: Framework – We learn a capsule encoder for 3D point cloudsby relating the decomposition result of two random rigid transformations Taand Tb, of a given point cloud, i.e., a Siamese training setup. We learn theparameters of an encoder E , a per-capsule decoder Dk, as well as a networkthat represents a learnt canonical frame K. For illustrative purposes, weshade-out the outputs that do not flow forward, and with Σ summarize theaggregations in (4.2).to-end, this results in a worse performance than ours, as it will be shownin Section 4.5.3.Registration Besides canonicalization, many pairwise registration tech-niques based on deep learning have been proposed (e.g. [157, 158]), evenusing semantic keypoints and symmetry to perform the task [159]. Thesemethods typically register a pair of instances from the same class, but lackthe ability to jointly and consistently register all instances to a shared canon-ical frame.4.4 MethodOur network trains on unaligned point clouds as illustrated in Fig. 4.1:we train a network that decomposes point clouds into parts, and enforceinvariance/equivariance through a Siamese training setup [160]. We thencanonicalize the point cloud to a learnt frame of reference, and performauto-encoding in this coordinate space. The losses employed to train E , K,and D, will be covered in Section 4.4.1, while the details of their architectureare in Section 4.4.2.Decomposition In more detail, given a point cloud P ∈ RP×D of P pointsin D dimensions (in our case D=3), we perturb it with two random trans-formations Ta,Tb ∈ SE(D) to produce point clouds Pa,Pb. We then usea shared permutation-equivariant capsule encoder E to compute a K-foldattention map A ∈ RP×K for K capsules, as well as per-point feature map444.4. MethodF ∈ RP×C with C channels:A,F = E(P) , (4.1)where we drop the superscript indexing the Siamese branch for simplic-ity. From these attention masks, we then compute, for the k–th capsule itspose θk ∈ R3 parameterized by its location in 3D space, and the correspond-ing capsule descriptor βk ∈ RC :θk =∑pAp,kPp∑pAp,k, βk =∑pAp,kFp∑pAp,k. (4.2)Hence, as long as E is invariant w.r.t. rigid transformations of P, the pose θkwill be transformation equivariant, and the descriptor βk will be transforma-tion invariant. Note that this simplifies the design (and training) of the en-coder E , which only needs to be invariant, rather than equivariant [135, 138].Canonicalization Simply enforcing invariance and equivariance with theabove framework is not enough to learn 3D representations that are object-centric, as we lack an (unsupervised) mechanism to bring information into ashared “object-centric” reference frame. Furthermore, the “right” canonicalframe is nothing but a convention, thus we need a mechanism that allows thenetwork to make a choice – a choice, however, that must then be consistentacross all objects. For example, a learnt canonical frame where the cockpitof airplanes is consistently positioned along +z is just as good as a canonicalframe where it is positioned along the +y axis. To address this, we proposeto link the capsule descriptors to the capsule poses in canonical space, thatis, we ask that objects with similar appearance to be located in similarEuclidean neighborhoods in canonical space. We achieve this by regressingcanonical capsules poses (i.e. canonical keypoints) θ¯ ∈ RK×3 using thedescriptors β ∈ RK×C via a fully connected deep network K:θ¯ = K (β) (4.3)Because fully connected layers are biased towards learning low-frequencyrepresentations [161], this regressor also acts as a regularizer that enforcessemantic locality.Auto-encoding Finally, in the learnt canonical frame of reference, to trainthe capsule descriptors via auto-encoding, we reconstruct the point cloudswith per-capsule decoders Dk:P˜ = ∪k{Dk(R¯θk + t¯,βk)}, (4.4)454.4. Methodwhere ∪ denotes the union operator. The canonicalizing transformation T¯ =(R¯, t¯) can be readily computed by solving a shape-matching problem [134],thanks to the property that our capsule poses and regressed keypoints arein one-to-one correspondence:R¯, t¯ = argminR,t1K∑k∥(Rθk + t)− θ¯k∥22 . (4.5)While the reconstruction in (4.4) is in canonical frame, note it is trivialto transform the point cloud back to the original coordinate system afterreconstruction, as the transformation T¯−1 is available.4.4.1 LossesAs common in unsupervised methods, our framework relies on a numberof losses that control the different characteristics we seek to obtain in ourrepresentation. Note none of these losses require ground truth labels. Weorganize the losses according to the portion of the network they supervise:decomposition, canonicalization, and reconstruction.DecompositionWhile a transformation invariant encoder architecture shouldbe sufficient to achieve equivariance/invariance, this does not prevent theencoder from producing trivial solutions/decompositions once trained. Ascapsule poses should be transformation equivariant, the poses of the two ro-tation augmentations θak and θbk should only differ by the (known) relativetransformation:Lequivariance = 1K∑k∥θak − (Ta)(Tb)−1θbk∥22 . (4.6)Conversely, capsule descriptors should be transformation invariant, and asthe two input points clouds are of the same object, the corresponding capsuledescriptors β should be identical:Linvariance = 1K∑k∥βak − βbk∥22 . (4.7)We further regularize the capsule decomposition to ensure each of the Kheads roughly represent the same “amount” of the input point cloud, hencepreventing degenerate (zero attention) capsules. This is achieved by penal-izing the attention variance:Lequilibrium = 1K∑k∥ak − 1KΣkak∥22 , (4.8)464.4. Methodwhere ak = Σp(Ap,k) denotes the total attention exerted by the k-th headon the point cloud.Finally, to facilitate the training process, we ask for capsules to learn alocalized representation of geometry. We express the spatial extent of acapsule by computing first-order moments of the represented points withrespect to the capsule pose θk:Llocalization = 1K∑k1ak∑pAp,k∥θk −Pp∥22 . (4.9)Canonicalization To train our canonicalizer K, we relate the predictedcapsule poses to regressed canonical capsule poses via the optimal rigidtransformation from (4.5):Lcanonical = 1K∑k∥(R¯θk + t¯)− θ¯k∥22 . (4.10)Recall that R¯ and T¯ are obtained through a differentiable process. Thus,this loss is forcing the aggregated pose θk to agree with the one that goesthrough the regression path, θ¯k. Now, since θ¯k is regressed solely fromthe set of capsule descriptors, similar shapes will result in similar canonicalkeypoints, and the coordinate system of θ¯k is one that employs Euclideanspace to encode semantics.Reconstruction To learn canonical capsule descriptors in an unsupervisedfashion, we rely on an auto-encoding task. We train the decoders {Dk} byminimizing the Chamfer Distance (CD) between the (canonicalized) inputpoint cloud and the reconstructed one, as in [128, 129]:Lrecon = CD(R¯P+ t¯, P˜). (4.11)4.4.2 Network ArchitecturesWe briefly summarize our implementation details, including the networkarchitecture; for further details, please refer to the Sec. 4.4.3.Encoder – E Our architecture is based on the one suggested in [17]: apointnet-like architecture with residual connections and attentive contextnormalization. We utilize Batch Normalization [80] instead of the GroupNormalization [162], which trained faster in our experiments. We further474.4. Methodextend their method to have multiple attention maps, where each attentionmap corresponds to a capsule.Decoder – D The decoder from (4.4) operates on a per-capsule basis. Ourdecoder architecture is similar to AtlasNetV2 [4] (with trainable grids). Thedifference is that we translate the per-capsule decoded point cloud by thecorresponding capsule pose.Regressor – K We concatenate the descriptors and apply a series of fullyconnected layers with ReLU activation to regress the P capsule locations.At the output layer we use a linear activation and subtract the mean ofthe outputs to make our regressed locations zero-centered in the canonicalframe.Canonicalizing the descriptorsAs our descriptors are only approximatelyrotation invariant (via Linvariance), we found it useful to re-extract the capsuledescriptors βk after canonicalization. Specifically, we compute F¯ with thesame encoder setup, but with P¯=R¯P+T¯ instead of P and use it to computeβ¯k; we validate this empirically in the Sec. 4.64.4.3 More Details of Network ArchitecturesWe detail our architecture design for the capsule encoder E , the decoder Dand the regressor K.Capsule Encoder – EThe capsule encoder E takes in input a point cloud P ∈ RP×D and outputsthe K-fold/head attention map A ∈ RP×K and the feature map F ∈ RP×C .Specifically, the encoder E is based on ACNe [17] and composed of 3 residualblocks where each residual block consists of two hidden MLP layers with128 neurons each. Each hidden MLP layer is followed by Attentive ContextNormalization (ACN) [17], batch normalization [80], and ReLU activation.To be able to attend to multiple capsules, we extend the ACN layer intomulti-headed ACN.Multi-headed ACN For each MLP layer where we apply ACN, if we de-note this layer as i, we first train a fully-connected layer that creates a K-headed attention map Ai ∈ RP×K given the Fi ∈ RP×C of this layer. Thisis similar to ACN, but instead of a single attention map, we now have K.The normalization process is similar to ACN: we utilize the weighted mo-484.4. Methodments of Fi with Ai, but results in K normalized outcomes instead of one.We them aggregate these K normalization results into one by summation.In more details, given the Fip ∈ RP×C , if we denote the weights and biasesto be trained for the kth attention head to be Wik ∈ RC×1 and bik ∈ R1 wewrite:Aip,k =exp(FipWik + bik)∑k exp(FipWik + bik). (4.12)We then compute the moments that are used to normalize in ACN, but nowfor each attention head:µk =∑pAip,kFip∑pAip,k, σk =∑pAip,k(Fip − µk)2∑pAip,k, (4.13)which we then use to normalize and aggregate (sum) to get our final nor-malized feature map:Fip =∑kAip,k(Fip − µk)√σk + ϵ, (4.14)where ϵ = 0.001 is to avoid numerical instabilities.Capsule Decoder – DThe decoder D is composed of K per-capsule decoders Dk. Each per-capsuledecoder Dk maps the kth capsule in canonical frame (R¯θk+t¯, βk) to a groupof points P˜k ∈ RM×D that corresponds to a part of the entire object. Wethen obtain the auto-encoded (reconstructed) point clouds P˜ by collectingthe outputs of K per-capsule decoders and taking their union as in (4.4).Specifically, each Dk consists of 3 hidden MLP layers of (1280, 640, 320)neurons, each followed by batch normalization and a ReLU activation. Atthe output layer, we use a fully connected layer, followed by tanh activationto regress the coordinates for each point. Similarly to AtlasNetV2 [4], Dkadditionally receives a set of trainable grids of size (M × 10)5 and deformsthem into P˜k, based on the shape code βk. Note that the generated pointclouds from Dk lie in the reference frame of the canonicalized pose θ¯k, sowe rotate and translate the point cloud by θ¯k in the output layer.5Note the constant 10 is not related to K here.494.5. ResultsRegressor– KThe regressor K learns to regress the canonical pose for each capsule θ¯ ∈RK×D from their descriptors βk. In order to do so, we concatenate the(ordered) set of descriptors {βk} into a single, global descriptor, which wethen feed into a fully connected layer with 128 × K neurons and a ReLUactivation, followed by an additional fully connected layer with D×K neu-rons, creating K D-dimensional outputs (i.e. canonical pose). We do notapply any activation on the second layer. To make θ¯ zero-centered in thecanonical frame, we further subtract the mean of the outputs.4.5 ResultsWe first discuss the experimental setup, and then validate our method ona variety of tasks: auto-encoding, canonicalization, and unsupervised clas-sification. While the task differs, our learning process remains the same:we learn capsules by reconstructing objects in a learnt canonical frame. Wealso provide an ablation study, which is expanded in detail in the Sec. 4.6,where we provide additional qualitative results.4.5.1 Experimental setupTo evaluate our method, we rely on the ShapeNet (Core) dataset [121]6.We follow the category choices from AtlasNetV2 [4], using the airplane andchair classes for single-category experiments, while for multi-category exper-iments we use all 13 classes: airplane, bench, cabinet, car, chair, monitor,lamp, speaker, firearm, couch, table, cellphone, and watercraft. To makeour results most compatible with those reported in the literature, we alsouse the same splits as in AtlasNetV2 [4]: 31747 shapes in the train, and7943 shapes in the test set. Unless otherwise noted, we randomly sample1024 points from the object surface for each shape to create our 3D pointclouds.De-canonicalizing the datasetAs discussed in the introduction, ShapeNet(Core) contains substantial inductive bias in the form of consistent semanticalignment. To remove this bias, we create random SE(3) transformations,and apply them to each point cloud. We first generate uniformly sampledrandom rotations, and add uniformly sampled random translations withinthe range [−0.2, 0.2], where the bounding volume of the shape ranges in6Available to researchers for non-commercial research and educational use.504.5. Results[−1,+1]. Note the relatively limited translation range is chosen to givestate-of-the-art methods a chance to compete with our solution. We thenuse the relative transformation between the point clouds extracted fromthis ground-truth transformation to evaluate our methods. We refer to thisunaligned version of the ShapeNet Core dataset as the unaligned setup,and using the vanilla ShapeNet Core dataset as the aligned setup. For thealigned setup, as there is no need for equivariance adaptation, we simplytrain our method without the random transformations, and so Lequivarianceand Linvariance are not used. This setup is to simply demonstrate how Canon-ical Capsules would perform in the presence of a dataset bias.We emphasize here that a proper generation of random rotation is important.While some existing works have generated them by uniformly sampling thedegrees of freedom of an Euler-angle representation, this is known to bean incorrect way to sample random rotations [14], leading to biases in thegenerated dataset; see the Sec. 4.6.Implementation details For all our experiments we use the Adam op-timizer [114] with an initial learning rate of 0.001 and decay rate of 0.1.We train for 325 epochs for the aligned setup to match the AtlasNetV2 [4]original setup. For the unaligned setting, as the problem is harder, we trainfor a longer number of 450 epochs. We use a batch size of 16. The trainingrate is ∼2.5 iters/sec. We train each model on a single NVidia V100 GPU.Unless stated otherwise, we use k=10 capsules and capsule descriptors ofdimension C=128. We train three models with our method: two that aresingle-category (i.e., for airplane and chairs), and one that is multi-category(i.e., all 13 classes). To set the weights for each loss term, we rely on the re-construction performance (CD) in the training set. We set weights to be onefor all terms except for Lequivariance (5) and Lequilibrium (10−3). In the alignedcase, because Lequivariance and Linvariance are not needed (always zero), wereduce the weights for the other decomposition losses by 103; Llocalization to10−3 and Lequilibrium to 10−6.4.5.2 Auto-encoding – Fig. 4.2 and Tab. 4.1We evaluate the performance of our method for the task that was used totrain the network – reconstruction / auto-encoding – against three base-lines (trained in both single-category and multi-category variants): (1) 3D-PointCapsNet [13], an auto-encoder for 3D point clouds that utilize a cap-sule architecture; (2) AtlasNetV2 [4], a state-of-the-art auto-encoder whichutilizes a multi-head patch-based decoder; (3) AtlasNetV2 [4] with a spatial-514.5. ResultsMethod Airplane Chair MultiAligned 3D-PointCapsNet [13] 1.94 3.30 2.49AtlasNetV2 [4] 1.28 2.36 2.14Our method 0.96 1.99 1.76Unaligned 3D-PointCapsNet [13] 5.58 7.57 4.66AtlasNetV2 [4] 2.80 3.98 3.08AtlasNetV2 [4] w/ STN 1.90 2.99 2.60Our method 1.11 2.58 2.22Table 4.1: Auto-encoding / quantitative – Performance in terms ofChamfer distance with 1024 points per point cloud – metric is multiplied by103 as in [4].transformer network (STN) aligner from PointNet [23], a baseline withcanonicalization. We do not compare against [138], as unfortunately sourcecode is not publicly available.Quantitative analysis – Tab. 4.1 We achieve state-of-the-art perfor-mance in both the aligned and unaligned settings. The wider margin in theunaligned setup indicates tackling this more challenging scenario damagesthe performance of AtlasNetV2 [4] and 3D-PointCapsNet [13] much morethan our method7. We also include a variant of AtlasNetV2 for which a STN(Spatial Transformer Network) is used to pre-align the point clouds [23],demonstrating how the simplest form of pre-aligner/canonicalizer is not suf-ficient.Qualitative analysis – Fig. 4.2 We illustrate our decomposition-basedreconstruction of 3D point clouds, as well as the reconstructions of 3D-PointCapsNet [13] and AtlasNetV2 [4]. As shown, even in the unalignedsetup, our method is able to provide semantically consistent capsule de-compositions – e.g. the wings of the airplane have consistent colours, andwhen aligned in the canonical frame, the different airplane instances are7Results in this table differ slightly from what is reported in the original papers as weuse 1024 points to speed-up experiments throughout this work. However, in the Sec. 4.6the same trend holds regardless of the number of points, and match with what is reportedin the original papers with 2500 points.524.5. ResultsInputOur capsuledecomposi-tionOur recon-struction incanonicalframeOur recon-struction ininput frame3D-PointCapsNet [13]reconstruc-tionAtlasNetV2 [4]reconstruc-tionFigure 4.2: Auto-encoding / qualitative – Example decomposition andreconstruction results setup using Canonical Capsules on several unalignedpoint cloud instances from the test set. We color each Canonical Capsulewith a unique colour, and similarly color “patches” from the reconstructionheads of 3D-PointCapsNet [13] and AtlasNetV2 [4]. Canonical Capsulesprovide semantically consistent decomposition that is aligned in canonicalframe, leading to improved reconstruction quality.well-aligned. Compared to AtlasNetV2 [4] and 3D-PointCapsNet [13], thereconstruction quality is also visibly improved: we better preserve detailsalong the engines of the airplane, or the thin structures of the bench; notealso that the decompositions are semantically consistent in our examples.Results are better appreciated in our Sec. 4.6, where we visualize the per-formance as we continuously traverse SE(3).4.5.3 Canonicalization – Tab. 4.2We compare against three baselines: (1) Deep Closest Points [157], a deeplearning-based pairwise point cloud registration method; (2) DeepGMR [158]a state-of-the-art pairwise registration method that decomposes clouds intoGaussian mixtures and utilizes Rigorously Rotation-Invariant (RRI) fea-534.5. Resultstures [163]; (3) Compass [5] a concurrent work on learnt alignment / canon-icalization. For all compared methods we use the official implementation.For DeepGMR we use both RRI and the typical XYZ coordinates as in-put. We also try our method with the RRI features, following DeepGMR’straining protocol and train for 100 epochs.Metrics To evaluate the canonicalization performance, we look into thestability of the canonicalization – the shakiness shown in the videos inour project website (https://canonical-capsules.github.io/) – repre-sented as the mean standard deviation of the rotations (mStd):mStd = 1n∑ni=1√1m∑mj=1(∠(Rij ,Rimean))2 , (4.15)where ∠ is the angular distance between two rotation matrices [158, 164,165], Rij is the rotation matrix of the j-th instance of the i-th object incanonical frame, and Rimean is the mean rotation [166] of the i-th object.Note that with mStd we measure the stability of canonicalization with re-spect to rotations to accommodate for methods that do not deal with trans-lation [5]. To allow for comparisons with pairwise registration methods, wealso measure performance in terms of the RMSE metric [158, 167].Quantitative analysis – Tab. 4.2 Compared to Compass [5], our methodprovides improved stability in canonicalization. This also provides an advan-tage in pairwise registration, delivering state-of-the-art results when XYZ-coordinates are used. Note that while pairwise methods can align two sets ofgiven point clouds, creating a canonical frame that simultaneously registersall point clouds is a non-trivial extension to the problem.When RRI is used as input, our method is on par with DeepGMR [158], up toa level where registration is near perfect – alignment differences when errorsare in the 10−4 ballpark are indiscernible. We note that the performance ofDeep Closest Points [157] is not as good as reported in the original paper,as we uniformly draw rotations from SO(3). When a sub-portion of SO(3)is used, e.g. a quarter of what we are using, DCP performs relatively well(0.008 in the multi-class experiment). While curriculum learning could beused to enhance the performance of DCP, our technique does not need torely on these more complex training techniques.We further note that, while RRI delivers good registration performance, us-ing RRI features cause the learnt canonicalization to fail – training becomesunstable. This hints that RRI features may be throwing away too much544.5. ResultsCanonicalization (mStd) ↓ Pairwise registration (RMSE) ↓Method Airplane Chair Multi Airplane Chair MultiXYZ-coord. Deep Closest Points [157] – – – 0.318 0.160 0.131DeepGMR [158] – – – 0.079 0.082 0.077Compass [5] 19.105 19.508 51.811 0.412 0.482 0.515Our method 8.278 5.730 21.210 0.022 0.027 0.074RRI DeepGMR [158] – – – 0.0001 0.0001 0.0001Our method (unstable) (unstable) (unstable) 0.0006 0.0009 0.0016Table 4.2: Canonicalization – Quantitative evaluation for canonicaliza-tion, where we highlight massively better performance than the concurrentwork Compass [5]. While pairwise registration is not our main objective,the global alignment frame created by our method still allows for effectiveregistration (on par, or better) than the state-of-the-art.Method SVM K-MeansAligned 3D-PointCapsNet [13] 93.81 65.87AtlasNetV2 [4] 94.07 61.66Our method 94.21 69.82Unaligned 3D-PointCapsNet [13] 71.13 14.59AtlasNetV2 [4] 64.85 17.12AtlasNetV2 [4] w/ STN 78.55 20.03Our method 87.33 43.04Table 4.3: Classification – Top-1 accuracy(%)information to achieve transformation invariance. Our method using rawXYZ coordinates as input, on the other hand, provides comparable registra-tion performance, and is able to do largely more than just registration (i.e.classification, reconstruction).4.5.4 Unsupervised classification – Tab. 4.3Beyond reconstruction and canonicalization, we evaluate the usefulness ofour method via a classification task that is not related in any way to thelosses used for training. We compute the features from the auto-encodingmethods from Section 4.5.2 against those from our method (where we buildfeatures by combining pose with descriptors) to perform 13-way classificationwith two different techniques:554.5. Results• We train a supervised linear Support Vector Machine (SVM) on the ex-tracted features [168, Ch. 7];• We perform unsupervised K-Means clustering [168, Ch. 9] and then labeleach cluster via bipartite matching with the actual labels through theHungarian algorithm.Note the former provides an upper bound for unsupervised classification,while better performance on the latter implies that the learnt features areable to separate the classes into clusters that are compact (in an Euclideansense).Analysis of results – SVM Note how our method provides best resultsin all cases, and when the dataset is not unaligned the difference is large.This shows that, while 3D-PointCapsNet and AtlasNetV2 (with and with-out STN) are able to somewhat auto-encode point clouds in the unalignedsetup, what they learn does not translate well to classification. However, thefeatures learned with Canonical Capsules are more related to the semanticsof the object, which helps classification.Analysis of results – K-Means The performance gap becomes widerwhen K-Means is used – even in the aligned case. This could mean thatthe features extracted by Canonical Capsules are better suited for otherunsupervised tasks, having a feature space that is close to being Euclideanin terms of semantics. The difference is striking in the unaligned setup. Weargue that these results emphasize the importance of the capsule framework– jointly learning the invariances and equivariances in the data – is cardinalto unsupervised learning [169, 170].4.5.5 Ablation studyTo make the computational cost manageable, we perform all ablations withthe airplane category (the category with most instances), and in the unalignedsetup (unless otherwise noted). Please also see Sec. 4.6 for more ablationstudies.Losses – Tab. 4.4 We analyze the importance of each loss term, with theexception of Lrecon which is necessary for training. All losses beneficiallycontribute to reconstruction performance, but note how Lequiv, Llocalizationand Lequilibrium affect it to a larger degree. By considering our canonicaliza-tion metric, we can motivate this outcome by observing that the method failsto perform canonicalization when these losses are not employed (i.e. training564.6. More ExperimentsFull ¬Linvar ¬Lequiv ¬Lcanonical ¬Llocalization ¬LequilibriumCD 1.11 1.12 1.16 1.12 1.44 1.60Std 8.278 9.983 110.174 8.421 113.204 92.970Table 4.4: Effect of losses – Reconstruction performance, and canonical-ization performance when loss terms are removed.PointNet PointNet++ DGCNN ACNeCD 1.28 1.34 1.21 1.11Table 4.5: Backbone – Auto-encoding performance (Chamfer distance)when we use various permutation-invariant backbones.collapses).Encoder architecture – Tab. 4.5 Our method can be used with anybackbone, as our main contribution lies in the self-supervised canonicaliza-tion architecture. For completeness, we explore variants of our method usingdifferent back-bones: PointNet [23], PointNet++ [22], DGCNN [171], andACNe [17]. Among these, the ACNe backbone performs best; note that allthe variants in Tab. 4.5, regardless of backbone choice, outperform all othermethods reported in Tab. 4.1 with a large margin.4.6 More Experiments4.6.1 Additional ablation studiesAuto-encoding with aligned data – Fig. 4.3 and Tab. 4.1 For com-pleteness, we further show qualitative results for auto-encoding on an aligneddataset, the most common setup of prior works in the literature. As shownin Fig. 4.3, our method provides best reconstruction performance even inthis case; for quantitative results, please see Tab. 4.1. Interestingly, whileour decoder architecture is similar to AtlasNetV2 [4], our reconstructions areof much higher quality; our methods provides finer details at the propellerson the airplane, at the handle of the firearm, and at the back of the chair.This further supports the effectiveness of our capsule encoding.Number of points P – Tab. 4.6 To speed-up experiments we have mostly574.6. More ExperimentsInputOur capsuledecomposi-tionOur recon-struction3D-PointCapsNet [13]reconstruc-tionAtlasNetV2 [4]reconstruc-tionFigure 4.3: Auto-encoding / qualitative – Example decompositionresults using Canonical Capsules on the test set, with the aligned setup;we color each decomposition (capsule) with a unique color. For 3D-PointCapsNet [13] and AtlasNetV2 [4], rather than capsules, these corre-spond to “patches” in the reconstruction network. Our method clearly pro-vides the best qualitative results.1024 pts 2500 pts3D-PointCapsNet [13] 2.49 1.49AtlasNetV2 [4] 2.14 1.22Our method 1.76 0.97Table 4.6: Number of points P – Auto-encoding performance (Chamferdistance × 103) as we vary the input point cloud cardinality; aligned setupfor both training and testing, with all object categories.584.6. More ExperimentsAtlasNetV2 [4] 5 capsules 10 capsules 20 capsules2.80 1.25 1.11 1.15Table 4.7: Ablation study on the number of capsules – We show thereconstruction performance (Chamfer distance × 103) with varying numberof capsules. While they all perform better than competitors, 10 capsulesgive best performance. Note the representation power is kept constant aswe vary the number of capsules.Number of capsules 5 10 20CD 1.51 1.11 1.02Table 4.8: Number of capsules w/ fixed descriptor dimension percapsule – Better reconstruction performance is obtained as number ofcapsules are increased, since the network capacity expands proportionally.used P=1024, but in the table we show that our findings are consistent re-gardless of the number of points used. Note that the results of AtlasNetV2 [4]are very similar to what is reported in the original paper. The slight differ-ences exist due to random subsets that were used in AtlasNetV2.8Effect of number of capsules – Tab. 4.7 To verify how the number ofcapsules affect performance, we test with varying the number of capsules.We keep the representation power constant by reducing the dimension ofdescriptors as more capsules are used; for example, with 10 capsules we usea 128-dimensional descriptor, with 20 we use 64, and with 5, we use 256. Ourexperimental results show that representation with 10 capsules achieves thebest performance. Note that our method, even with the sub-optimal numberof capsules, still outperforms compared methods by a large margin.Increasing the number of capsules w/ fixed descriptor dimension– Tab. 4.8 We also report the performance for the increasing number ofcapsules while keeping descriptor dimension fixed. Note this is different fromTab. 4.7, as in that setting the overall network capacity was kept fixed, buthere it grows linearly to the number of capsules. As shown in the Tab. 4.8,unsurprisingly, more capsules lead to better reconstruction performance as8And to a minor bug in the evaluation code (i.e. non deterministic test set creation)that we have already communicated to the authors of [129].594.6. More ExperimentsAirplane Chair AllOne shot alignment 1.10 2.92 2.37Our method 1.11 2.58 2.22Table 4.9: One-shot canonicalization – Reconstruction performance ofour method against a naive one-shot alignment, where an arbitrary pointcloud is selected as reference.AtlasNetV2 [4] Ours (βk) Ours (β¯k)Aligned 1.28 0.96 0.99Unaligned 2.80 2.12 1.08Table 4.10: Canonical descriptors – Auto-encoding performance (Cham-fer Distance) of our method with descriptors βk vs. β¯k. Note the unalignedsetup is the one of primary interest.the network capacity is enhanced.One-shot canonicalization – Tab. 4.9 A naive alternative to our learntcanonicalizer would be to use one point cloud as a reference to align to.Using the canonicalizer provides improved reconstruction performance overthis naïve approach, removes the dependency on the choice of the referencepoint cloud, and allows our method to work effectively when dealing withmulti-class canonicalization.Canonical descriptors – Tab. 4.10 We evaluate the effectiveness of thedescriptor enhancement strategy described in Section 4.4.2. We report thereconstruction performance with and without the enhancement. Recomput-ing the descriptor in canonical frame helps when dealing with the unalignedsetup. Note that even without this enhancement, our method still outper-forms the state-of-the-art.w/o Lrecon w/ Lreconw/o Lcanonical w/ Lcanonical w/o Lcanonical w/ LcanonicalCD - - 1.12 1.11mStd 16.0 7.747 8.421 8.278Table 4.11: The impact of Lcanonical on canonicalization – Best canon-icalization is achieved when Lcanonical is used; see text for details.604.6. More Experiments(a) (b) (c)Figure 4.4: Random sampling of rotations – (a) Sampling Euler anglesuniformly results in a non-uniform coverage of SO(3); note we visualize onlyone-eighth of a sphere for ease of visualization. (b) Non uniform samplingresults in auto-encoding error to be biased w.r.t rotations (we use a cold-warm colormap to visualize the Chamfer distance error). (c) By properlysampling rotations [14], this bias can be alleviated.The impact of Lcanonical Using Lcanonical is required to achieve the bestperformance; see the inset table. However, the reconstruction loss Lreconalone is able to guide canonicalization up to some degree. It is worth notingthat excluding Lrecon, thus training the canonicalization component in astandalone fashion, unsurprisingly provides best mStd performance. Thus,this could be an alternative strategy to train our framework, should one notrequire end-to-end differentiability. Nonetheless, our observations still holdand our method delivers the best performance (including when compared toCompass [5]) with all losses enabled.Supervising the attention’s invariance Since θ is inferred by weightedaveraging of P with the attention map A in (4.2), we also considered di-rectly adding a loss on A that enforces invariance instead of a loss onθ. This variant degrades only slightly in terms reconstruction (CD=1.13,where ours provides CD=1.11) but performs very poorly when canonical-izing (mStd=94.906 vs mStd=8.278 of ours). We hypothesize that this isbecause Lequivariance directly supervises the end-goal (capsule pose equivari-ance) whereas supervising A is an indirect one.Random sampling of rotations – Fig. 4.4 Lastly, we revisit how rota-tions are randomly sampled to generate the augmentations used by Siamesetraining. Uniform sampling of Euler angles (i.e., yaw,pitch,roll) leads to anon-uniform coverage of the SO(3) manifold as shown in Fig. 4.4 (a). Dueto this non-uniformity, the reconstruction quality is biased with respect to614.7. ConclusionsRange of translation [−0.2, 0.2] [−0.4, 0.4] [−0.8, 0.8] [−1.6, 1.6]CD 1.11 1.12 1.1 1.1Table 4.12: Large range of random translations – The performancechanges are negligible as the magnitude of random translations increases.Method Bench Cabinet Car Cellphone Chair Couch Firearm Lamp Monitor Airplane Speaker Table Watercraft AllAligned 3D-PointCapsNet [13] 2.06 3.23 2.64 2.25 2.64 2.99 0.93 3.40 2.85 1.36 4.26 2.56 2.05 2.49AtlasNetV2 [4] 1.67 2.81 2.42 2.00 2.26 2.63 0.78 2.68 2.52 1.18 3.80 2.16 1.73 2.14Our method 1.44 2.37 2.10 1.77 1.90 2.26 0.59 1.61 2.09 0.99 3.04 1.80 1.31 1.76Unaligned 3D-PointCapsNet [13] 4.34 5.20 4.57 3.87 5.55 5.33 2.00 4.97 4.30 3.24 6.05 5.17 3.38 4.66AtlasNetV2 [4] 2.93 3.65 3.46 2.46 3.53 3.72 1.28 2.91 3.06 2.17 4.42 3.20 2.27 3.08AtlasNetV2 [4] w/ STN 2.44 3.30 3.10 2.17 2.93 3.28 0.96 2.56 2.70 1.60 3.96 2.67 1.86 2.60Our method 1.84 2.72 2.45 1.86 2.72 2.70 0.67 2.20 2.46 1.30 3.55 2.36 1.53 2.22Table 4.13: Auto-encoding / per-class quantitative – Performance interms of Chamfer distance with 1024 points per point cloud – metric ismultiplied by 103 as in [4].test-time rotations; see Fig. 4.4 (b). Instead, by properly sampling that fol-lows [14] where we fold Gaussian generation into unit quaternion generationefficiently, the reconstruction performance is much more uniform across themanifold; see see Fig. 4.4 (c) In our experiments, this leads to a significantdifference in auto-encoding performance; CD=1.11 with proper uniformsampling vs CD=1.19 with the Euclidean random sampling.Large range of random translations – Tab. 4.12We increase the rangeof random translations ([−0.2, 0.2] in the original paper). As shown in theTab. 4.12, we observe the negligible changes in reconstruction performance(CD) with larger random translations.4.6.2 Per-class results for auto-encodingIn addition to the auto-encoding results in Tab. 4.1, we provide per-classperformance for the models trained with multiple categories. As shown inTab. 4.13, we achieve the best performance for all classes.4.7 ConclusionsIn this work, we provide a self-supervised framework to train primary cap-sule decompositions for 3D point clouds. We rely on a Siamese setup thatallows self-supervision and auto-encoding in canonical space, circumvent-ing the customary need to train on pre-aligned datasets. Despite being624.7. Conclusionstrained in a self-supervised fashion, our representation achieves state-of-the-art performance across auto-encoding, canonicalization and classificationtasks. These results are made possible by allowing the network to learn acanonical frame of reference. We interpret this result as giving our neuralnetworks a mechanism to construct a “mental picture” of a given 3D object– so that downstream tasks are executed within an object-centric coordinateframe.Future work As many objects have natural symmetries that we do notconsider at the moment [4], providing our canonicalizer a way to encodesuch a prior is likely to further improve the representation. We perform de-composition at a single level, and it would be interesting to investigate howto effectively engineer multi-level decompositions [172]; one way could be toover-decompose the input in a redundant fashion (with large K), and use adownstream layers that “selects” the decomposition heads to be used [173].We would also like to extend our results to more “in-the-wild” 3D computervision and understand whether learning object-centric representations is pos-sible when incomplete (i.e., single view [147] or occluded) data is given ininput, when an entire scene with potentially multiple objects is given [174],or where our measurement of the 3D world is a single 2D image [160], or byexploiting the persistence of objects in video [175].Broader impact While our work is exclusively on 3D shapes, and thusnot immediately subject to any societal concerns, it enhances how ArtificialIntelligence (AI) can understand and model 3D geometry. Thus, similarto how image recognition could be misused, one should be careful whenextending the use of our method. In addition, while not subject to howthe data itself is aligned, the learnt canonical frame of our method is stilldata-driven, thus subject to any data collection biases that may exist –canonicalization will favour shapes that appear more often. This shouldalso be taken into account with care to prevent any biased decisions whenutilizing our method within a decision making AI platform.63Chapter 5Instance ProposalGeneration via BilateralFiltering: NeuralBF5.1 Problem StatementIn this chapter, we propose an iterative layer for 3D instance proposal gener-ation. While bounding box regression is commonly adopted in the 2D imagedomain and has achieved great success in top-down instance segmentation,it bottlenecks the performance in 3D instance segmentation. As shown inmy toy experiments – see Sec. 5.5.1, it’s hard to predict the precise bound-ing box even for a simple point cloud that is composed of simple primitives,leading to the inferior instance segmentation.We propose neural bilateral filtering to address this challenge. Given a querysampled from an input point cloud, our method predicts the affinity scoremap highlighting the queried instance. The affinity score map is composedof semantic and spatial similarity, thus proposing instances by consideringtwo aspects. The semantic similarity is simply an Euclidean kernel definedon feature space. And our spatial kernel is a hyper coordinate network basedon the convex hull. Given a query feature, we utilize an MLP to predict aset of hyperplanes, constructing a convex hull that can isolate the queriedinstances. Different from existing Euclidean kernels that are isotropic, ourconvex hull is a non-isotropic kernel which allows for separating two nearbyinstances of the same semantic class. We iteratively refine the instanceproposal with the spatial and semantic kernels, thus dubbing our method asneural bilateral filtering.Our method achieves the state-of-the-art top-down instance segmentationof point clouds. And key to success is the spatial kernel, an effective neuralfield for proposal generation. Evidently, it outperforms the bounding box645.2. Backgroundin point cloud learning. More importantly, thanks to its explicit nature, it’smemory and computation efficient. This allows our method to act as a smallmodule in a large instance segmentation pipeline.5.2 BackgroundInstance segmentation is a critical component of semantic 3D understanding,with applications including robotic manipulation [176–180] and autonomousdriving [181–185]. An essential step of instance segmentation [186–190] is togenerate a set of reliable instance proposals. For natural images, state-of-the-art methods generally follow the top-down paradigm [188, 189], whereone first detects candidate instance proposals and then prunes them via non-maximum suppression (NMS). Conversely, bottom-up methods [191, 192]learn per-point embeddings that are then used to cluster points into a disjointset of proposals.It is quite surprising that the dominance of top-down methods in natural im-ages (2D) is not reaffirmed when we change our domain to point clouds (3D),where bottom-up methods dominate public leaderboards [193, 194]. Whilethey perform well, bottom-up methods rely heavily on hand-crafted heuris-tics in the clustering step, such as the specification of spatial distance thresh-olds [190] and average instance sizes [194]. Still, because of the performancegap recent 3D computer vision literature naturally focused on incrementalcontributions attempting to improve the performance of bottom-up tech-niques, leaving top-down methods relatively under-investigated. Therefore,this prompts consideration of the reasons behind the notable differences inthe approaches to 2D versus 3D instance segmentation. Furthermore, itraises the question of whether it is feasible to develop a competitive top-down method.In this work, we argue that a critical bottleneck exists in the proposal gen-eration process for point clouds. Early works follow a similar process as fornatural images, where bounding boxes are regressed [55, 195–197], but thisregression does not generally lead to sufficiently accurate proposals. We ab-late these techniques on a simple synthetic dataset, demonstrating how theylead to weak performance (i.e. mAP<50%), while we achieve near-perfectresults.Our top-down technique generates the proposal associated to a given queryon the input point cloud; see Fig. 5.1. We encode a proposal as an affinityscore: a [0,1] point-wise labeling of the point cloud that is conditional on a655.2. BackgroundFigure 5.1: Teaser – (top) Given a query point, we define the correspondinginstance as the element-wise product of feature (points with similar class)and spatial (points with similar localization) affinities; this leads to for-mulating instance segmentation as a neural bilateral filter. (bottom) Ourtechnique can be applied to large-scale instance segmentation of real-worldScanNet scenes, leading to the best-performance-in-class amongst top-downmethods.665.2. Backgroundquery point (i.e. as the query point changes, the affinity scores changes). Wedraw inspiration from bottom-up methods [193], and determine two pointsbelong to the same instance if they are “close” to each other in both space andsemantic class; see Fig. 5.1. Unlike bottom-up methods which would utilizethe dual-space affinity to group all points into distinct clusters [190, 193],our proposal generation step identifies the points that are affiliated witha given query. The semantic affinity compares the similarity of semanticfeatures in order to separate points from distinct object types. The spatialaffinity is responsible for bounding the spatial extent of the instance in orderto separate semantically-similar objects from each other. Hence, query-conditional affinity can be factorized in two terms, leading us naturally to aneural bilateral filter formulation.In representing spatial affinity, we note the predominant representation em-ployed in the 2D image domain axis-aligned bounding boxes. And whileparameterizing spatial affinity with 3D bounding boxes is possible, this ei-ther requires careful handling of SE(3) equivariance [60, 198, 199], or carefulprediction of rotations [200]. We avoid this issue by introducing the use ofdifferentiable convex hulls [151] for instance proposal. Note that convexhulls are universal approximators of bounding boxes, capable of implicitlymodelling rotated bounding boxes. From a different standpoint, our tech-nique, which models convexes as the level set of a field, can be viewed asthe first method that attempts to apply the rapidly growing area of Neu-ral Fields [201] to 3D instance segmentation.Contributions We validate the effectiveness of our method on both syn-thetic and real datasets leading to the following contributions:• we introduce a simple synthetic dataset that reveals a major bottleneckin instance proposal generation for point clouds;• we pose the problem of instance segmentation as generating the affinityof points in the cloud to a query point;• we formulate the computation of affinity as a neural bilateral filter, anddemonstrate how an iterative formulation improves its performance;• we introduce the use of coordinate networks representing convex domainsto model the spatial affinity in our neural bilateral filter;• collectively, these contributions results in a method that tops the leader-board in point cloud instance segmentation on ScanNet amongst top-down methods.675.3. Related works5.3 Related worksWe briefly describe the recent works on 2D and 3D instance segmentationand review methods on mean shift and bilateral kernel. For a survey on3D instance segmentation, please refer to [176], and to [202] for 2D instancesegmentation.2D instance segmentation Top-down methods [186, 187] predict redun-dant instance proposals for sampled locations in images, which typicallyrequires NMS to remove the overlap. Mask-RCNN [186] detects a set ofbounding boxes as the initial instance proposals, and then applies a seg-mentation module and NMS to output the final mask. PolarMask [187]enhances the performance by using “center priors” – locations close thecenter of object tends to predict better bounding boxes. SOLO [188, 189]predict instance masks for every location, obviating the need of segmenta-tion module. This is similar to our method where we also output instancemasks without segmentation module. Other mainstream instance segmenta-tion pipelines [191, 203] follow the bottom-up paradigm clustering pixels intosegments as instance proposals, resulting in performance typically inferiorto that of top-down methods.3D instance segmentation In contrast to the 2D image domain, bottom-up methods dominate 3D instance segmentation benchmarks. PointGroup [190]first labels points with semantic prediction and center votes, and then clusterpoints into segments as the instance proposals. Follow-up works [193, 194]further enhance the clustering method in different aspects. HAIS [194]develops hierarchical clustering to have better instance proposals. Soft-Group [193] proposes to group points using soft semantic scores and intro-duces a hybrid top-down/bottom-up technique via a proposal refinementmodule. While bottom-up methods rely on the heuristics such as objectsizes and distance threshold, top-down methods largely lag in performance.Top-down methods [55, 197] rely on precise bounding box prediction as theinitial instance proposal. In more detail, 3DBoNet [55] directly predicts afixed set of 3D bounding boxes, while GSPN [197] proposes a synthesis-and-analysis strategy to predict better bounding boxes.Neural Bilateral Filtering The idea of combining bilateral filtering withneural networks has been mostly in the context of filtering and enhancingnatural images [204–206]. However, to the best of our knowledge, learnedbilateral filtering has not been applied to the context of 3D point clouds, al-685.4. Method – Fig. 5.2Figure 5.2: Overview – (a) Given a query (⊙), we regress the boundinghull of the corresponding instance; (b) Together with semantic segmentation,this defines an affinity function on the entire point cloud; (c) This affinitycan be threshold to generate a candidate instance proposal; (d) Instanceproposals are then grouped by non-maximal suppression to generate thescene’s instance segmentation.though classical point cloud processing layers for point clouds do exist [207].5.4 Method – Fig. 5.2Given a point cloud of N points in D-dimensional space P={pn} and cor-responding C-dimensional features F={fn}=F(P;θF ), computed by a deeplearning backbone with learnable parameters θF , we generate instance pro-posals by regressing the bounding volume (i.e. a convex hull in RD) cor-responding to the instance of a query point (pq, fq), where q∼[1, N ] is theindex of query. Together with segmentation features, bounding volumes im-ply an affinity A∈RN between the query (pq, fq) and the whole point cloud,which can be thresholded to generate an instance proposal (Section 5.4.1).These instance proposals are then aggregated by classical non-maximumsuppression (NMS) to generate the desired instance segmentation.5.4.1 Affinity definitionAs illustrated in Fig. 5.1, the affinity of points in the point cloud to aquery (pq, fq) can be intuitively defined as the element-wise product of twoaffinities:• Affinity in feature space: whether a point in the point cloud belongsto the same class as the query;• Affinity in geometric space: whether a point in the point cloud be-longs to the same spatial region as the query.695.4. Method – Fig. 5.2Figure 5.3: Spatial similarity – The semantic feature is uninformative inseparating the two instances: (left) an isotropic affinity kernel w.r.t. thequery point would mistakenly assign points on the left instance to the rightone, regardless of bandwidth choice; (right) a non-isotropic kernel does notsuffer this shortcoming.More formally, we define our affinity function A(q) as:A(q) = Ap(q)⊙Af (q), (5.1)Af (q)[n] = exp(−τf · Kf (q, n)), (5.2)Ap(q)[n] = exp(−τp · Kp(q, n)), (5.3)where ⊙ is the element-wise product, [n] indexes the n-th element of thearray, and τ are hyperparameters controlling the bandwidth of the kernels.We can then learn the parameters of kernels Kf and Kp, whose internalsare provided in what follows, by directly attempting to reproduce the targetaffinity given a randomly drawn query point:Eq∼[1,N ]∥∥∥A(q)−Agt(q)∥∥∥21. (5.4)Kf – semantic similarity We measure whether two points have similarsemantic classes via:Kf (q, n) = ∥ϕ(fq;θϕ)− ϕ(fn;θϕ)∥22 . (5.5)where ϕ(·;θϕ) is a small projection layer with parameters θϕ that extractssemantic similarity features from the (task agnostic) backbone features f .Kp – spatial similarityWhile classical bilateral filtering employs isotropickernels to account for spatial similarity (i.e. gaussian with tunable band-width), this is not optimal for instance segmentation. We illustrate our705.4. Method – Fig. 5.2intuition in Fig. 5.3, where the proximity of two objects of the same seman-tic class implies that no isotropic kernel centered at the query point couldbe used to isolate the desired instance. We achieve this while retainingcommutative symmetry9:Kp(q, n) = C(pn−pq;ψ(fq;θψ)) + C(pq−pn;ψ(fn;θψ)), (5.6)where ψ(·;θψ) is a small projection layer with parameters θψ that extractsspatial similarity features from the generic backbone features f . This leads usto the question of how to design the function C(x; f). One potential solutionis to define C as a coordinate neural network [201] whose shape is describedby the feature f , and that is evaluated at location x. We opt to model C withCvxNet [151] – a coordinate neural network that is a universal approximatorof convex domains. This choice is particularly well-suited, because:• convex hulls are a topologically equivalent, yet more flexible and detailedreplacement for 2D/3D bounding boxes, the core representation employedin 2D/3D object detection/instance segmentation, making them a par-ticularly well-suited choice for our problem;• compared to coordinate neural networks implemented as multi-layer per-ceptrons, CvxNet-like hyper-networks generate very small output net-works and are more memory efficient, allowing us to use larger mini-batchsizes leading to faster training.We further detail the design of C in Section 5.4.2, which will fulfill thefollowing base property with respect to the convex domain specified by thefeature f :C(x; f){= 0 if x inside convex defined by f ,> 0 otherwise (≈ boundary distance). (5.7)5.4.2 Convex parameterization C(x; f)From f , via a fully connected decoder (with shared parameters θD), wederive the normals {nh∈RD | ∥nh∥2=1, h∼[1, H]} specifying the H half-space orientations, and their distances {dh∈R+} from the origin o∈RD:o, {nh}, {dh} = D(f ;θD), (5.8)9Affinity ought to be symmetric, because if point pn belongs to the same instance aspq then we should ideally have K(q, n) ≡ K(n, q).715.4. Method – Fig. 5.21: Input:2: q ∈ [1, N ] ▷ query index3: P ∈ RN×D ▷ (const) cloud positions4: F ∈ RN×C ▷ (const) cloud features5: function NeuralBilateralFilter6: p(0) = pq = P[q]7: f (0) = fq = F[q]8: for t = 1, . . . , T do9: A(t−1)(q) = A(p(t−1), f (t−1))10: A(t−1)(q) = A(t−1)(q)/∥A(t−1)(q)∥111: p(t) = A(t−1)(q) ·P12: f (t) = A(t−1)(q) · F13: end for14: Return A(T )(q)15: end functionFigure 5.4: Neural Bilateral Filter (Section 5.4.3)– Given a query(position and feature) we iteratively apply the learned filters to advect thequery point position and features. Ultimately, the final attention A(T )(q) isused for downstream tasks.and define the distance of x from the h−th hyperplane as:Hh(x) = nh · (x+ o) + dh, (5.9)which can be assembled into an (approximate, see [151]) distance functionfrom the convex polytope as:Φ(x; f) = maxh{Hh(x)}, (5.10)finally leading to our convex spatial proximity:C(x; f) = max(Φ(x; f), 0), (5.11)which can then, if necessary, be converted as an indicator function (i.e.occupancy) for a convex [151]:O(x; f) = sigmoid(−Φ(x; f)). (5.12)725.4. Method – Fig. 5.25.4.3 Neural Bilateral Filter – Fig. 5.4The resemblance of (5.1) to the product of kernels in bilateral filtering [35,208] inspired us to investigate the use of iterative inference. Specifically,given a query, we advect both query position and features, where the ad-vection weights are given by the affinity definition from (5.1). Note thatthe point cloud P and corresponding features F remain unchanged, onlythe query is affected. With a slight abuse of notation, we denote A(t)(q) asthe affinity at t-th iteration for the query q. The outcome is simply that,rather than attention A(0)(q)=A(q) in downstream processing, A(T )(q) willinstead be used.5.4.4 TrainingTo train our network, we optimize:argminθϕ,θψ ,θD,θFLaffinity + Lsem + Lpoly + Lshift. (5.13)Of these losses Laffinity is our core loss, while the rest provide “skip-connection”supervision to the network to facilitate learning. Since our method performsiterative inference, we discount (α=0.8) contributions of later iterations – wefound empirically that focusing on later iterations cause training instability:Laffinity = Eq∼[1,N ]T∑t=1αt∥∥∥A(t)(q)−Agt(q)∥∥∥21(5.14)Semantic supervision To encourage the semantic features in (5.5) torepresent only the semantic similarity, we inject semantic information bymapping intermediate point-wise backbone feature to semantic logits (seeFig. 5.5), and supervising with ground truth labels sgtq :Lsem = Eq∼[1,N ][CrossEntropy(sq, sgtq )](5.15)Instance centroid supervision To minimize the learning complexity ofD, we incentivize predicted convex hulls to be expressed with respect to astable coordinate frame10. We employ the ground truth instance origin cgtq10If two points a and b belong to the same instance, then the predicted convex originoa ≡ ob, and the same half-space configuration can be used for all queries within aninstance; Note this is similar to the coordinate frame normalization in NASA [209].735.4. Method – Fig. 5.24P<latexit sha1_base64="EDHZSJ4Gt+X3HhAexE56ukebMlo=">AAAHj3ichVXbb [binary]</latexit><latexit sha1_base64="tT/n4F6lpoyikou86YXRSoJmx7w=">AAAHj3ichVXbb [binary]</latexi t> <latexit sha1_base64="yQM2NWzeFCUI9IjrReXg1svu28M=">AAAHqnichVVRj [binary] 8J+vlfwLOz7fRA==</latexit>Kp Kf<latexit sha1_base64="yQM2NWzeFCUI9IjrReXg1svu28M=">AAAHqnichVVRj [binary] 8J+vlfwLOz7fRA==</latexit>Kp KfNBF<latexit sha1_base64="W+evF7ew9XETTUgrlbjPuIDx06g=">AAAHjnichVVdb [binary]</latexi t>sq<latexit sha1_base64="kKL5kCuWrFL Awq+6roMy1k4UyAM=">[binary] i5DtH8AfO5w+7QpIT</latexit>S1 <latexit sha1_base64="vf/ZnI11tWZxOTiTNwF434cycDw=">AAAB9HicbVDLS [binary]</latexit>S2<latexit sha1_base64="Ri7xvkBpo5M 9c7nif026pExhb10=">[binary] 2VdueUpeXZjEPvT++2v5vNbbWVw==</latexit>f˜q……<latexit sha1_base64="dlC30sQKkQwb4Owwf/iNO2uQ1fk=">AAAHlHichVXbj [binary] </latexit>f (0)q<latexit sha1_base64="wIjr5IcfM8c1aA/n8UXpcgHz3pk=">AAAHlHichVXbj [binary] </latexit>f (T )qUNetFigure 5.5: Architecture – The point cloud is processed by a backbone toproduce fq=f (0)q , which is then processed by a neural bilateral filter by ourkernels K∗. We supervise sq via ground truth semantic classification labels,where S1 is a 2-layer MLP, and S2 is a linear layer.and supervise the predicted origin relative offset:Lshift = Eq∼[1,N ]T∑t=1αt∥∥∥(pq + o(t)q )− cgtq ∥∥∥1 (5.16)where the offset o(t)q is computed from D(f (t)q ).Convex occupancy supervision Note the affinity supervision in (5.14)only penalizes points that are incorrectly marked as outside the convex hull.To correct this, let Ogtq (p) be the ground truth occupancy of point p belong-ing to the convex hull of query q, we then penalize:Lpoly = EqEnT∑t=1ωq,nαt∥O(pn−pq;ψ(f (t)q ))−Ogtq (p)∥22 (5.17)where ωq,n is a term to control class imbalance: if the instance correspondingto q has Q points and the scene has N points, then ωq,n=1/Q if point nbelongs to the instance, and ωq,n=1/(N−Q) otherwise.5.4.5 Implementation detailsWe briefly discuss the core implementation details.745.4. Method – Fig. 5.2Network architecture For the backbone we utilize the U-Net-like back-bone in [190, 194] which is implemented with sparse convolution [31]. Weset the dimension C of the backbone feature f to 32 as in [194].The projection layers ϕ(·;θϕ) in (5.5) and ψ(·;θψ) in (5.6) The layerϕ is composed of semantic layers (S1) and an embedding layer (S2). Thesemantic layers convert the backbone features into semantic scores with atwo-layer MLP with 32 neurons and then outputs the semantic feature witha linear layer of C neurons. Note that during the iterative process, wedirectly update the query’s semantic feature without re-using the semanticbranch. The embedding layer is a linear layer of C neurons. For ψ we use asmall projection layer and rely on the D for reasoning about the 3D convexpolytopes. Specifically, we use a simple identity mapping layer as the ψ,which we found to be good enough.The polytope network D(·;θD) in (5.6) The network D consists of twoMLP blocks. The first block – a two-layer ReLU-activated MLP with 128neurons – predicts o from the query feature f and a residual to f . We thenadd the residual to f and utilize the second block – a three-layer ReLU-activated MLP with 128 neurons – to predict normals and offsets. Forpredicting the plane offset dh, we use the strategy from [210] and discretizethe offset values into 32 equal bins in the range [0, 8] meters, and obtain thepredicted value via the weighted sum of classification scores. We representeach 3D convex polytope with twelve planes, striking a good balance betweenprecision and computational load, which linearly increases with the numberof planes. Finally, τF=1 and τP=50 in (5.2) and (5.3).Forming the training batch While possible, training with all points inthe point cloud is impractical and inefficient, as it would create a quadraticincrease in both memory and computation. We use a batch of four scenes,and randomly sample 32 random points/scene during training to form asingle training sample. Our algorithm has the computation and space com-plexity linear to the number of sampled queries. We further set the numberof mean shift iterations T=2, which we ablate in Sec. 3.5.2.Training As in RAFT [211], we detach the gradient flow between differentiterations to stabilize training. We use the Adam optimizer [212] with cosineannealing for the learning rate [213], with 0.001 as the initial learning rate.We further follow standard data augmentation/voxelization schemes of ex-isting instance segmentation methods [194]. Coefficients for all loss termsare set as one.755.5. ResultsLine segment Circle AveragemAP AP50 AP25 mAP AP50 AP25 mAP AP50 AP25BBox 46.4±1.1 67.7±1.9 69.8±1.1 21.2±1.4 54.7±2.3 90.6±0.3 33.8±0.9 61.2±1.7 80.2±0.7BBox w/ center 54.1±1.6 77.9±1.5 80.4±1.2 28.0±0.8 64.0±0.7 89.2±0.7 41.0±1.0 71.0±0.7 84.8±0.8BBox + GT filtering 53.9±1.4 68.2±1.6 69.0±1.1 31.9±1.9 71.1±1.7 91.7±0.5 42.9±1.2 69.7±1.2 80.3±0.5BBox w/ center + GT filtering 65.3±1.7 79.3±1.5 80.1±1.5 41.4±1.1 75.4±1.4 90.3±0.5 53.3±1.1 77.3±0.6 85.2±0.9Ours 95.9±0.3 97.6±0.4 97.9±0.3 98.2±0.5 98.9±0.3 99.3±0.3 97.1±0.2 98.3±0.2 98.6±0.1Table 5.1: Query-conditioned instance proposal generation – werandomly sample a single query for each instance and generate the non-overlapped proposals. We report the mean and standard deviation of aver-age precision by running the evaluation pipeline five times.Non-maximum suppression To obtain the final instance segmentationresults for the ScanNet dataset we use standard non-maximum suppres-sion [188, 214] to remove redundant proposals. In more details, we visit aqueue of input candidate proposals in confidence score order; see Sec. 5.5.2.For each candidate proposal, we compute the IoU with all other candidatesand merge/prune those that have IoU higher than 0.25.5.5 ResultsIn our results section, we:• Sec. 5.5.1 – validate our method in a controlled synthetic setup, where weshow that current proposal generation methods have limited effectiveness;• Sec. 5.5.2 – demonstrate the potential of our method in a more complexinstance segmentation pipeline on the real-world ScanNet dataset [8];• Sec. 3.5.2 – perform an ablation study.5.5.1 Synthetic datasetWe create a 2D synthetic dataset composed of lines, circles, and randomnoise; see Fig. 5.6. For each scene, we randomly place 16 primitives sampledfrom a large pool (10k in total) of randomly generated line segments andcircles in a 2D space. We sample 4096 points for foreground instances and512 points for the background noise. To keep a similar point density forinstances of different sizes, we make the number of points for each instanceproportional to the length of the primitive instance. We generate thesescenes on-the-fly while training and keep 100 scenes for testing. We limitthe 2D coordinates to be within [−4, 4] to match the typical size of ScanNetscenes, allowing us to reuse the same backbone across both synthetic and765.5. Resultsreal scenes.MetricsWith the dataset, to show that instance proposals are a bottleneck,we are interested in their direct evaluation without any downstream Non-Maximum Suppression (NMS) heuristic. We randomly select a single pointfor each instance in the point cloud and measure the quality of the generatedproposal for the selected point. Once the proposals are provided, we usethe standard metrics used in the ScanNet benchmark [8]—AP50 and AP25,which are the accuracy computed with the intersection-over-union (IoU)threshold of 50% and 25%, respectively, and mAP, which is the average APover different thresholds ranging from 50% to 95% with the step size of 5%.Baselines A commonly-used baseline is to directly predict the boundingbox for each instance, within which post-processing is applied [55, 195–197].To do so, similarly to GICN [195], we train a 2-layer MLP that predicts thebounding box, parameterized by its two corners relative to the query. Wefurther compare against VoteNet [215], where one first regresses a spatialoffset given a query point and then regresses the bounding box corners rel-ative to the offset. For these baselines, the bounding box often containspoints from noise or other classes (lines vs circle), so we utilize the semanticpredictions from the backbone to filter those points out of each instanceproposal. Clearly, this would not perfectly filter out cases where the sameclass instances overlap; hence, we further propose an oracle baseline whichuses ground-truth semantic and instance labels as an oracle for filtering, thusemulating an ideal post-processing step for the methods based on boundingboxes. We train with 10k iterations, which is enough for all methods toconverge on this simple dataset.Results – Tab. 5.1 and Fig. 5.6 Our method outperforms the baselinesby a large margin. Despite the success of bounding box proposals in 2Dimages, these method achieves a surprisingly low performance on this simplesynthetic dataset, even when ground-truth filtering is employed. On theother hand, our method delivers near-perfect results, as one would expectfor such a simple dataset. For the baselines, as shown in the examples inFig. 5.6, we find that many of the proposals are slightly off, with some beingcompletely off. While small errors in bounding box position/size are notcritical for detection in 2D images, they can be catastrophic for point cloudssampled from the object surface near the bounding box exterior, where asmall misalignment could remove entire sections of geometry. For example,in Fig. 5.6 top-left, the bottom right circle is detected with a bounding box775.5. ResultsBBox BBox w/ center prediction OursFigure 5.6: Qualitative/Synthetic – Our method generates nearly perfectquery-conditioned instance proposals while the baseline is limited by thenoisy bounding box. Note the red large points are the sampled queries. Wecolor the points detected by different instance proposals. And the blackpoints are the background points or the points detected by more than twoinstance proposals.that would be considered accurate should one consider only the boundingbox, but the majority of the point cloud points for this circle lie outside, asthe box is slightly smaller than the actual circle.Visualizing the spatial kernels – Fig. 5.7 We visualize the learnedspatial kernel. As shown, the learned spatial kernel forms a polytope thattightly bounds the instance in question as desired. These learned kernels en-able our method to easily separate different instances spatially, even withoutconsidering semantics. Such easy separation would not be possible, for ex-ample with standard Euclidean distance as points far away from each otheron the line or on the circle would be confused with other nearby points.785.5. Results2Figure 5.7: Visualizing the spatial kernels – Our method learns theeffective convex hulls that act as the tight bounding box of the target in-stance. For each convex hull, the magenta lines are the learned half-planes.The red polygon is the intersection between half-planes. Points are coloredwith spatial similarity where red means larger similarity while blue meanssmaller similarity.5.5.2 Instance segmentation on ScanNetV2The ScanNetV2 [8] dataset consists of 1613 scenes in total with 1201, 312,and 100 scenes dedicated for training, validation, and testing, respectively.We use the standard evaluation pipeline and report the standard metrics,the same ones as the ones used for the 2D synthetic data. To evaluate ourmethod for top-down instance segmentation pipeline for point clouds, weintroduce basic post-processing steps that are commonly used in the litera-ture [188, 189, 195, 218], as well as a scoring function to provide confidencescores for each instance proposal, as required by the benchmark protocol.Notably, our post-processing steps are relatively simple compared to trickslike “matrix NMS” [189] and query sampling using “center priors” [195, 218].Specifically, we first segment out all background points using ϕ(fi) in (5.5).We then sample 256 query points from the predicted foreground points andgenerate 256 instance proposals. When sampling queries, we apply far-thest point sampling [45] to ensure maximum coverage. We then remove re-795.5. ResultsMethods Validation TestmAP AP50 AP25 mAP AP50 AP25Bottom-upPointGroup [190] 34.8 56.7 71.3 40.7 63.6 77.8SSTNet [216] 49.4 64.3 74.0 50.6 69.8 78.9HAIS [194] 43.5 64.4 75.6 45.7 69.9 80.3Mix Dyco3D [217] 35.4 57.6 72.9 39.5 64.1 76.1SoftGroup [193] − 67.6 78.9 50.4 76.1 86.5Top-down3D-SIS [196] − 18.7 35.7 16.1 38.2 55.8GSPN [197] 19.3 37.8 53.4 − 30.6 −3D-Bonet [55] − − − 25.3 48.8 68.7Ours 36.0 55.5 71.1 35.3 55.5 71.8Table 5.2: Quantitative/ScanNetV2 – instance segmentation bench-mark; our method provides the top performance for the top-down category.For looser thresholds our method performs slightly worse, which may beimproved with advanced post-processing.dundant instance proposals by applying Non-Maximum Suppression (NMS)to instance proposals with an IoU threshold of 30%. We train the entirepipeline end-to-end for 500 epochs as in [194].Confidence scores As the benchmark protocol requires instance propos-als to have associated confidence scores, we provide a confidence score foreach proposal based on both the semantic segmentation score (provided bysq) and an MLP that is trained to regress the IoU of each proposal withrespect to ground-truth. Specifically, we train a two-layer MLP with an ℓ1loss for the IoU. The final confidence score for each proposal is computed bymultiplying the regressed IoU value and the average semantic segmentationconfidence. We also use these confidence scores and NMS to filter our back-ground points, which typically have low foreground semantic confidence.Dropping low-confidence proposals In addition to the above, we dropproposals that have low confidence values (i.e. proposals with semantic con-fidence lower than 0.1, or with estimated IoU less than 0.2). Furthermore, wedrop proposals that have different predicted labels for the proposal and thequery point. These proposals are from points that are often located wheretwo different instances of different classes meet, and hence are unreliable.State-of-the-art comparisons – Tab. 5.2 and Fig. 5.8 Our methodshows promising results compared to the state-of-the-art. Among purely805.5. Results5Figure 5.8: Qualitative/ScanNet – Instance segmentation results on testset.top-down methods, our method achieves top performance validating theeffectiveness of our instance proposals generation. We leave further im-provement via better post-processing steps for future work. While ourmethod performs worse than the most recent bottom-up methods or hybridmethods [193], we note that these are methods heavily fine-tuned to achieveSOTA benchmark results, whereas ours is not. Note that our top-downmethod beats the leading bottom-up method that was SOTA around CVPR2020 [190]. Considering that top-down methods have intriguing properties(e.g., their dominant performance in image benchmarks [188, 189] and bet-ter generalization ability), and are worthwhile to explore further, we believeour work provides progress for instance segmentation on point clouds.5.5.3 AblationsNumber of iterations – Fig. 5.9 Our algorithm is capable to shift querypoints to the center of each instance in just two iterations. This leads to815.5. Resultsnum_iter 1 2 3 4mAP 94.46±0.55 96.80±0.35 96.54±0.40 96.18±0.6810-th iter 1-st iter 2-nd iterFigure 5.9: Ablation: number of iterations T – (top) We report theaverage and standard deviation of mAP by repeating the experiment 5 times.(bottom) Our algorithm shifts the queries (red points) to the centroid aftera small number of iterations.queries from the same instance to share a similar coordinate frame, leadingto a reduction of representation complexity as noted in NASA [209]. This isbeneficial, as a smaller number of iterations reduces the GPU memory loadof training. Training with more than two iterations seems to simply causetraining to become less stable and introduces slight performance degrada-tion.Losses – Fig. 5.10 With the proposed regularizers, our algorithm learnstighter instance polytopes (w/ Lpoly and Loffset) and semantic similarity (w/Lsem), leading to largely improved performance. Note that, since we evaluateAP for each the semantic category, we provide ground-truth semantic labelfor the models trained without semantic prediction (i.e., w/o Lsem), Finally,note also that even without Lpoly and Loffset, our algorithm can still learnpolytopes that roughly segment instances.5.5.4 Visualization of convex hulls in ScanNet – Fig. 5.11We further illustrate the learned convex hulls in ScanNet. As shown inFig. 5.11, the learned convex hulls act as the rough bounding box of thequeried instance.825.5. Resultsloss w/o Lpoly w/o Lshift w/o Lsem FullmAP 95.42±0.79 95.38±0.34 95.06±0.62 96.80±0.35Figure 5.10: Ablation: losses – We report the average and standard de-viation of mAP by repeating the experiment 5 times.5.5.5 More qualitative results in ScanNet – Fig. 5.12We further provide more qualitative results in Scannet.5.5.6 Interactive 3D visualizationWe further provide the interactive visualization of convex hulls and instancesegmentation. It’s available at our project website: https://neuralbf.github.io.835.5. ResultsFigure 5.11: Visualization of convex hulls on ScanNet – Note wevisualize only some of the hulls to reduce visual density. We visualize thewireframe of the hulls for better visibility. And we color the points accordingto the instance prediction.845.5. ResultsInput Prediction GroundtruthFigure 5.12: More qualitative results on ScanNet – Instance segmen-tation results in the validation set.855.6. Conclusions5.6 ConclusionsWe have proposed an instance proposal method for point clouds. We formu-late instance proposals as a query-conditioned attention model and employneural bilateral filtering to provide much more accurate proposals than di-rect regression. We demonstrate through synthetic data that the proposalgeneration process is indeed a bottleneck, which our method can largely im-prove. We further demonstrate the potential of our method on the ScanNetdataset, achieving competitive performance amongst top-down methods.Limitations and Future works While we have shown clearly that a bot-tleneck exists, and that it can be avoided, its benefit has not been as strik-ingly revealed when compared to pipelines that are carefully engineered forinstance segmentation. We believe there is much room for this potential tobe realized, similar to how top-down methods are the dominant strategy fornatural images [188, 189].86Chapter 6Multi-scale Point Rendering:PointNeRF++6.1 Problem StatementIn this chapter, we study the explicit point-based representation for recentpopular rendering based on the neural radiance field (NeRF). First of all,point-based NeRF and its follow-ups have already shown the advantages ofexplicit representation in rendering. Nevertheless, it’s limited in the need fora good initial point cloud and hand-crafted heuristics for adding or deletingpoints. Specifically, when there are no neural points near the query points,the queries would be invalid and skipped. However, skipped queries are pos-sibly from the surface of objects, potentially causing incomplete rendering.We propose to address this issue with a simple multi-scale strategy. Given aninitial point of low quality, we construct the hierarchical point cloud struc-ture. Given a query, we interpolate features from different levels and thenfuse the information via mean pooling. At the coarsest level, it’s a globalNeRF, guaranteeing that the query has at least one valid level. At the finestlevel, it’s the original point cloud, defining an explicit point-based NeRF.Therefore, our method allows us to seamlessly integrate typical NeRF andexplicit point-based NeRF, thereby combining the advantages of both NeRFand point-based NeRF. The explicit point-based NeRF achieves sharperrendering than typical NeRF and typical NeRF guarantees the completerendering. We validate our method in different types of point clouds anddemonstrate the state of art rendering especially when the initial point cloudis of lower quality.876.2. BackgroundFigure 6.1: Teaser – We introduce a novel volume-rendering framework toeffectively leverage point clouds for Neural Radiance Fields. Our formulationaggregates points over multiple scales—including a global scale governing theentire scene, equivalent to the standard, point-agnostic NeRF. Our solutionleads to much better novel-view synthesis in challenging real-world situationswith sparse or incomplete point clouds. Here, we show example renderingsfrom the KITTI-360 test set.6.2 BackgroundWith the introduction of Neural Radiance Fields (NeRF) [6], the quality ofnovel-view synthesis from a collection of images has increased dramatically.However, the problem is far from solved when field-of-view overlaps sparselyamongst cameras [219–221], which makes them difficult to apply to manyuncontrolled, real-world scenarios. Researchers have attempted to solve thisproblem in various ways, including content-based regularization [220], patch-based regularization [222], image features [219], or diffusion priors [223, 224].One way to address this issue is to leverage point clouds obtained from addi-tional sensors and/or photogrammetry [9, 225, 226]. The use of point clouds886.2. Background(as a representation) for neural rendering was pioneered by PointNeRF [9],which demonstrated that point clouds can indeed help achieve higher-qualityrenderings. However, as we demonstrate through experiments, the benefitsof PointNeRF diminish when point clouds are sparse and/or incomplete.This is often the case in most real-world applications, such as for point cloudsobtained by LiDAR scanners in autonomous-driving datasets [7, 227–232].We posit that this shortcoming is mainly due to a missing key element: thelack of multi-scale modeling within the architecture of PointNeRF. Multi-scale modeling is helpful in point cloud processing, as small ‘holes’ (regionswithout points) can often be naturally filled-in via multi-scale aggregation.We liken this intuition to that followed by two seminal papers in pointcloud semantic understanding—PointNet [44] and PointNet++ [22]—wherethe latter improved upon the former by simply introducing a multi-scalenetwork design, and the notion of hierarchical structure.In this work, we introduce a simple multi-scale representation for pointcloud-based rendering. Specifically, we aggregate point clouds at variousscale levels, defined as voxel grids (Sec. 6.4.1), up to a scale level that en-compasses the entire scene. We then use this multi-scale representation tovolume-render as in PointNeRF (Sec. 6.4.2)—but instead of averaging fea-tures locally, we do so across multiple scale levels. This allows us to naturallydeal with the sparsity of point clouds, without the need for failure-proneheuristics such as ‘pruning’ and ‘growing’ from PointNeRF [9]. To accountfor the large support region required at coarser scales, we propose to re-place the commonly used Multi-Layer Perceptron (MLP) with a tri-planerepresentation (Sec. 6.4.3). We note that using a single voxel at the coars-est scale (i.e., global) is equivalent to a ‘standard’ (i.e., not point-based)NeRF model. Therefore, in a sense, our solution unifies classical with pointcloud-based NeRF formulations (Sec. 6.4.1).As we illustrate in Figure 6.1, our approach results in novel-view synthesisthat is of much higher quality than previous methods. Compared to Point-NeRF, our approach is able to deal with regions with both high and lowpoint cloud density, even those without points (highlighted with red boxes).The recently popular 3D Gaussian Splatting [15] also suffers at these emptyregions as Gaussians are often initialized from point clouds. We evaluate ourmethod across three datasets, NeRF Synthetic [6], ScanNet [8], and KITTI-360 [7], outperforming the state of the art with a large margin (Sec. 6.5).We summarize our main contributions:896.3. Related work• we introduce an effective multi-scale representation for point-based NeRF;• we propose to incorporate a global voxel/scale, uniting “classical” andpoint-based NeRF formulations;• we propose to use a tri-plane representation for coarser scales to effec-tively cover larger support regions;• we outperform all baselines, and specifically show large improvementsover point-based NeRF, especially when the point clouds are sparse orincomplete.6.3 Related workThe introduction of Neural Radiance Fields [6] represented a paradigm shiftfor scene representation and realistic novel-view synthesis. NeRF employsa 5D implicit function to model a scene through a continuous volumetricapproach, which estimates both density and radiance for any given posi-tion and direction. Among many applications [233], NeRFs have been usedto reconstruct individual objects [6] and unbounded scenes [234], in un-controlled [235–237] or dynamic environments [238–242], in few-shot set-tings [219–222, 224] and large urban landscapes [226, 243, 244].Accelerated training. While NeRF yields remarkable results, this comesat the cost of long training time, owing to the need to evaluate large MLPmodels hundreds of times for each pixel. The prevailing approach to tacklingthis issue involves making a trade-off between compute and memory. Thisis achieved by storing features within various types of grids, including densegrids [245], sparse grids [246], multi-resolution hash grids [247], large sets ofsmall MLPs [248], low-rank tensor approximations of dense grids [249, 250],and hybrid planar and volumetric representations [251].Neural rendering with point clouds. While the techniques above cantrain efficiently, it is difficult to adapt them to model large environments. Analternative approach is to use point clouds to model the geometric structureof the scene [9, 15, 225, 252, 253]. Point clouds can have variable density,helping allocate computational resources where needed, and conveniently(not) represent empty space. To perform volume rendering, point cloudfeatures are queried in the local neighborhood of a ray to produce densityand color. These approaches can be classified on the basis of their neuralpoint representations, e.g. per-point features [9, 253], factorized volumetricrepresentations [254], tetrahedral meshes [255], and learnable Gaussians [15].906.3. Related workFigure 6.2: Overview – Given an input point cloud, we aggregate it overmulti-scale voxel grids (Sec. 6.4.1). For clarity, we draw the voxel grids in 2D.We then perform volume rendering based on points, relying on feature vec-tors stored thereon, which we aggregate across multiple scales (Sec. 6.4.2).Importantly, when aggregating across scales, we only take into account‘valid’ scales, i.e., those with nearby points—indicated with solid bluelines and illustrated as the two overlaid scales in the middle—naturallydealing with incomplete/sparse point clouds. The coarsest scale (the toprow in the figure) is a single, global voxel, equivalent to standard NeRF—i.e., it is not point-based.With PointNeRF, Xu et al. [9] and Chang et al. [253] use point cloud datato learn per-scene representations, by querying per-point features within alocal neighborhood. Kulhanek et al. [255] create tetrahedra using the pointsfrom COLMAP [10] and use barycentric interpolation to query the featureswithin a tetrahedron. Gaussian splatting [15] represents a 3D scene with 3Danisotropic Gaussians initialized by COLMAP, and optimizes their locationto faithfully represent the scene. Despite the high rendering quality, overall,Gaussian splatting is limited by the heuristics that they use to grow andprune points, similar to PointNeRF. For example, as shown in Fig. 6.1. Incontrast to these works, our approach builds a hierarchy of feature represen-tation, efficiently aggregating features in the local neighborhood at differentlevels; does not require optimizing the location of the points nor heuristics togrow and prune points; and leads to superior performance even with sparseor incomplete point clouds.Finally, rather than using geometric proximity, one can learn a point-to-query affinity function via transformers. Ost et al. [225] use transformers tocombine features of points along a ray to predict its color. A shortcomingof this approach is that it does not take into account occlusions and com-916.4. Methodbines all points in the neighborhood of a ray. Similarly, Chang et al. [252]use a set-transformer to find ray-surface intersections and use local featuresand blending weights to estimate ray colors. Both of these approaches aredifferent from ours, as we employ geometric, rather than learned, proximity.6.4 MethodAn overview of our method is shown in Fig. 5.2. We build a representationstarting from an input point cloud, which we then use to volume-render [6]a scene. Specifically, given an input point cloud Pin, we spatially aggregatethe point cloud to build a point cloud hierarchy with S levels. Denoting thisoperation as A(.), we write{Ps}Ss=1 = A(Pin), (6.1)and equip each point cloud level Ps with randomly initialized point featuresFs. We then optimize the features Fs by volume-rendering them along a ray(pixel) r by R(.), so that the estimated color matches that of the ground-truth pixels Cgt, using a photometric loss:argmin{Fs}Er[∥Cgt(r)−R(r|{Ps,Fs})∥22]. (6.2)We next detail our multi-scale aggregation strategy to define a hierarchicalrepresentation for point clouds (Sec. 6.4.1), and how we use it to volume-render a scene (Sec. 6.4.2). Finally, we propose to use a tri-plane-basedfeature representation in lieu of MLPs, in order to obtain a good trade-offbetween representation capacity and speed (Sec. 6.4.3).6.4.1 Multi-scale aggregation – AWe first detail our aggregation operation A in Eq. (6.1). To obtain a pointcloud that represents a desired scale level s, we cluster based on voxels.At level s, consider a regular grid of resolution Vs×Vs×Vs, consisting ofa set of voxels {Vvs}. We perform voxel-wise clustering to determine onerepresentative point per voxel aspvs = Ep∈Vvs [p] s.t. p ∈ Pin. (6.3)Importantly, note that this is performed only over non-empty voxels, hencethe resulting representation is sparse. Note also that the aggregation is built926.4. MethodFigure 6.3: Increasing coverage with multiple scales – We illustrateour sparse, hierarchical representation at three granularity levels, includinga single, global voxel (left). We also show three query points, with theirrespective neighborhoods (dotted circles) at each scale level—color-coded inblue if they have neighbouring features, and in orange if they do not. Ourmulti-scale approach naturally fills in empty regions, removing the need forfailure-prone region-growing heuristics [9]. Drawn in 2D, for clarity.at each scale level independently, and that while some fine-grained scalesmay not have valid aggregated points, more space regions will be coveredat the coarser scales. This allows for point clouds with variable density,or even incomplete ones to a certain degree, to be dealt with naturally.Finally, we set the coarsest voxel to cover the entire scene, effectively settingp00 = Ep∈Pin [p]. This coarsest scale can also be understood as a global NeRFmodel that is independent of the local distribution of the point cloud—providing a unified representation for both standard and point-based NeRF.6.4.2 Point-based rendering – RWe use volume rendering to render an image from the multi-scale pointcloud. Given a set of quadrature points along ray (pq, fq) ∈ r, let us denotethe volume rendering integral [6]Cˆr = R(pq ,fq)∈r(c(pq, fq), σ(pq, fq)), (6.4)936.4. Methodwhere c(pq, fq) is the radiance and σ(pq, fq) is the density of a location (pq, fq)in space. To obtain these values, we operate on our point cloud hierarchy, asopposed to the raw point cloud Pin as in PointNeRF [9]. More explicitly, weextend [9] to multiple scales by averaging over valid scale levels, i.e., scalelevels with any points within the vicinity of (pq, fq):c(pq, fq), σ(pq, fq) =M(Es∈S((pq ,fq),{Ps})[E ((pq, fq) | Ps,Fs)]), (6.5)where S((pq, fq), {Ps}) is the set of valid scale levels associated to query(pq, fq); E is the feature extraction operation in PointNeRF [9] that convertsthe point cloud into a feature embedding at the query location (pq, fq); andM is an MLP that converts those feature embeddings into radiance anddensity. We now describe S and E in more detail.Valid scale levels – S((pq, fq), {Ps}). Given a scale S, define N the localneighbors of (pq, fq) within distance τVs, where τ is the threshold ratio:N ((pq, fq),Ps) ={p | p ∈ Ps & ∥p− (pq, fq)∥2 ≤ τVs}. (6.6)which is then aggregated across levels to define:S((pq, fq), {Ps}) = {s | N ((pq, fq),Ps) ̸= ∅} , (6.7)Point cloud to feature embedding – E((pq, fq) | Ps,Fs). We aggre-gate the features within the support defined by Eq. (6.6) using normalizedinverse-distance weights w(p, (pq, fq)) = (∥p − (pq, fq)∥2 + ε)−1, where ε isa small number to avoid numerical problems:E((pq, fq)|Ps,Fs)=∑p∈N ((pq ,fq),Ps)w(p, (pq, fq)) F(fp,p− (pq, fq))∑p∈N ((pq ,fq),Ps)w(p, (pq, fq)),(6.8)where F is a learnable function, and fp is the feature in Fs correspondingto p ∈ Ps. Note that this is a simplified version of PointNeRF [9], as we donot use ‘per-point‘ weights [9, Sec. 4.1], which we experimentally found tonot contribute to improvements in rendering quality. Rather than relyingon large MLPs to implement F at coarse levels s, we employ a tri-planerepresentation, described in Sec. 6.4.3. This effectively increases the rep-resentation power of F at coarse levels so that less populated regions inspace can still be modeled effectively, without incurring an excessive com-putational burden.946.5. Results6.4.3 Per-point tri-plane featuresAs illustrated in Figure 6.3, points in coarser levels represent larger regions,and thus require preserving more information into each point feature. Wecould solve this by increasing either the feature dimension or the capacityof the MLPs used to parameterize F . They both come with a hefty price,greatly increasing the computational cost incurred to evaluate F . Instead,we build on recently-proposed factorized representations [250], and representlocal features with a local tri-plane factorization. In more detail, we storefeatures within three orthogonal feature planes fp ≡ {fXYp , fY Zp , fXZp }, whichare then accessed at (local) 3D coordinates u = ((pq, fq)− p)/(τVs):F(fp,u) = fXYp [u] + fY Zp [u] + fXZp [u], (6.9)where f∗∗p [u] denotes querying the plane at position u with bilinear inter-polation. We combine tri-planes at coarser levels with the standard MLPsat finer levels, where we find the latter are sufficient (see Sec. 6.5.2 for de-tails). At first glance, Eq. (6.9) may seem like a large deviation from usingan MLP, since the features seem to be independent of each other, due tothe lack of a shared MLP. Note, however, that those features are eventuallyprocessed by the shared decoder M that converts them into radiance fieldvalues. Finally, we note that at the coarsest scale level, i.e., the global voxel,our representation is effectively K-Planes [250].6.5 Results6.5.1 Implementation DetailsWe detail multi-scale point generation and coordinate system of the globaltriplane.Multi-scale Point Generation via Grid-subsamplingWe build multi-scale points from an input point cloud using grid subsam-pling, which is more robust to varying density as shown in KPConv [24].Specifically, a new support point at each scale is the barycenter of the orig-inal input points contained in a grid cell. Thereby, we control the scale anddensity at each level via the grid size of cells.We set the grid size at the level s as ω ∗ γs−1 where ω is the initial gridsize at the first level and γ is the stride size. We use the larger grid sizefor severely incomplete point clouds and a small grid size for the complete956.5. Results(a) Input pointcloud [8](b) Gauss.Splat. [15] &PointNeRF [9](c) PointNeRF++(Ours)(d) Ground truthFigure 6.4: Examples on KITTI-360 – We show novel-view renderingsobtained with our method, 3D Gaussian splatting [15] (pink colored, 1-4rows ) and PointNeRF [9] (green colored, 4-8 row) on a challenging out-doors dataset, using the same point clouds as input. Our approach providessharper renderings with more details, and better coverage in areas withoutpoints, where Gaussian Splatting and PointNeRF produce highly salient ar-tifacts highlighted with red boxes.point cloud. Specifically, for KITTI-360 [7], we set ω as 8cm and γ as 2.92.As a result, the grid size at the coarsest point level (i.e., s=4) is 2 meters.For ScanNet [8], we set ω as 0.008 and γ as 2.0. For Nerf Synthetic [6]where point clouds are relatively complete, we set ω as 0.004 and γ as 1.6.The Coordinate System of Global TriplaneWe align world coordinate system and the normalized coordinate systemof global triplane. We utilize the principal component analysis (PCA) tocalculate the reference coordinate frame of input point cloud. The resultantreference frame consists of rotation, translation and scale, thereby definingthe alignment matrix transforming world coordinates to coordinate systemof global triplane. In ScanNet [8] and Nerf Synthetic [6] where points966.5. ResultsFigure 6.5: Rendering across scales – We study the behavior of ourhierarchical approach by rendering an image adding one scale level at atime, from the global scale to the finest one. As expected, the coarse scalesare responsible for filling in empty regions (highlighted with red boxes) inthe point cloud, different from the well-covered regions (highlighted withgreen boxes) that can be modeled via fine scales.distribute uniformly along three axes and center at the origin, we simplynormalize world coordinates using the scale part of reference frame. ForKITTI 360 [7], we use full reference frame instead – i.e., we rotate, translateand scale the world coordinates – because, in this dataset, the car movesalong one major direction, leading to the points heavily unbalanced alongthree axes. With this PCA-based canonicalization, we compactly compressall possible query points into triplane’s normalized frame, allowing for fullyutilizing the capacity of global triplane.6.5.2 Experimental setupDatasets and metrics. We primarily use Peak Signal-to-Noise Ratio(PSNR) as a metric, and also structural (SSIM [256]) and perceptual (LPIPS[257]) similarity. We evaluate our method on three well-known datasets:976.5. Results• KITTI-360 [7] is a recent benchmark of outdoor driving sequences, highlychallenging due to the sparsity of views, which have much less visualoverlap than other datasets. Each sequence consists of an average of 80images. We use a random 10% subset for validation purposes, and alsofor our ablation study, as the ground truth for the test set is not publiclyavailable. To obtain results on the test set, we follow the standard prac-tice of training with the entire training set, to roughly the same number ofiterations required for convergence, discovered with the validation split.We use the point clouds provided with the dataset, from LiDAR scansthat are accumulated over all views. As this accumulated point cloud isvery dense, we resample it over a grid with a cell size of 8cm, and removepoints outside the camera frustum of the training views to make it moretractable.• ScanNet [8] is a dataset of indoor scans. We use the point clouds pro-vided with the dataset, which are sampled from mesh reconstructions us-ing RGB-D cameras with BundleFusion [258]. Following PointNeRF [9],we evaluate on two scenes, Scene-101 and Scene-241. The point cloudin Scene-101 has more incomplete regions, which makes it harder thanScene-241. As in PointNeRF [9], we sample 20% of the images, i.e. 1463images for Scene-241, and 1000 images for Scene-101, for training, anduse the rest for evaluation. We use the code provided by [9].• NeRF Synthetic [6] is a synthetic dataset with eight objects, each with100 training images and 200 test images. The images are purely synthetic,rendered with Blender. We use this dataset, as in PointNeRF [9], tovalidate our method when the scene is favorable to the standard NeRFsetting. We take the point clouds provided by PointNeRF [9], which areobtained with COLMAP [10].Implementation details. We implement our method with PyTorch [259].We use a total of 5 scale levels, including the global scale. We use a tri-plane resolution of 512× 512 for the global scale level. For the largest (i.e.,coarsest) two of the remaining scale levels we use tri-planes, where each tri-plane is built as a small two-layer pyramid with 4× 4 and 2× 2 grid. For alltri-planes, we store 32-dimensional feature vectors followed by a four-layerMLP with 64 neurons. For the remaining two (i.e., finest) scales, we simplyuse 32-dimensional point features and a four-layer MLP with 64 neurons.To further allow for the global scale to capture details that may be beyondthe capacity of its resolution, we augment the features extracted from theglobal tri-plane with positional encodings with 5 frequencies, as in [9]. We986.5. ResultsFigure 6.6: Examples on NeRF Synthetic – Our multi-scale approachconsistently fills in the holes in the input point cloud. PointNeRF relies on‘pruning’ and ‘growing’ heuristics, which can fail where the point cloud is notsufficiently dense, as shown here. While Gaussian Splatting also previousfailed to fill in holes with their densification heuristics (Fig. 6.4 and Fig. 6.7),for NeRF Synthetic these heuristics work well.found this to be especially important when modeling large scenes, such asfor KITTI-360. For M, we use one linear layer for density prediction and afour-layer MLP with 64 neurons for its hidden layers for color prediction.To speed up neighborhood search, we rely on voxel-grid-based approximatenearest neighbors, as in [9]. We use the same search radius as our neigh-borhood threshold τ in Eq. (6.6) after normalizing so that the approximatesearch is equivalent to a ball query. For speed-ups and to limit GPU memorygrowth, we set the maximum number of neighboring points to 8 for ScanNetand NeRF Synthetic, and 6 for KITTI-360, as the scenes are larger. Wefollow PointNeRF [9] to sample 400 points for each ray on ScanNet and NeRFSynthetic. As KITTI-360 is larger, we sample 1,000 points for each ray tocompensate. We use the contraction function of [234] for regions outside thebounding box of the point cloud. For KITTI-360, as in [226], we model thesky using a four-layer MLP that maps ray direction to color. To improvesample efficiency, we also use a proposal network [234].We train our model with a single NVidia V100 GPU for 200k iterations.996.5. ResultsUsespoints Avg. Chair Drums Ficus Hotdog Lego Materials Mic ShipNeRF [6] ✗ 31.01 33.00 25.01 30.13 36.18 32.54 29.62 32.91 28.65Plenoxels [245] ✗ 31.76 33.98 25.35 31.83 36.81 34.1 29.14 33.26 29.62InstantNGP [247] ✗ 33.18 35.00 26.02 33.51 37.40 36.39 29.78 36.22 31.10MipNeRF [260] ✗ 33.09 35.14 25.48 33.29 37.48 35.7 30.71 36.51 30.41Gauss. Splat. [15] ✗ 33.32 35.83 26.15 34.87 37.72 35.78 30.00 35.36 30.80FreqPCR [261] ✓ 31.24 33.06 25.95 32.19 35.82 31.56 29.69 33.64 27.97TetraNeRF [255] ✓ 32.53 35.05 25.01 33.31 36.16 34.75 29.30 35.49 31.13Gauss. Splat. [15] ✓ 33.60 36.10 26.17 34.87 37.77 36.14 30.15 36.48 31.12PointNeRF [9] ✓ 31.77 35.09 25.01 33.24 35.49 32.65 26.97 35.54 30.18Ours ✓ 33.70 36.32 26.11 34.43 37.45 36.75 30.32 36.85 31.34Table 6.1: PSNR↑ on NeRF Synthetic [6] – Our method performs bestoverall, even on object-centric data with dense point clouds.For the learning rate we follow [9] and use the same exponential decayingscheduling with an initial learning rate of 5e-4 for M and a larger initiallearning rate of 2e-3 for F. Following [9], we decay every 1000k steps witha rate of 0.1. We will make our code available with an Apache 2.0 license,for reproducibility.6.5.3 KITTI-360 results – Fig. 6.4 and Sec. 6.5.3We first compare our method to the state of the art on KITTI-360 [7], achallenging outdoors dataset with incomplete point clouds from real LiDARmeasurements.Baselines. We report numbers on the hidden test set, which requires up-loading samples to the evaluation server to compare with methods featuredon the public leaderboard. We also report numbers on our validation splitfor methods that do not have an entry in the public leaderboard. We con-sider methods based on images and, optionally, semantics [6, 260, 262–265]as well as those that use the LiDAR point clouds [7, 9, 15, 266]. Note thatfor all PointNeRF experiments in this work we use the point ‘pruning’ and‘growing’ heuristics introduced in their work [9, Sec. 4.2], which aim at im-proving geometry modeling and image rendering quality, and can help dealwith point cloud sparsity—our algorithm does not rely on it.Discussions. We show qualitative highlights in Fig. 6.4 and report resultson Sec. 6.5.3. Our method achieves a new state of the art in the color-1006.5. ResultsValidation TestUsespoints PSNR↑ SSIM↑ LPIPS↓ PSNR↑ SSIM↑ LPIPS↓w/Sem. Nerflets [262] ✗ – – – 21.69 – –PNF [263] ✗ – – – 22.07 0.820 0.221PLANeRF [265] ✗ – – – 22.64 0.855 0.200ColoronlyFVS [264] ✗ – – – 20.00 0.790 0.193NeRF [6] ✗ – – – 21.18 0.779 0.343Mip-NeRF [260] ✗ – – – 21.54 0.778 0.365PBNR [266] ✓ – – – 19.91 0.811 0.191PCL [7] ✓ – – – 12.81 0.576 0.549Gauss. Splat. [15] ✓ 18.59 0.642 0.257 22.08 0.844 0.139PointNeRF [9] ✓ 17.63 0.629 0.337 19.44 0.796 0.266Ours ✓ 20.05 0.665 0.305 22.44 0.828 0.212Table 6.2: Results on KITTI-360 [7] – Our method achieves the bestperformance among methods that supervise only with color. It performs onpar with those that also rely on semantics. We provide results on the (public)validation set and the (hidden) test set as some baselines have results forone, but not the other.only category among NeRF methods, and performs on par with methodsthat also use semantic supervision and Gaussian Splatting. Importantly, weimprove over other point-based methods with a large margin. Comparedwith PointNeRF, our approach yields better renderings on regions wherethe point cloud is sparse, and the global scale allows us to tackle thosewith no nearby points, such as structures too far away to be captured byLiDAR. Also of note is how Gaussian splatting, while it provides improvedrendering quality in terms of SSIM and LPIPS thanks to its SSIM-basedloss, can have failure modes, as highlighted in Fig. 6.4. We thus believecombining our strategy of utilizing our multi-scale strategy with Gaussiansplatting can further lead to performance improvements, but we leave asfuture work.6.5.4 ScanNet results – Fig. 6.7 and Tab. 6.3Next, we consider indoor scans, using ScanNet [8]. This dataset is lesschallenging than KITTI-360, but is a typical use-case for point cloud-basedneural rendering, and is where the benefit of using point clouds was stronglydemonstrated in PointNeRF [9].Baselines. We compare our method against NeRF [6], PointNeRF [9], and1016.5. ResultsAvg. Scene-101 Scene-241Usespoints PSNR↑ SSIM ↑ LPIPS↓ PSNR↑ PSNR↑NeRF [6] ✗ 24.43 0.670 0.494 27.16 21.69Gauss. Splat. [15] ✗ 29.56 0.812 0.301 29.01 30.11Gauss. Splat. [15] ✓ 29.93 0.818 0.275 29.55 30.31PointNeRF [9] ✓ 25.92 0.784 0.263 21.98 29.86Ours ✓ 30.56 0.808 0.238 30.27 30.85Table 6.3: Results on two ScanNet scenes [8] as pre-processed by[9] – Our method outperforms all others, especially PointNeRF, the methodmost similar to ours, by a large margin demonstrating the effectiveness ofour multi-scale approach.PSNR↑ SSIM↑ LPIPS↓MLP 19.59 0.643 0.353Triplane 20.05 0.665 0.305Table 6.4: Impact of the tri-plane – We evaluate tri-planes vs. MLPs.Using tri-planes for the coarsest scales improves performance, at a compa-rable computational cost.Gaussian Splatting [15]. For the latter, we consider randomly initializedpoint clouds as well as those provided by the dataset.Discussions. As shown in Tab. 6.3 and Fig. 6.7, our method performs best.NeRF [6], for this dataset does not perform well as the scene is relativelytextureless and smooth. PointNeRF [9] improves over it by leveraging thepoint clouds. It does, however, have issues on Scene-101, because its pointcloud has large incomplete areas, which impair its performance, as shownin Fig. 6.7. Our method is able to cope with these empty regions, thanksto our multi-scale framework. Interestingly, Gaussian Splatting also worksbetter than typical NeRF while trained purely with images, even when start-ing from random points—point cloud initialization can further improve itsperformance. This suggests the importance of including the notion of local-ity introduced by points to the representation. Our method outperforms allbaselines, point-based or not. We use point clouds from mesh inputs insteadof depth images, also reported by PointNeRF, as the latter are extremely1026.5. Results(a) Input pointcloud [8](b) Gauss.Splat. [15](c) PointNeRF [9] (d) Ours (e) Ground truthFigure 6.7: Examples on ScanNet – PointNeRF fails to reconstruct thescene on regions where the point cloud is empty. Both our method and Gaus-sian Splatting are able to fill them in, but our approach produces cleanerresults, with fewer artifacts. This is especially noticeable for Scene-101 (topand bottom rows), where the mesh has large holes where PointNeRF fails torender meaningful pixels, even with their ‘growing’ heuristic that is aimedtowards filling such gaps.dense (see [9, Tbl. 8]).6.5.5 NeRF Synthetic results – Fig. 6.6 and Tab. 6.1Finally, we verify the effectiveness of our method on NeRF Synthetic, todemonstrate that it remains helpful even use-cases designed for NeRF.Baselines. We compare our approach against both methods that onlyutilize RGB images [6], which this dataset is typically used to evaluate,and those that use point clouds [9, 255, 261], including the recent GaussianSplatting [15].Discussions. We report PSNR results in Tab. 6.1 and provide qualitativeexamples in Fig. 6.6. Our method still performs best overall, slightly aheadof Gaussian Splatting. More importantly, it outperforms the point-basedbaselines by a larger margin.1036.5. ResultsPSNR↑ SSIM↑ LPIPS↓w/o global 19.78 0.675 0.290global only 17.95 0.520 0.442full 20.05 0.665 0.305Table 6.5: Using a global feature – We measure PSNR↑ on two datasetsfor different variants of our approach. Adding a global voxel at the coarsestscale to the hierarchical structure (right), improves performance (left), butis not sufficient by itself (middle).Num. of scales PSNR↑ SSIM↑ LPIPS↓0 17.95 0.520 0.4421 19.88 0.674 0.2832 19.89 0.669 0.2893 19.90 0.666 0.2994 20.05 0.665 0.305Table 6.6: Number of scales vs rendering quality – As expected, morelevels lead to better PSNR↑. We always use a global scale—‘0’ correspondsto using only a global scale, ‘1’ the finest scale plus the global scale, andlater adding coarser scale levels, up to our full model (‘4’).6.5.6 Ablation studyWe thoroughly ablate our method in this section on KITTI-360 using thevalidation split. We consider the number of scale levels, using a tri-plane vsan MLP for F at the coarsest scale levels, and study the effect of adding aglobal scale. We also evaluate performance at different levels of point cloudsparsity.Number of scale levels – Tab. 6.6 and Fig. 6.5. We ablate how thenumber of scale levels affects performance, by training and evaluating modelsusing a different number of scales. We also illustrate what each scale level isadding, by rendering views with a multi-scale model adding one scale levelat a time, in Fig. 6.5. As clearly shown in the figure, the global scale isinstrumental in rendering accurate pixels in those areas, and each successive1046.5. ResultsRatio of pts. PSNR↑ SSIM↑ LPIPS↓0 17.95 0.520 0.4421 18.71 0.558 0.43410 19.35 0.621 0.370full 20.05 0.665 0.305Table 6.7: Number of points – We show that our method remains ap-plicable to sparser point clouds, with noticeable improvement over points-agnostic model even at drastic downsampling rates (1%).scale adds finer details, improving the overall quality of the rendering.Tri-plane vs MLP – Tab. 6.4. We also provide an ablation study toevaluate the advantages of using a tri-plane instead of a regular MLP forthe parameterized function F . As outlined in Sec. 6.4.3, we always use MLPsat the two finest scale levels. Using a tri-plane performs slightly better at asimilar computational cost.Using a global voxel – Tab. 6.5. We evaluate the importance of adding aglobal voxel at the coarsest scale. We compare three variants of our method:one using four local scales (“w/o global”); one using only the global scale(“global-only”), i.e., a traditional point-agnostic NeRF; and one using both(“full”). The point-based formulation outperforms point-agnostic NeRF,but combining them with our multi-scale-plus-global approach does best.Number of points – Tab. 6.7. We measure performance while randomlydownsampling the point cloud with increasing ratios. Our approach per-forms while even at downsampling rates below 1% and using as few as 10kpoints.6.5.7 More rendering resultsWe furthermore provide more rendering results – rendering more frames andforming videos. For more details, please refer to https://pointnerfpp.github.io.1056.6. Conclusions6.6 ConclusionsNeural Radiance Fields are a paradigm shift in novel-view synthesis. Despitetheir promise, and a large number of follow-up papers, challenges persist,particularly when few views of the scene are available. Point clouds providea very attractive data stream, complementary to images, and often readilyavailable in both indoor and outdoor settings—but have a different set ofchallenges, due to incompleteness and sparsity. We mitigate this with asimple yet novel multi-scale representation that combines global and localinformation, yielding performance improvements with a large margin acrossthe board. Our work unifies point cloud-based and standard NeRF pipelinesand adapts effectively to variable point densities and empty regions, pushingnovel view synthesis on uncontrolled, real-world data closer to practice.Limitations and future work. While our method provides improvementswith a large margin over PointNeRF by combining point-based ones withclassic NeRF, the computational cost of our method is naturally boundby classic NeRF—on NeRF Synthetic, our method induces a 20% in com-pute overhead to the classic NeRF backbone that we use. Thus, an inter-esting research direction would be to combine the strengths of our multi-scale strategy in handling incomplete and sparse point clouds with the highcomputational-efficiency of 3D Gaussian Splatting [15], especially given thepitfalls of 3D Gaussian Splatting demonstrated in our work.106Chapter 7Conclusion and OutlookIn this thesis, we present four explicit models applied to different tasks re-lated to point clouds, each addressing limitations existing in prior works andlargely improving task performance. These explicit models constrain someintermediate neural features to carry a certain meaning like poses and at-tention, allowing for traditional ways of solving target tasks in the realm ofpoint cloud. Specifically, (1), ACNe improves the noisy point cloud learningby following the optimization method – we iteratively refine points’ confi-dence to eliminate the impact of outliers in point cloud on feature learning.(2), Canonical capsule eliminates the dataset bias prevalent in prior 3D deeplearning by decomposing objects into the equivariant pose and invariant de-scriptor. (3), NeuralBF formulates 3D instance proposal generation as atraditional bilateral filtering algorithm. (4), PointNeRF++ combines thepoint-based representation and vanilla NeRF – a hybrid of point-based andvanilla NeRF for neural rendering.While addressing different tasks, these chapters share a common theme –i.e., we improve point cloud learning by making neural networks explicit.More importantly, these explicit models all achieve improvement with alarge margin over prior baselines in target tasks. While scaling up networkswithout enforcing any meaning to intermediate features – e.g., large foun-dation models – has achieved great success, our philosophy of being explicitis still valuable to the research community. Despite the success of largefoundation models, they require lots of resources – such as computing, data,and parameters to discover the explicit meanings that we enforce into ourmodel without scaling up networks. This is particularly essential in pointcloud learning, where scaling up networks is unsolved yet due to the uniquechallenges of point clouds. We believe that this philosophy of being explicitcan be generalized to other domains where scalability is limited – e.g., thesmall size of training data or small compute budget in edge devices.Notably, explicit models are highly interpretable. In the era of large foun-1077.1. Practical tips for the design of explicit representationdation models, the interpretability of the model is increasingly importantas it assures AI safety [267, 268]. Our exploration of explicit models forpoint cloud learning can inspire future research on the interpretability ofthe model.7.1 Practical tips for the design of explicitrepresentationWhile explicit models have advantages, it’s nontrivial to design explicit mod-els as different tasks may require different explicit representations such asthe four models we explored. Thus, we summarize practical tips for thedesign of explicit representation given a task.Factorization of the task: The target of a task typically depends onmultiple key factors. Given a task, we factorize the task target into thesefactors and choose to model some of the key factors that are essential tothe task’s objective. For example, in the autoencoding of unaligned pointclouds (Canonical Capsules [36]), we consider factors such as the pose andcontent of the object. In scene representation (PointNeRF++ [21]), wedecompose a global MLP-based NeRF into a set of sparse local point-basedNeRF. It requires a deep understanding of the task to perform an effectivefactorization. Even though, we believe it’s valuable to the studied task,as evidenced in all four chapters of this thesis—-an explicit representationconsistently leads to performance improvements by a large margin.Being explicit by design: Once we decide what to model explicitly, webegin with its math definition and construct the explicit representation fromfirst principles. For example, in NeuralBF, we closely follow the math defi-nition of a convex hull to compose a distance field from the learned hyper-planes. This assures that we make the full use of all hyperplanes, leading toa compact and efficient neural field.Being explicit by regularization: We can also encourage the learnedfeatures to adhere to their mathematical definitions through regularization.For example, in Canonical Capsules, we enforce the transformation invari-ance/equivariance properties of capsules using a set of training losses. Inthis case, the explicit representation doesn’t strictly follow its mathematicalprinciple but tends to align more closely with it as training progresses. Theprimary benefit of this approach is that it allows us to avoid computation-ally intensive layers, such as transformation-equivariant layers, which are1087.2. Future worksFigure 7.1: A surprising failure case in ChatGPT [16]: giventhe prompt ( “ We know small explicit models are awesome insome aspects. But it’s really the time to embrace large scalefoundation model"), ChatGPT (version of GPT-4o in Jun. 2024) – i.e.,one of world’s largest language model, while generating the images of highimage quality, has the grammar errors in the image (multiple words aremisspelled). And we observe this grammar error consistently with differentrandom seeds.typically required to generate strictly equivariant features.7.2 Future worksWhile we demonstrate the advantages of explicit models in point cloud learn-ing, it is undeniable that large foundation models have achieved great successin other domains. Therefore, it is necessary for us to embrace large founda-tion models in the future for point cloud learning. As shown in Fig. 7.1 ,we demonstrate an impressive result from the large foundation models (e.g.,ChatGPT[16]). It generates highly informative images that effectively illus-trate the differences between explicit models and large models. In light ofthis, we will investigate how to scale up point cloud learning.But it does not mean that we should not work on explicit models as thelarge foundation model is not always perfect [269]. We argue that we shouldcombine explicit models and large-scale networks. This is not only becauseexplicit models aim to provide highly interpretable and safe models, butalso because they can address the limitations of large models. For instance,1097.2. Future worksas shown in Fig. 7.1, ChatGPT, while impressive, still makes grammar er-rors despite it’s capabilities in language modeling. We believe that explicitmodels can address such limitations through our proposed strategies. Forexample, following the tips that we suggested above, we can factorize thetask of generating images with captions into two subtasks: image contentgeneration and caption generation.In conclusion, we argue that combining explicit models with large founda-tion models can harness the strengths of both approaches, leading to moreinterpretable, safe, and effective solutions in point cloud learning and be-yond.110Bibliography[1] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet:Deep Learning on Point Sets for 3D Classification and Segmentation.In Conference on Computer Vision and Pattern Recognition, 2017.[2] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, MathieuSalzmann, and Pascal Fua. Learning to Find Good Correspondences.In Conference on Computer Vision and Pattern Recognition, 2018.[3] Daniel Detone, Tomasz Malisiewicz, and Andrew Rabinovich. Su-perpoint: Self-Supervised Interest Point Detection and Description.CVPR Workshop on Deep Learning for Visual SLAM, 2018.[4] Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir Kim,Bryan Russell, and Mathieu Aubry. Learning Elementary Structuresfor 3D Shape Generation and Matching. In Advances in Neural Infor-mation Processing Systems, 2019.[5] Riccardo Spezialetti, Federico Stella, Marlon Marcon, Luciano Silva,Samuele Salti, and Luigi Di Stefano. Learning to Orient Surfacesby Self-supervised Spherical CNNs. Advances in Neural InformationProcessing Systems, 2020.[6] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T.Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenesas Neural Radiance Fields for View Synthesis. In ECCV, 2020.[7] Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A Novel Datasetand Benchmarks for Urban Scene Understanding in 2d and 3d. PAMI,2022.[8] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, ThomasFunkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3dReconstructions of Indoor Scenes. In CVPR, pages 5828–5839, 2017.111Bibliography[9] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, KalyanSunkavalli, and Ulrich Neumann. Point-NeRF: Point-based Neural Ra-diance Fields. In Conference on Computer Vision and Pattern Recog-nition, 2022.[10] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion Revisited. In CVPR, 2016.[11] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recogni-tion. In Proceedings of the IEEE international conference on computervision, pages 945–953, 2015.[12] Nima Sedaghat, Mohammadreza Zolfaghari, Ehsan Amiri, andThomas Brox. Orientation-boosted voxel nets for 3D object recog-nition. 2017.[13] Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari.3D Point Capsule Networks. In Conference on Computer Vision andPattern Recognition, 2019.[14] Ken Shoemake. Uniform random rotations. In Graphics Gems III(IBM Version), pages 124–132. Elsevier, 1992.[15] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and GeorgeDrettakis. 3D Gaussian Splatting for Real-Time Radiance Field Ren-dering. TOG, 2023.[16] OpenAI. Chatgpt, 2024. Personal communication, June 5, 2024.[17] Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, andKwang Moo Yi. Acne: Attentive context normalization for robustpermutation-equivariant learning. In Proceedings of the IEEE/CVFconference on computer vision and pattern recognition, pages 11286–11295, 2020.[18] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, MathieuSalzmann, and Pascal Fua. Learning to find good correspondences.In Conference on Computer Vision and Pattern Recognition. IEEE,2018. (Oral).[19] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang,and Oscar Beijbom. Pointpillars: Fast encoders for object detection112Bibliographyfrom point clouds. In Proceedings of the IEEE/CVF conference oncomputer vision and pattern recognition, pages 12697–12705, 2019.[20] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Pro-ceedings of the IEEE Conference on Computer Vision and PatternRecognition, pages 3075–3084, 2019.[21] Weiwei Sun, Eduard Trulls, Yang-Che Tseng, Sneha Sambandam,Gopal Sharma, Andrea Tagliasacchi, and Kwang Moo Yi. Point-nerf++: A multi-scale, point-based neural radiance field. arXivpreprint arXiv:2312.02362, 2023.[22] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++:Deep Hierarchical Feature Learning on Point Sets in a Metric Space.Advances in Neural Information Processing Systems, 2017.[23] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet:Deep Learning on Point Sets for 3D Classification and Segmentation.In Conference on Computer Vision and Pattern Recognition, 2017.[24] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, BeatrizMarcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flex-ible and Deformable Convolution for Point Clouds. In ICCV, 2019.[25] Hugues Thomas, Yao-Hung Hubert Tsai, Timothy D Barfoot, and JianZhang. Kpconvx: Modernizing kernel point convolution with kernelattention. arXiv Preprint, 2024.[26] Andrea Vedaldi and Karel Lenc. Matconvnet: Convolutional neuralnetworks for matlab. In Proceedings of the 23rd ACM internationalconference on Multimedia, pages 689–692, 2015.[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenetclassification with deep convolutional neural networks. Advances inneural information processing systems, 25, 2012.[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. DeepResidual Learning for Image Recognition. In Conference on ComputerVision and Pattern Recognition, 2016.[29] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu,Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point113Bibliographytransformer v3: Simpler, faster, stronger. In Conference on ComputerVision and Pattern Recognition, 2024.[30] Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mo-hammed Bennamoun. Deep learning for 3d point clouds: A sur-vey. IEEE transactions on pattern analysis and machine intelligence,43(12):4338–4364, 2020.[31] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3dSemantic Segmentation with Submanifold Sparse Convolutional Net-works. In Conference on Computer Vision and Pattern Recognition,2018.[32] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, ThanhNguyen, and Sai-Kit Yeung. Revisiting point cloud classification: Anew benchmark dataset and classification model on real-world data. InProceedings of the IEEE/CVF international conference on computervision, pages 1588–1597, 2019.[33] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis,Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scaleindoor spaces. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 1534–1543, 2016.[34] Peng-Shuai Wang. Octformer: Octree-based transformers for 3d pointclouds. ACM Transactions on Graphics (TOG), 42(4):1–11, 2023.[35] Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray andcolor images. In International Conference on Computer Vision, 1998.[36] Weiwei Sun, Andrea Tagliasacchi, Boyang Deng, Sara Sabour, SorooshYazdani, Geoffrey E Hinton, and Kwang Moo Yi. Canonical capsules:Self-supervised capsules in canonical pose. Advances in Neural infor-mation processing systems, 34:24993–25005, 2021.[37] Weiwei Sun, Daniel Rebain, Renjie Liao, Vladimir Tankovich, SorooshYazdani, Kwang Moo Yi, and Andrea Tagliasacchi. Neuralbf: Neuralbilateral filtering for top-down instance segmentation on point clouds.In Proceedings of the IEEE/CVF Winter Conference on Applicationsof Computer Vision, pages 551–560, 2023.[38] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao.Gvcnn: Group-view convolutional neural networks for 3d shape recog-114Bibliographynition. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 264–272, 2018.[39] Ze Yang and Liwei Wang. Learning relationships for multi-view 3Dobject recognition. In Proceedings of the IEEE/CVF internationalconference on computer vision, pages 7505–7514, 2019.[40] Xin Wei, Ruixuan Yu, and Jian Sun. View-gcn: View-based graphconvolutional network for 3d shape analysis. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition,pages 1850–1859, 2020.[41] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutionalneural network for real-time object recognition. In 2015 IEEE/RSJ in-ternational conference on intelligent robots and systems (IROS), pages922–928. IEEE, 2015.[42] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and XinTong. O-CNN: Octree-Based Convolutional Neural Networks for 3dShape Analysis. ACM Transactions on Graphics, 36(4):1–11, 2017.[43] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet:Learning deep 3d representations at high resolutions. In Proceedingsof the IEEE conference on computer vision and pattern recognition,pages 3577–3586, 2017.[44] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet:Deep Learning on Point Sets for 3D Classification and Segmentation.CVPR, 2016.[45] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Point-net++: Deep hierarchical feature learning on point sets in a metricspace. Advances in Neural Information Processing Systems, 2017.[46] Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing net-work for point cloud analysis. In Proceedings of the IEEE conferenceon computer vision and pattern recognition, pages 9397–9406, 2018.[47] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolu-tional networks on 3d point clouds. In Proceedings of the IEEE/CVFConference on computer vision and pattern recognition, pages 9621–9630, 2019.115Bibliography[48] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan.Relation-shape convolutional neural network for point cloud analy-sis. In Proceedings of the IEEE/CVF conference on computer visionand pattern recognition, pages 8895–8904, 2019.[49] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, ZhihuaWang, Niki Trigoni, and Andrew Markham. Randla-net: Efficientsemantic segmentation of large-scale point clouds. In Proceedings ofthe IEEE/CVF conference on computer vision and pattern recognition,pages 11108–11117, 2020.[50] Alexandre Boulch. Convpoint: Continuous convolutions for pointcloud processing. Computers & Graphics, 88:24–34, 2020.[51] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, TianweiShen, Yurong Chen, Long Quan, and Hongen Liao. Learning Two-View Correspondences and GeometryUsing Order-Aware Network. InInternational Conference on Computer Vision, 2019.[52] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and VladlenKoltun. Point transformer. In Proceedings of the IEEE/CVF interna-tional conference on computer vision, pages 16259–16268, 2021.[53] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu,Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer.Computational Visual Media, 7:187–199, 2021.[54] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Pa-conv: Position adaptive convolution with dynamic kernel assemblingon point clouds. In Proceedings of the IEEE/CVF conference on com-puter vision and pattern recognition, pages 3173–3182, 2021.[55] Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, An-drew Markham, and Niki Trigoni. Learning Object Bounding Boxesfor 3D Instance Segmentation on Point Clouds. Advances in NeuralInformation Processing Systems, 2019.[56] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poc-zos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep Sets. InAdvances in Neural Information Processing Systems, 2017.[57] François Chollet. Xception: Deep learning with depthwise separa-ble convolutions. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 1251–1258, 2017.116Bibliography[58] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang,Stephen Lin, and Baining Guo. Swin transformer: Hierarchical visiontransformer using shifted windows. In Proceedings of the IEEE/CVFinternational conference on computer vision, pages 10012–10022, 2021.[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LlionJones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Atten-tion Is All You Need. In Advances in Neural Information ProcessingSystems, 2017.[60] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, AndreaTagliasacchi, and Leonidas J Guibas. Vector neurons: A general frame-work for so (3)-equivariant networks. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision, pages 12200–12209,2021.[61] Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, andYaron Lipman. Frame averaging for equivariant shape space learning.In Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 631–641, 2022.[62] Jiahui Huang, Zan Gojcic, Matan Atzmon, Or Litany, Sanja Fidler,and Francis Williams. Neural kernel surface reconstruction. In Pro-ceedings of the IEEE/CVF Conference on Computer Vision and Pat-tern Recognition, pages 4369–4379, 2023.[63] Shaobo Xia, Jun Yue, Kacper Kania, Leyuan Fang, Andrea Tagliasac-chi, Kwang Moo Yi, and Weiwei Sun. Densify your labels: Unsuper-vised clustering with bipartite matching for weakly supervised pointcloud segmentation. arXiv preprint arXiv:2312.06799, 2023.[64] Cheng-Kun Yang, Ji-Jia Wu, Kai-Syun Chen, Yung-Yu Chuang, andYen-Yu Lin. An MIL-Derived Transformer for Weakly SupervisedPoint Cloud Segmentation. In CVPR, 2022.[65] Cheng-Kun Yang, Min-Hung Chen, Yung-Yu Chuang, and Yen-YuLin. 2D-3D Interlaced Transformer for Point Cloud Segmentation withScene-Level Supervision. In ICCV, 2023.[66] Rick Chartrand and Wotao Yin. Iteratively Reweighted Algorithmsfor Compressive Sensing. In International Conference on Acoustics,Speech, and Signal Processing, 2008.117Bibliography[67] Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz, andRichard Szeliski. Building Rome in One Day. In International Con-ference on Computer Vision, 2009.[68] Noah Snavely, Rahul Garg, Steven M Seitz, and Richard Szeliski.Finding Paths Through the World’s Photos. ACM Transactions onGraphics, 2008.[69] Johannes Kopf, Michael F Cohen, and Richard Szeliski. First-PersonHyper-Lapse Videos. ACM Transactions on Graphics, 33(4):78, 2014.[70] David G. Lowe. Distinctive Image Features from Scale-Invariant Key-points. International Journal of Computer Vision, 20(2):91–110, 2004.[71] Martin A. Fischler and Robert C. Bolles. Random Sample Consensus:A Paradigm for Model Fitting with Applications to Image Analysisand Automated Cartography. Communications ACM, 24(6):381–395,1981.[72] Amir R. Zamir, Tilman Wekel, Pulkit Argrawal, Colin Weil, JitendraMalik, and Silvio Savarese. Generic 3D Representation via Pose Esti-mation and Matching. In European Conference on Computer Vision,2016.[73] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer,Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. DeMoN: Depth andMotion Network for Learning Monocular Stereo. In Conference onComputer Vision and Pattern Recognition, 2017.[74] Tinghui Zhou, Matthew Brown, Noah Snavely, and David Lowe. Un-supervised Learning of Depth and Ego-Motion from Video. In Con-ference on Computer Vision and Pattern Recognition, 2017.[75] Zheng Dang, Kwang Moo Yi, Yinlin Hu, Fei Wang, Pascal Fua, andMathieu Salzmann. Eigendecomposition-Free Training of Deep Net-works with Zero Eigenvalue-Based Losses. In European Conference onComputer Vision, 2018.[76] René Ranftl and Vladlen Koltun. Deep Fundamental Matrix Estima-tion. In European Conference on Computer Vision, 2018.[77] Chen Zhao, Zhiguo Cao, Chi Li, Xin Li, and Jiaqi Yang. NM-Net:Mining Reliable Neighbors for Robust Feature Correspondences. Con-ference on Computer Vision and Pattern Recognition, 2019.118Bibliography[78] Florian Kluger, Eric Brachmann, Hanno Ackermann, Carsten Rother,Michael Ying Yang, and Bodo Rosenhahn. CONSAC: Robust Multi-Model Fitting by Conditional Sample Consensus. CVPR, 2020.[79] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++:Deep Hierarchical Feature Learning on Point Sets in a Metric Space.In Advances in Neural Information Processing Systems, 2017.[80] Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerat-ing Deep Network Training by Reducing Internal Covariate Shift. InInternational Conference on Machine Learning, 2015.[81] Jimmy L. Ba, Jamie R. Kiros, and Geoffrey E. Hinton. Layer Normal-ization. arXiv Preprint, 2016.[82] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. InstanceNormalization: The Missing Ingredient for Fast Stylization. arXivPreprint, 2016.[83] Yuxin Wu and Kaiming He. Group Normalization. In European Con-ference on Computer Vision, 2018.[84] Sofien Bouaziz, Andrea Tagliasacchi, Hao Li, and Mark Pauly. ModernTechniques and Applications for Real-Time Non-rigid Registration. InSIGGRAPH Asia (Technical Course Notes), 2016.[85] Andrew Cotter, Maya Gupta, Heinrich Jiang, Erez Louidor, JamesMuller, Tamann Narayan, Serena Wang, and Tao Zhu. Shape con-straints for set functions. In International Conference on MachineLearning, pages 1388–1396, 2019.[86] Muhammad J Mirza and Kim L Boyer. Performance Evaluationof a Class of M-estimators for Surface Parameter Estimation inNoisy Range Data. IEEE Transactions on Robotics and Automation,9(1):75–85, 1993.[87] John Fox. An R and S-Plus Companion to Applied Regression. Sage,2002.[88] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Hong-gang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual AttentionNetwork for Image Classification. In Conference on Computer Visionand Pattern Recognition, 2017.119Bibliography[89] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, JosefSivic, Akihiko Torii, and Torsten Sattler. D2-Net: A Trainable CNNfor Joint Detection and Description of Local Features. Conference onComputer Vision and Pattern Recognition, 2019.[90] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Con-volutional Neural Networks on Graphs with Fast Localized SpectralFiltering. In Advances in Neural Information Processing Systems,2016.[91] Thomas N. Kipf and MaxWelling. Semi-Supervised Classification withGraph Convolutional Networks. International Conference on LearningRepresentations, 2017.[92] Yin Zhou and Oncel Tuzel. VoxelNet: End-to-End Learning for PointCloud Based 3D Object Detection. In Conference on Computer Visionand Pattern Recognition, 2018.[93] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Yi Zhou.Tangent Convolutions for Dense Prediction in 3D. In Conference onComputer Vision and Pattern Recognition, 2018.[94] Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. Mining PointCloud Local Structures by Kernel Correlation and Graph Pooling. InConference on Computer Vision and Pattern Recognition, pages 4548–4557, 2018.[95] Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker.Point2Sequence: Learning the Shape Representation of 3D PointClouds with an Attention-Based Sequence to Sequence Network.American Association for Artificial Intelligence Conference, 2019.[96] Philip H.S. Torr and Andrew Zisserman. MLESAC: A New RobustEstimator with Application to Estimating Image Geometry. ComputerVision and Image Understanding, 78:138–156, 2000.[97] JiaWang Bian, Wen-Yan Lin, Yasuyuki Matsushita, and Sai-Kit Ye-ungand Tan-Dat Nguyena nd Ming-Ming Cheng. GMS: Grid-BasedMotion Statistics for Fast, Ultra-Robust Feature Correspondence. InConference on Computer Vision and Pattern Recognition, 2017.[98] Tim Salimans and Diederik P. Kingma. Weight Normalization: ASimple Reparameterization to Accelerate Training of Deep Neural Net-works. In Advances in Neural Information Processing Systems, 2016.120Bibliography[99] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and YuichiYoshida. Spectral Normalization for Generative Adversarial Networks.In International Conference on Learning Representations, 2018.[100] Xun Huang and Serge Belongie. Arbitrary Style Transfer in Real-timewith Adaptive Instance Normalization. In International Conferenceon Computer Vision, 2017.[101] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effec-tive Approaches to Attention-based Neural Machine Translation. Con-ference on Empirical Methods in Natural Language Processing, 2015.[102] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville,Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, At-tend and Tell: Neural Image Caption Generation with Visual Atten-tion. International Conference on Machine Learning, 2015.[103] Kevin Jarrett, Koray Kavukcuoglu, Marc A. Ranzato, and Yann Le-Cun. What is the Best Multi-Stage Architecture for Object Recogni-tion? In International Conference on Computer Vision, 2009.[104] Alex Krizhevsky, Ilya Sutskever, and Geoffery E. Hinton. ImageNetClassification with Deep Convolutional Neural Networks. In Advancesin Neural Information Processing Systems, 2012.[105] H K. Hartline, Henry G. Wagner, and Floyd Ratliff. Inhibition in theEye of Limulus. Journal of General Psychology, 1956.[106] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and KorayKavukcuoglu. Spatial Transformer Networks. In Advances in Neu-ral Information Processing Systems, 2015.[107] Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu. AttentionalShapeContextNet for Point Cloud Recognition. In Conference onComputer Vision and Pattern Recognition, 2018.[108] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. DeepResidual Learning for Image Recognition. In Conference on ComputerVision and Pattern Recognition, 2016.[109] Sofien Bouaziz, Andrea Tagliasacchi, and Mark Pauly. Sparse IterativeClosest Point. In Computer Graphics Forum, 2013.121Bibliography[110] Ondřej Chum, Jiří Matas, and Josef Kittler. Locally OptimizedRANSAC. In Joint Pattern Recognition Symposium, 2003.[111] Ondrej Chum, Tomas Werner, and Jiri Matas. Two-view GeometryEstimation Unaffected by a Dominant Plane. In Conference on Com-puter Vision and Pattern Recognition, 2005.[112] Daniel Barath, Jana Noskova, and Jiří Matas. MAGSAC: Marginaliz-ing Sample Consensus. In Conference on Computer Vision and PatternRecognition, 2019.[113] Richard Hartley and Andrew Zisserman. Multiple View Geometry inComputer Vision. Cambridge University Press, 2000.[114] Diederik P. Kingma and Jimmy Ba. Adam: A Method for StochasticOptimisation. In International Conference on Learning Representa-tions, 2015.[115] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang,Xiaoou Tang, and Jianxiong Xiao. 3D ShapeNets: A Deep Represen-tation for Volumetric Shapes. In Conference on Computer Vision andPattern Recognition, 2015.[116] Changchang Wu. Towards Linear-Time Incremental Structure fromMotion. In 3DV, 2013.[117] Rahul Raguram, Ondrej Chum, Marc Pollefeys, Jiri Matas, and Jan-Michael Frahm. USAC: a Universal Framework for Random SampleConsensus. IEEE Transactions on Pattern Analysis and Machine In-telligence, 35(8):2022–2038, 2012.[118] Daniel Barath and Jiří Matas. Graph-cut RANSAC. In Conferenceon Computer Vision and Pattern Recognition, 2018.[119] Peter J. Rousseeuw. Least Median of Squares Regression. Journal ofthe American Statistical Association, 1984.[120] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-Net:Learning Local Features from Images. In Advances in Neural Infor-mation Processing Systems, 2018.[121] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanra-han, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, ShuranSong, Hao Su, et al. Shapenet: An Information-Rich 3D Model Repos-itory. arXiv Preprint, 2015.122Bibliography[122] David G Lowe. Distinctive Image Features from Scale-Invariant Key-points. International Journal of Computer Vision, 60:91–110, 2004.[123] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A Dis-criminatively Trained, Multiscale, Deformable Part Model. In Con-ference on Computer Vision and Pattern Recognition, 2008.[124] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. YouOnly Look Once: Unified, Real-Time Object Detection. In Conferenceon Computer Vision and Pattern Recognition, 2016.[125] Karen Simonyan and Andrew Zisserman. Very Deep ConvolutionalNetworks for Large-Scale Image Recognition. International Conferenceon Learning Representations, 2015.[126] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hin-ton. A Simple Framework for Contrastive Learning of Visual Repre-sentations. International Conference on Machine Learning, 2020.[127] Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas J. Guibas,and Or Litany. PointContrast: Unsupervised Pre-training for 3D PointCloud Understanding. In European Conference on Computer Vision,2020.[128] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. FoldingNet:Point Cloud Auto-Encoder via Deep Grid Deformation. In Conferenceon Computer Vision and Pattern Recognition, 2018.[129] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Rus-sell, and Mathieu Aubry. A Papier-Mâché Approach to Learning 3DSurface Generation. In Conference on Computer Vision and PatternRecognition, 2018.[130] Lars Mescheder, Michael Oechsle, Michael Niemeyer, SebastianNowozin, and Andreas Geiger. Occupancy Networks: Learning 3DReconstruction in Function Space. In Conference on Computer Vi-sion and Pattern Recognition, 2019.[131] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and ThomasFunkhouser. Deep Structured Implicit Functions. In Conference onComputer Vision and Pattern Recognition, 2020.123Bibliography[132] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Ge-offrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. NASA:Neural Articulated Shape Approximation. In European Conference onComputer Vision, 2020.[133] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transform-ing Auto-Encoders. In International Conference on Artificial NeuralNetworks, 2011.[134] Olga Sorkine-Hornung and Michael Rabinovich. Least-Squares RigidMotion Using SVD. Computing, 2017.[135] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li,Kai Kohlhoff, and Patrick Riley. Tensor Field Networks: Rotation-andTranslation-Equivariant Neural Networks for 3D Point Clouds. arXivPreprint, 2018.[136] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, AndreaTagliasacchi, and Leonidas Guibas. Vector neurons: a general frame-work for so(3)-equivariant networks, 2021.[137] Yongheng Zhao, Tolga Birdal, Jan Eric Lenssen, Emanuele Menegatti,Leonidas Guibas, and Federico Tombari. Quaternion Equivariant Cap-sule Networks for 3D Point Clouds. In European Conference on Com-puter Vision, 2020.[138] Nitish Srivastava, Hanlin Goh, and Ruslan Salakhutdinov. GeometricCapsule Autoencoders for 3D Point Clouds. arXiv Preprint, 2019.[139] Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton.Stacked Capsule Autoencoders. In Advances in Neural InformationProcessing Systems, 2019.[140] Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a Multi-View Stereo Machine. In Advances in Neural Information ProcessingSystems, 2017.[141] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A Point Set Genera-tion Network for 3D Object Reconstruction from a Single Image. InConference on Computer Vision and Pattern Recognition, 2017.[142] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee.Perspective Transformer Nets: Learning Single-View 3D Object Re-124Bibliographyconstruction without 3D Supervision. In Advances in Neural Informa-tion Processing Systems, 2016.[143] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, andYu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from singlergb images. In European Conference on Computer Vision, 2018.[144] Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Maka-dia. Deformable Shape Completion with Graph Convolutional Autoen-coders. In Conference on Computer Vision and Pattern Recognition,2018.[145] Peng-Shuai Wang, Chun-Yu Sun, Yang Liu, and Xin Tong. Adap-tive o-cnn: A patch-based deep representation of 3d shapes. ACMTransactions on Graphics (TOG), 37(6):1–11, 2018.[146] Zhantao Deng, Jan Bednařík, Mathieu Salzmann, and Pascal Fua.Better Patch Stitching for Parametric Surface Reconstruction. arXivPreprint, 2020.[147] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe,and Steven Lovegrove. DeepSDF: Learning Continuous Signed Dis-tance Functions for Shape Representation. In Conference on ComputerVision and Pattern Recognition, 2019.[148] Zhiqin Chen and Hao Zhang. Learning Implicit Fields for Genera-tive Shape Modeling. In Conference on Computer Vision and PatternRecognition, 2019.[149] Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, andJitendra Malik. Learning shape abstractions by assembling volumetricprimitives. In Conference on Computer Vision and Pattern Recogni-tion, 2017.[150] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William TFreeman, and Thomas Funkhouser. Learning Shape Templates withStructured Implicit Functions. In International Conference on Com-puter Vision, 2019.[151] Deng, Boyang and Genova, Kyle and Yazdani, Soroosh and Bouaziz,Sofien and Hinton, Geoffrey and Tagliasacchi, Andrea. CvxNet:Learnable Convex Decomposition. In Conference on Computer Vi-sion and Pattern Recognition, 2020.125Bibliography[152] Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Ju-lian Straub, Steven Lovegrove, and Richard Newcombe. Deep LocalShapes: Learning Local SDF Priors for Detailed 3D Reconstruction.In European Conference on Computer Vision, 2020.[153] Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojcic, Srinath Srid-har, and Leonidas J. Guibas. CaSPR: Learning Canonical Spatiotem-poral Point Cloud Representations. In Advances in Neural InformationProcessing Systems, 2020.[154] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, ShuranSong, and Leonidas J Guibas. Normalized Object Coordinate Spacefor Category-level 6d Object Pose and Size Estimation. In Conferenceon Computer Vision and Pattern Recognition, 2019.[155] Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng,Zihao Wang, Yuwen Xiong, Hao Su, and Raquel Urtasun. Weakly-Supervised 3D Shape Completion in the Wild. In European Conferenceon Computer Vision, 2020.[156] David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova,and Andrea Vedaldi. C3dpo: Canonical 3d pose networks for non-rigid structure from motion. In Conference on Computer Vision andPattern Recognition, 2019.[157] Yue Wang and Justin M Solomon. Deep Closest Point: Learning Rep-resentations for Point Cloud Registration. In International Conferenceon Computer Vision, 2019.[158] Wentao Yuan, Ben Eckart, Kihwan Kim, Varun Jampani, Dieter Fox,and Jan Kautz. DeepGMR: Learning Latent Gaussian Mixture Modelsfor Registration. In European Conference on Computer Vision, 2020.[159] Clara Fernandez-Labrador, Ajad Chhatkuli, Danda Pani Paudel,Jose J Guerrero, Cédric Demonceaux, and Luc Van Gool. Unsuper-vised Learning of Category-Specific Symmetric 3D Keypoints fromPoint Sets. In European Conference on Computer Vision, 2020.[160] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and Mo-hammad Norouzi. Discovery of Latent 3D Keypoints via End-to-EndGeometric Reasoning. In Advances in Neural Information ProcessingSystems, 2018.126Bibliography[161] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural TangentKernel: Convergence and Generalization in Neural Networks. In Ad-vances in Neural Information Processing Systems, 2018.[162] Yuxin Wu and Kaiming He. Group Normalization. In European Con-ference on Computer Vision, 2018.[163] Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Meng Wang, andLiang Lin. Clusternet: Deep Hierarchical Cluster Network with Rig-orously Rotation-Invariant Representation for Point Cloud Analysis.In Conference on Computer Vision and Pattern Recognition, 2019.[164] Zan Gojcic, Caifa Zhou, Jan D Wegner, Leonidas J Guibas, and TolgaBirdal. Learning multiview 3D point cloud registration. In Conferenceon Computer Vision and Pattern Recognition, 2020.[165] Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matchingusing learned features. In Conference on Computer Vision and PatternRecognition, 2020.[166] Richard Hartley, Jochen Trumpf, Yuchao Dai, and Hongdong Li. Ro-tation averaging. International Journal of Computer Vision, 2013.[167] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registra-tion. In European Conference on Computer Vision, 2016.[168] Christopher M Bishop. Pattern Recognition and Machine Learning.springer, 2006.[169] Geoffrey F Hinton. A Parallel Computation that Assigns CanonicalObject-based Frames of Reference. In International Joint Conferenceon Artificial Intelligence, 1981.[170] Geoffrey E Hinton and Kevin J Lang. Shape Recognition and IllusoryConjunctions. In International Joint Conference on Artificial Intelli-gence, 1985.[171] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bron-stein, and Justin M Solomon. Dynamic graph cnn for learning on pointclouds. ACM Transactions on Graphics, 2019.[172] Oliver van Kaick, Kai Xu, Hao Zhang, Yanzhen Wang, Shuyang Sun,Ariel Shamir, and Daniel Cohen-Or. Co-hierarchical analysis of shapestructures. ACM SIGGRAPH, 2013.127Bibliography[173] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Gener-ating compact meshes via binary space partitioning. In Conference onComputer Vision and Pattern Recognition, 2020.[174] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva,and Thomas Funkhouser. Semantic scene completion from a singledepth image. Proceedings of 30th IEEE Conference on Computer Vi-sion and Pattern Recognition, 2017.[175] Sara Sabour, Andrea Tagliasacchi, Soroosh Yazdani, Geoffrey E. Hin-ton, and David J. Fleet. Unsupervised part representation by flowcapsules, 2021.[176] Yong He, Hongshan Yu, Xiaoyan Liu, Zhengeng Yang, Wei Sun, Yao-nan Wang, Qiang Fu, Yanmei Zou, and Ajmal Mian. Deep Learningbased 3D Segmentation: A survey. arXiv Preprint, 2021.[177] Christopher Xie, Yu Xiang, Arsalan Mousavian, and Dieter Fox. Un-seen Object Instance Segmentation for Robotic Environments. IEEETransactions on Robotics, 2021.[178] Takaya Ogawa and Tomohiro Mashita. Occlusion Handling in OutdoorAugmented Reality using a Combination of Map Data and InstanceSegmentation. In 2021 IEEE International Symposium on Mixed andAugmented Reality Adjunct (ISMAR-Adjunct), 2021.[179] Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kpam:Keypoint affordances for category-level robotic manipulation. InISRR, 2019.[180] Douglas Morrison, Adam W Tow, Matt Mctaggart, R Smith, NortonKelly-Boxall, Sean Wade-Mccue, Jordan Erskine, Riccardo Grinover,Alec Gurman, T Hunn, et al. Cartman: The low-cost cartesian ma-nipulator that won the amazon robotics challenge. In 2018 IEEEInternational Conference on Robotics and Automation (ICRA), pages7757–7764. IEEE, 2018.[181] Kshitij Sirohi, Rohit Mohan, Daniel Büscher, Wolfram Burgard, andAbhinav Valada. Efficientlps: Efficient lidar panoptic segmentation.IEEE Transactions on Robotics, 2021.[182] Dingfu Zhou, Jin Fang, Xibin Song, Liu Liu, Junbo Yin, Yuchao Dai,Hongdong Li, and Ruigang Yang. Joint 3d instance segmentation and128Bibliographyobject detection for autonomous driving. In Conference on ComputerVision and Pattern Recognition, 2020.[183] Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic In-stance Segmentation for Autonomous Driving. In Conference on Com-puter Vision and Pattern Recognition Workshops, 2017.[184] Kelvin Wong, Shenlong Wang, Mengye Ren, Ming Liang, and RaquelUrtasun. Identifying Unknown Instances for Autonomous Driving. InConference on Robot Learning, 2020.[185] Andres Milioto, Jens Behley, Chris McCool, and Cyrill Stachniss.Lidar panoptic segmentation for autonomous driving. In 2020IEEE/RSJ International Conference on Intelligent Robots and Sys-tems (IROS), pages 8505–8512. IEEE, 2020.[186] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. MaskR-CNN. In International Conference on Computer Vision, 2017.[187] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, DingLiang, Chunhua Shen, and Ping Luo. Polarmask: Single Shot InstanceSegmentation with Polar Representation. In Conference on ComputerVision and Pattern Recognition, 2020.[188] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and LeiLi. Solo: A Simple Framework for Instance Segmentation. IEEETransactions on Pattern Analysis and Machine Intelligence, 2021.[189] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen.Solov2: Dynamic and Fast Instance Segmentation. Advances in NeuralInformation Processing Systems, 2020.[190] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu,and Jiaya Jia. Pointgroup: Dual-set Point Grouping for 3d InstanceSegmentation. In Conference on Computer Vision and Pattern Recog-nition, 2020.[191] Shu Kong and Charless C Fowlkes. Recurrent Pixel Embedding forInstance Grouping. In Conference on Computer Vision and PatternRecognition, 2018.[192] Xinlong Wang, Shu Liu, Xiaoyong Shen, Chunhua Shen, and Jiaya Jia.Associatively Segmenting Instances and Semantics in Point Clouds. InConference on Computer Vision and Pattern Recognition, 2019.129Bibliography[193] Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang DYoo. SoftGroup for 3D Instance Segmentation on Point Clouds. InConference on Computer Vision and Pattern Recognition, 2022.[194] Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and XinggangWang. Hierarchical Aggregation for 3d Instance Segmentation. InInternational Conference on Computer Vision, 2021.[195] Shih-Hung Liu, Shang-Yi Yu, Shao-Chi Wu, Hwann-Tzong Chen, andTyng-Luh Liu. Learning Gaussian Instance Segmentation in PointClouds. arXiv Preprint, 2020.[196] Ji Hou, Angela Dai, and Matthias Nießner. 3d-sis: 3d Semantic In-stance Segmentation of Rgb-d Scans. In Conference on ComputerVision and Pattern Recognition, 2019.[197] Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas J Guibas.Gspn: Generative Shape Proposal Network for 3d Instance Segmenta-tion in Point Cloud. In Conference on Computer Vision and PatternRecognition, 2019.[198] Shitong Luo, Jiahan Li, Jiaqi Guan, Yufeng Su, Chaoran Cheng,Jian Peng, and Jianzhu Ma. Equivariant point cloud analysis vialearning orientations for message passing. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition,pages 18932–18941, 2022.[199] Omri Puny, Matan Atzmon, Heli Ben-Hamu, Edward J Smith, IshanMisra, Aditya Grover, and Yaron Lipman. Frame averaging for invari-ant and equivariant network design. arXiv preprint arXiv:2110.03336,2021.[200] Jake Levinson, Carlos Esteves, Kefan Chen, Noah Snavely, AngjooKanazawa, Afshin Rostamizadeh, and Ameesh Makadia. An analysisof svd for deep rotation estimation. NeurIPS, 33:22554–22565, 2020.[201] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan,Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann,and Srinath Sridhar. Neural fields in visual computing and beyond.Computer Graphics Forum, 2022.[202] Wenchao Gu, Shuang Bai, and Lingxing Kong. A Review on 2D in-stance Segmentation Based on Deep Neural Networks. IVC, 2022.130Bibliography[203] Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic In-stance Segmentation with a Discriminative Loss Function. Conferenceon Computer Vision and Pattern Recognition Workshops, 2017.[204] Varun Jampani, Martin Kiefel, and Peter V. Gehler. Learning SparseHigh Dimensional Filters: Image Filtering, Dense CRFs and BilateralNeural Networks. In Conference on Computer Vision and PatternRecognition, 2016.[205] Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasi-noff, and Frédo Durand. Deep bilateral learning for real-time imageenhancement. ACM Transactions on Graphics, 2017.[206] Mehdi Khoshboresh Masouleh and Reza Shah-Hosseini. Fusion of DeepLearning with Adaptive Bilateral Filter for Building Outline Extrac-tion from Remote Sensing Imagery. Journal of Applied Remote Sens-ing, 2018.[207] Shachar Fleishman, Iddo Drori, and Daniel Cohen-Or. Bilateral MeshDenoising. ACM Transactions on Graphics, 2003.[208] Kaiming He, Jian Sun, and Xiaoou Tang. Guided Image Filtering.IEEE Transactions on Pattern Analysis and Machine Intelligence,2012.[209] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Ge-offrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. NASA:Neural Articulated Shape Approximation. In European Conference onComputer Vision, 2020.[210] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich,and Dacheng Tao. Deep ordinal regression network for monoculardepth estimation. In Conference on Computer Vision and PatternRecognition, 2018.[211] Zachary Teed and Jia Deng. Raft: Recurrent All-pairs Field Trans-forms for Optical Flow. In European Conference on Computer Vision,2020.[212] Diederik P Kingma and Jimmy Ba. Adam: A method for stochasticoptimization. International Conference on Learning Representations,2014.131Bibliography[213] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic Gradient Descentwith Warm Restarts. International Conference on Learning Represen-tations, 2017.[214] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn:Towards Real-time Object Detection with Region Proposal Networks.Advances in Neural Information Processing Systems, 2015.[215] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. DeepHough Voting for 3d Object Detection in Point Clouds. In Conferenceon Computer Vision and Pattern Recognition, 2019.[216] Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, and Kui Jia.Instance Segmentation in 3d Scenes using Semantic Superpoint TreeNetworks. In International Conference on Computer Vision, 2021.[217] Tong He, Chunhua Shen, and Anton van den Hengel. Dyco3d: Ro-bust instance segmentation of 3d point clouds through dynamic con-volution. In Conference on Computer Vision and Pattern Recognition,2021.[218] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully Con-volutional One-stage Object Detection. In Conference on ComputerVision and Pattern Recognition, 2019.[219] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixel-NeRF: Neural Radiance Fields from One or Few Images. In CVPR,2021.[220] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting NeRF on aDiet: Semantically Consistent Few-Shot View Synthesis. In ICCV,2021.[221] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xi-ang, Jingyi Yu, and Hao Su. MVSNeRF: Fast Generalizable RadianceField Reconstruction from Multi-view Stereo. In ICCV, 2021.[222] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M.Sajjadi, Andreas Geiger, and Noha Radwan. RegNeRF: RegularizingNeural Radiance Fields for View Synthesis from Sparse Inputs. InCVPR, 2022.132Bibliography[223] C. Deng, C. Jiang, C. R. Qi, X. Yan, Y. Zhou, L. Guibas, andD. Anguelov. NeRDi: single-view nerf synthesis with language-guideddiffusion as general image priors. In CVPR, 2023.[224] Jamie Wynn and Daniyar Turmukhambetov. Diffusionerf: Regular-izing Neural Radiance Fields with Denoising Diffusion Models. InCVPR, 2023.[225] Julian Ost, Issam Laradji, Alejandro Newell, Yuval Bahat, and FelixHeide. Neural Point Light Fields. In CVPR, 2022.[226] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan TBarron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Fer-rari. Urban Radiance Fields. In CVPR, 2022.[227] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.Vision meets Robotics: The KITTI Dataset. IJRR, 2013.[228] Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou, QichuanGeng, and Ruigang Yang. The ApolloScape Open Dataset for Au-tonomous Driving and Its Application. TPAMI, 2019.[229] Tobias Gruber, Frank Julca-Aguilar, Mario Bijelic, and Felix Heide.Gated2Depth: Real-Time Dense Lidar From Gated Images. In ICCV,2019.[230] Jakob Geyer, Yohannes Kassahun, Mentar Mahmudi, Xavier Ricou,Rupesh Durgesh, Andrew S Chung, Lorenz Hauswald, Viet HoangPham, Maximilian Mühlegg, Sebastian Dorn, et al. A2d2: Audi Au-tonomous Driving Dataset. ARXIV, 2020.[231] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard,Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Ben-jamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao,Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, AdityaJoshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and DragomirAnguelov. Scalability in Perception for Autonomous Driving: WaymoOpen Dataset. In CVPR, 2020.[232] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora,Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, GiancarloBaldan, and Oscar Beijbom. NuScenes: A Multimodal Dataset forAutonomous Driving. In CVPR, 2020.133Bibliography[233] Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, and JonathanLi. NeRF: Neural Radiance Field in 3d Vision, a Comprehensive Re-view. ARXIV, 2022.[234] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srini-vasan, and Peter Hedman. Mip-NeRF 360: unbounded anti-aliasedneural radiance fields. CVPR, 2022.[235] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi,Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRFin the wild: Neural Radiance Fields for Unconstrained Photo Collec-tions. In CVPR, 2021.[236] Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, XuanWang, and Jue Wang. Hallucinated Neural Radiance Fields in theWild. In CVPR, 2022.[237] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, andZhangyang Wang. NeuralLift-360: Lifting an In-the-Wild 2D Phototo a 3D Object With 360deg Views. In CVPR, 2023.[238] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and FrancescMoreno-Noguer. D-NeRF: Neural Radiance Fields for DynamicScenes. In CVPR, 2020.[239] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz,Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies:Deformable Neural Radiance Fields. In CVPR, 2021.[240] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, QingShuai, Xiaowei Zhou, and Hujun Bao. Animatable Neural RadianceFields for Modeling Dynamic Human Bodies. In ICCV, 2021.[241] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Bar-ron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, andSteven M. Seitz. HyperNeRF: A Higher-Dimensional Representationfor Topologically Varying Neural Radiance Fields. TOG, 2021.[242] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and AnuragRanjan. Neuman: Neural Human Radiance Field from a Single Video.In ECCV, 2022.134Bibliography[243] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, BenMildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kret-zschmar. Block-NeRF: Scalable large scene neural view synthesis. InCVPR, 2022.[244] Xiuchao Wu, Jiamin Xu, Xin Zhang, Hujun Bao, Qixing Huang, YujunShen, James Tompkin, and Weiwei Xu. ScaNeRF: Scalable Bundle-Adjusting Neural Radiance Fields for Large-Scale Scene Rendering. InTOG, 2023.[245] Fridovich-Keil and Yu, Matthew Tancik, Qinhong Chen, BenjaminRecht, and Angjoo Kanazawa. Plenoxels: Radiance Fields withoutNeural Networks. In CVPR, 2022.[246] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. De-bevec. Baking Neural Radiance Fields for Real-Time View Synthesis.In ICCV, 2021.[247] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.Instant Neural Graphics Primitives with a Multiresolution Hash En-coding. TOG, 2020.[248] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilo-NeRF: Speeding up Neural Radiance Fields with Thousands of TinyMLPs. In ICCV, 2021.[249] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.TensoRF: Tensorial Radiance Fields. In ECCV, 2022.[250] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg,Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit Radi-ance Fields in Space, Time, and Appearance. In CVPR, 2023.[251] Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P Srinivasan,Ben Mildenhall, Andreas Geiger, Jonathan T Barron, and Peter Hed-man. MERF: Memory-Efficient Radiance Fields for Real-time ViewSynthesis in Unbounded Scenes. SIGGRAPH 2023, 2023.[252] Jen-Hao Rick Chang, Wei-Yu Chen, Anurag Ranjan, Kwang Moo Yi,and Oncel Tuzel. Pointersect: Neural Rendering with Cloud-Ray In-tersection. In CVPR, 2023.[253] MingFang Chang, Akash Sharma, Michael Kaess, and Simon Lucey.Neural Radiance Field with LiDAR maps. In ICCV, 2023.135Bibliography[254] Tao Hu, Xiaogang Xu, Ruihang Chu, and Jiaya Jia. TriVol: PointCloud Rendering via Triple Volumes. In CVPR, 2023.[255] Jonas Kulhanek and Torsten Sattler. Tetra-NeRF: Representing Neu-ral Radiance Fields Using Tetrahedra. ARXIV, 2023.[256] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.Image Quality Assessment: from Error Visibility to Structural Simi-larity. TIP, 2004.[257] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, andOliver Wang. The Unreasonable Effectiveness of Deep Features asa Perceptual Metric. In CVPR, 2018.[258] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, andChristian Theobalt. Bundlefusion: Real-time Globally Consistent 3dReconstruction Using on-the-fly Surface Reintegration. TOG, 2017.[259] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Ed-ward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, LucaAntiga, and Adam Lerer. Automatic Differentiation in PyTorch. InNIPS-W, 2017.[260] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman,Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A Mul-tiscale Representation for Anti-aliasing Neural Radiance Fields. InICCV, 2021.[261] Yi Zhang, Xiaoyang Huang, Bingbing Ni, Teng Li, and Wenjun Zhang.Frequency-Modulated Point Cloud Rendering with Easy Editing. InCVPR, 2023.[262] Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, LeonidasGuibas, Hao Su, and Kyle Genova. Nerflets: Local Radiance Fieldsfor Efficient Structure-aware 3d Scene Representation from 2d super-vision. In CVPR, 2023.[263] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alirezabooktitle Fathi, Car-oline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Del-laert, and Thomas Funkhouser. Panoptic Neural Fields: A SemanticObject-aware Neural Scene Representation. In CVPR, 2022.[264] Gernot Riegler and Vladlen Koltun. Free view synthesis. In ECCV,2020.136Bibliography[265] Fusang Wang, Arnaud Louys, Nathan Piasco, Moussab Bennehar,Luis Roldão, and Dzmitry Tsishkou. PlaNeRF: SVD Unsupervised3D Plane Regularization for NeRF Large-Scale Scene Reconstruction.3DV, 2023.[266] Georgios Kopanas, Julien Philip, Thomas Leimkühler, and GeorgeDrettakis. Point-Based Neural Rendering with Per-View Optimiza-tion. In CGF, 2021.[267] Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics.MIT Press, 2023.[268] Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, PieterAbbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue,Shai Shalev-Shwartz, et al. Managing extreme ai risks amid rapidprogress. Science, 384(6698):842–845, 2024.[269] Ayush Sarkar, Hanlin Mai, Amitabh Mahapatra, Svetlana Lazebnik,David A Forsyth, and Anand Bhattad. Shadows don’t lie and linescan’t bend! generative models don’t know projective geometry... fornow. In Proceedings of the IEEE/CVF Conference on Computer Vi-sion and Pattern Recognition, pages 28140–28149, 2024.137
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Yi, Kwang Moo
Tagliasacchi, Andrea
Exploring explicit models for geometric point cloud learning
Text