Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Yuan, Kaiwen
Doctor of Philosophy - PhD
University of British Columbia
Modern robots are generally armed with diverse modalities of sensors, for various functionalities and safety redundancy. The recent breakthrough of deep learning (DL) technologies demonstrates superior performance on many high-level tasks, especially with using multi-sensor fusion. While the majority of multi-modal DL methods assume that sensors are well calibrated, synchronized and denoised, such efforts at the sensor layer are non-trivial and get increasingly expensive with the growing complexity of robotic systems. Currently dominant approaches heavily rely on specific hardware setup or high-end sensors, which generally are not cost-effective. This cost concern could be a bottleneck for the potential wide adoption of low-cost robots in the near future. Even though DL has a huge potential at the sensor layer, the difficulty of acquiring sufficient and accurate annotations for related tasks remain a major challenge. This thesis first formulates key problems at the robot sensor layer from the machine learning perspective, and further proposes efficient self-supervised learning approaches systematically. In our work, the popular and representative LiDAR-camera-inertial system is utilized as the study target. Firstly, the challenging LiDAR-camera online extrinsic calibration task is delved into, and we investigate the Riemannian metrics equipped self-supervised learning approach via synthetic data. This was the first work in the literature which demonstrates competing performance of data-driven methods when compared with conventional approaches at the sensor layer. It lays the foundation and shows the potential for later deeper explorations. Secondly, we address several overlooked limitations of the conventional synchronization pipelines and propose the first DL based LiDAR-camera synchronization framework, which is an innovative self-supervised learning schema. Thirdly, the problem of Inertial Measurement Unit (IMU) denoising for navigation is studied, and we propose a self-supervised multi-task framework. This work demonstrates the superiority of data-driven approaches on IMU denoising and presents one realistic self-supervised learning implementation. These explorations initialize the adoption of deep learning for robot sensor layer tasks and show case how self-supervised learning can be applied. Our work helps push the boundary of self-supervised learning at the sensor layer to an usable stage, demonstrate the potential for this direction and shed the lights for future research.
Self-supervised Learning at the Sensor Layer in RoboticsbyKaiwen YuanB.Eng., Xi’an Jiaotong University, 2013M.A.Sc. The University of British Columbia, 2015A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES(Electrical and Computer Engineering)The University of British Columbia(Vancouver)February 2023© Kaiwen Yuan, 2023The following individuals certify that they have read, and recommend to the Facultyof Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Self-supervised Learning at the Sensor Layer in Roboticssubmitted by Kaiwen Yuan in partial fulfillment of the requirements for the degreeof Doctor of Philosophy in Electrical and Computer Engineering.Examining Committee:Z. Jane Wang, Professor, Electrical and Computer Engineering, UBCSupervisorSeptimiu (Tim) E. Salcudean, Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberHelge Rhodin , Assistant Professor, Computer Science, UBCSupervisory Committee MemberAntony J. Hodgson, Professor, Mechanical Engineering, UBCUniversity ExaminerPurang Abolmaesumi, Professor, Electrical and Computer Engineering, UBCUniversity ExamineriiAbstractModern robots are generally armed with diverse modalities of sensors, for variousfunctionalities and safety redundancy. The recent breakthrough of deep learning(DL) technologies demonstrates superior performance on many high-level tasks,especially with using multi-sensor fusion. While the majority of multi-modal DLmethods assume that sensors are well calibrated, synchronized and denoised, suchefforts at the sensor layer are non-trivial and get increasingly expensive with thegrowing complexity of robotic systems. Currently dominant approaches heavilyrely on specific hardware setup or high-end sensors, which generally are not cost-effective. This cost concern could be a bottleneck for the potential wide adoption oflow-cost robots in the near future. Even though DL has a huge potential at the sensorlayer, the difficulty of acquiring sufficient and accurate annotations for related tasksremain a major challenge.This thesis first formulates key problems at the robot sensor layer from themachine learning perspective, and further proposes efficient self-supervised learningapproaches systematically. In our work, the popular and representative LiDAR-camera-inertial system is utilized as the study target. Firstly, the challenging LiDAR-camera online extrinsic calibration task is delved into, and we investigate the Rie-mannian metrics equipped self-supervised learning approach via synthetic data.This was the first work in the literature which demonstrates competing performanceof data-driven methods when compared with conventional approaches at the sensorlayer. It lays the foundation and shows the potential for later deeper explorations.Secondly, we address several overlooked limitations of the conventional synchro-nization pipelines and propose the first DL based LiDAR-camera synchronizationframework, which is an innovative self-supervised learning schema. Thirdly, theiiiproblem of Inertial Measurement Unit (IMU) denoising for navigation is studied,and we propose a self-supervised multi-task framework. This work demonstrates thesuperiority of data-driven approaches on IMU denoising and presents one realisticself-supervised learning implementation.These explorations initialize the adoption of deep learning for robot sensor layertasks and show case how self-supervised learning can be applied. Our work helpspush the boundary of self-supervised learning at the sensor layer to an usable stage,demonstrate the potential for this direction and shed the lights for future research.ivLay SummaryThe objective of this thesis is to address limitations of dominant conventionalmethods and systematically explore self-supervised learning (SSL) approaches atthe robot sensor layer. There are three core concerns at the robot sensor layer:1) spatial relation; 2) temporal relation; and 3) noise relation. For each relation,a representative task has been studied. More specifically, we study: 1) LiDAR-camera spatial calibration; 2) LiDAR-camera temporal synchronization; and 3) IMUdenoising. Significant contributions have been made to the related area, buildingthe foundation for future researches on SSL at the robot sensor layer.vPrefaceThis dissertation is written based on a collection of manuscripts, focusing on theself-supervised learning at the sensor layer in robotics. The majority of the researchworks, including literature reviews, problem formulation, methods implementationand manuscript writing, were conducted by the candidate, under the supervision ofProf. Z. Jane Wang. The manuscripts were primarily written by the candidate, withtechnical suggestions and editorial feedback mainly from Prof. Z. Jane Wang.Chapter 3 is based on the following manuscript:• K. Yuan, Z. Guo and Z. J. Wang, "RGGNet: Tolerance Aware LiDAR-CameraOnline Calibration With Geometric Deep Learning and Generative Model,"IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 6956-6963, Oct.2020, doi: 10.1109/LRA.2020.3026958. [177]The author proposed the idea and was responsible for the algorithm design,method implementation and manuscript writing. These works were conducted withthe guidance, extensive technical suggestions and editorial feedback from Prof. Z.Jane Wang. Dr. Zhenyu Guo provided many important feedback on the algorithmdesign and manuscript writing.Chapter 4 is based on the following manuscript:• K. Yuan, L. Ding, M. Abdelfattah and Z. J. Wang, "LiCaS3: A SimpleLiDAR–Camera Self-Supervised Synchronization Method," IEEE Trans-actions on Robotics, vol. 38, no. 5, pp. 3203-3218, Oct. 2022, doi:10.1109/TRO.2022.3167455. [178]The author proposed the idea and was responsible for the algorithm design,method implementation and manuscript writing. These works were conducted withvithe guidance, extensive technical suggestions and editorial feedback from Prof.Z. Jane Wang. Li Ding and Mazen Abdelfattah helped prepare a complementarydataset.Chapter 5 is based on the following manuscript:• K. Yuan and Z. J. Wang, "A Simple Self-Supervised IMU Denoising Methodfor Inertial Aided Navigation," in IEEE Robotics and Automation Letters, vol.8, no. 2, pp. 944-950, Feb. 2023, doi: 10.1109/LRA.2023.3234778. [176]The author proposed the idea and was responsible for the algorithm design,method implementation and manuscript writing. These works were conducted withthe guidance, extensive technical suggestions and editorial feedback from Prof. Z.Jane Wang.viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Multisensor Fusion and Integration . . . . . . . . . . . . . . . . . 11.2 Three Core Concerns at the Sensor Layer . . . . . . . . . . . . . 21.3 Limitations of Currently Dominant Methods . . . . . . . . . . . . 41.4 Challenges of Hardware-less Solutions . . . . . . . . . . . . . . . 51.4.1 Correlating High Dimensional Data Spaces . . . . . . . . 51.4.2 Hand-Crafted Features . . . . . . . . . . . . . . . . . . . 51.5 Represent Relations in Three Ways . . . . . . . . . . . . . . . . . 61.5.1 Represent Relations as Output . . . . . . . . . . . . . . . 61.5.2 Represent Relations as Model . . . . . . . . . . . . . . . 7viii1.5.3 Hybrid Representation . . . . . . . . . . . . . . . . . . . 81.6 Challenges of Supervised Learning . . . . . . . . . . . . . . . . . 91.7 Challenges of Self-supervised Learning . . . . . . . . . . . . . . 101.8 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . 111.9 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.10 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.1 Robotic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 152.2 Sensor Layer in Robotics . . . . . . . . . . . . . . . . . . . . . . 162.2.1 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.2.2 Extrinsic Calibration . . . . . . . . . . . . . . . . . . . . 212.2.3 Synchronization . . . . . . . . . . . . . . . . . . . . . . 222.2.4 Denoising . . . . . . . . . . . . . . . . . . . . . . . . . . 242.3 Self-Supervised Learning . . . . . . . . . . . . . . . . . . . . . . 252.3.1 Data Augmentation . . . . . . . . . . . . . . . . . . . . . 262.3.2 Pseudo Labels . . . . . . . . . . . . . . . . . . . . . . . 282.3.3 Sequential Self-labels . . . . . . . . . . . . . . . . . . . . 292.3.4 Cross Modality . . . . . . . . . . . . . . . . . . . . . . . 312.3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 313 RGGNet: Tolerance Aware LiDAR-Camera Online Calibration withGeometric Deep Learning and Generative Model . . . . . . . . . . . 333.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . 343.2 RELATED WORKS . . . . . . . . . . . . . . . . . . . . . . . . 363.2.1 Target-based Methods . . . . . . . . . . . . . . . . . . . 363.2.2 Targetless Methods . . . . . . . . . . . . . . . . . . . . . 363.2.3 Deep Learning Methods . . . . . . . . . . . . . . . . . . 373.2.4 Spatial Transform Networks . . . . . . . . . . . . . . . . 383.3 PROBLEM FORMULATION . . . . . . . . . . . . . . . . . . . 393.4 METHODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 413.4.2 Intensity Depth Map and Point Map . . . . . . . . . . . . 43ix3.4.3 RGGNet Architecture . . . . . . . . . . . . . . . . . . . 443.4.4 Distance on SE(3) . . . . . . . . . . . . . . . . . . . . . 453.4.5 Tolerance Regularizer . . . . . . . . . . . . . . . . . . . 453.5 EXPERIMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . 483.5.1 Data Preparation . . . . . . . . . . . . . . . . . . . . . . 483.5.2 RGGNet Training . . . . . . . . . . . . . . . . . . . . . . 493.5.3 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.6 EVALUATION AND DISCUSSION . . . . . . . . . . . . . . . . 503.7 CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . 564 LiCaS3: A Simple LiDAR-Camera Self-Supervised SynchronizationMethod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . 584.2 RELATED WORK . . . . . . . . . . . . . . . . . . . . . . . . . 624.2.1 LiDAR-camera Synchronization . . . . . . . . . . . . . . 624.2.2 Self-supervised Cross-modal Retrieval . . . . . . . . . . . 634.2.3 Monocular Depth Estimation . . . . . . . . . . . . . . . . 644.3 METHODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664.3.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . 664.3.2 Comparisons with Motion-Based Methods . . . . . . . . 664.3.3 Supervised Learning . . . . . . . . . . . . . . . . . . . . 674.3.4 Self-supervised Cross-modal Retrieval . . . . . . . . . . . 684.3.5 Self-supervised Learning with Pseudo Labels . . . . . . . 724.3.6 Deep Neural Networks . . . . . . . . . . . . . . . . . . . 754.4 EXPERIMENTS AND RESULTS . . . . . . . . . . . . . . . . . 764.4.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . 774.4.2 Training Details . . . . . . . . . . . . . . . . . . . . . . . 804.4.3 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . 804.4.4 Comparisons with Hardware Synchronization . . . . . . . 884.4.5 Comparisons with Software Synchronization . . . . . . . 894.4.6 Evaluation of Offline Synchronization . . . . . . . . . . . 904.4.7 Comparisons with Supervised Methods . . . . . . . . . . 904.4.8 Evaluation with LIMO . . . . . . . . . . . . . . . . . . . 91x4.4.9 Lower Beams LiDARs . . . . . . . . . . . . . . . . . . . 944.4.10 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . 944.4.11 Inference Speed Profiling . . . . . . . . . . . . . . . . . . 954.4.12 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 954.4.13 Failed Attempts . . . . . . . . . . . . . . . . . . . . . . . 964.5 CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . 965 IMUDB: Self-Supervised IMU Denoising For Inertial Aided Navigation 985.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . 995.2 RELATED WORKS . . . . . . . . . . . . . . . . . . . . . . . . 1005.2.1 Conventional IMU Denoising . . . . . . . . . . . . . . . 1005.2.2 Supervised Learning on IMU Denoising For Navigation . 1015.2.3 Self-supervised Learning on IMU . . . . . . . . . . . . . 1015.3 METHODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.3.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . 1025.3.2 Self-supervised Learning . . . . . . . . . . . . . . . . . . 1035.3.3 Future-Aware Uncertainty . . . . . . . . . . . . . . . . . 1055.3.4 Deep Neural Networks . . . . . . . . . . . . . . . . . . . 1085.4 EXPERIMENTS AND RESULTS . . . . . . . . . . . . . . . . . 1095.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 1095.4.2 Training Details . . . . . . . . . . . . . . . . . . . . . . . 1105.4.3 Evaluations with Odometry Estimation . . . . . . . . . . 1105.4.4 Latency Profiling . . . . . . . . . . . . . . . . . . . . . . 1165.4.5 Limitations and Future Works . . . . . . . . . . . . . . . 1175.5 CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . 1176 Conclusion and Future Directions . . . . . . . . . . . . . . . . . . . 1186.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1186.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . 1216.2.1 A General and Unified Framework . . . . . . . . . . . . . 1216.2.2 Ensemble Method . . . . . . . . . . . . . . . . . . . . . 1226.2.3 More Sophisticated Learning . . . . . . . . . . . . . . . . 1226.2.4 Streaming Processing . . . . . . . . . . . . . . . . . . . . 123xi6.2.5 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . 123Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124xiiList of TablesTable 3.1 PERFORMANCE COMPARISONS WITH DNN BASED METH-ODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51Table 3.2 PERFORMANCE COMPARISONS WITH A NON-DNN BASEDMETHOD . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51Table 4.1 LAST THREE LAYERS OF h(,) . . . . . . . . . . . . . . . . 76Table 4.2 KITTI AND NEWER COLLEGE DATASET SPECS . . . . . 77Table 4.3 CLASSIFICATION PERFORMANCES OF SELF-SUPERVISEDLiCaS3 AND SUPERVISED LEARNING ONLINE SYNCHRO-NIZATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Table 4.4 CLASSIFICATION PERFORMANCES OF SELF-SUPERVISEDLiCaS3 AND SOFTWARE SYNCHRONIZATION WITH NEWERCOLLEGE MANUAL ANNOTATIONS . . . . . . . . . . . . 83Table 4.5 KITTI: CLASSIFICATION PERFORMANCES OF SELF-SUPERVISEDLiCaS3 OFFLINE SYNCHRONIZATION (ON TRAINING SE-QUENCES) . . . . . . . . . . . . . . . . . . . . . . . . . . . 83Table 4.6 APE RMSE OF BASELINE, MIS-SYNCHRONIZED RUNAND LICAS3 RUN FOR SELECTED KITTI TEST SEQUENCES 93Table 4.7 PERFORMANCES OF LiCaS3 ON 32 BEAMS LIDAR . . . . 93Table 5.1 EUROC DATASET: the POSITIONAL RMSE (M) OVER THEENTIRE TRAJECTORY OF TEST SEQUENCES. . . . . . . 112Table 5.2 TUM-VI DATASET: the POSITIONAL RMSE (M) OVER THEENTIRE TRAJECTORY OF TEST SEQUENCES. . . . . . . 113xiiiTable 5.3 KITTI DATASET: absolute pose error (APE) and relative pose er-ror (RPE) results over the entire trajectory OF TEST SEQUENCES.114xivList of FiguresFigure 1.1 Three core concerns at the sensor layer . . . . . . . . . . . . 3Figure 1.2 LiDAR point clouds projected to the camera RGB image, withmiss-aligned data of the truck. It’s a KITTI dataset example[46]. The yellow dashed line bounding box contains the truckon the camera RGB image. The green dashed line boundingbox contains the truck on the LiDAR projected depth map. . . 3Figure 1.3 Model the relation as the output. When the data from differentsensors sample the same physical world property, they are fedinto one model and output the relation. . . . . . . . . . . . . . 6Figure 1.4 Represent a relation as a model. The input of the model is onesensor data and the output is the data of another sensor, whichis embedded with the relation information. . . . . . . . . . . 7Figure 1.5 Illustration of the difficulty of the LiDAR-camera synchroniza-tion task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Figure 1.6 Overview of the research topics and objectives. . . . . . . . . 11Figure 1.7 The research roadmap of this thesis. . . . . . . . . . . . . . . 14Figure 2.1 Shakey Robot [109] . . . . . . . . . . . . . . . . . . . . . . . 17Figure 2.2 A popular robotic system architecture of modern autonomousrobots. The figure is adapted from the popular open-sourceautonomous vehicle architecture, Project Apollo [152]. . . . . 17Figure 2.3 Illustration of a rotational LiDAR . . . . . . . . . . . . . . . 20Figure 2.4 Illustration of a 9-axis IMU. . . . . . . . . . . . . . . . . . . 21Figure 2.5 Extrinsic transformation between two sensor coordinate systems. 22xvFigure 2.6 Illustration of the LiDAR-camera hardware synchronization. . 23Figure 2.7 Illustration of the PTP. . . . . . . . . . . . . . . . . . . . . . 24Figure 2.8 Illustration of the image reconstruction / denoising task. . . . 26Figure 2.9 Illustration of the task of predicting the order of the patches. . 27Figure 2.10 Illustration of the task of predicting the angle difference be-tween two images. . . . . . . . . . . . . . . . . . . . . . . . 28Figure 2.11 Example of the future video prediction task. . . . . . . . . . . 30Figure 2.12 Overview of the relationship between three tasks studied in thisthesis and the supervised learning approaches. . . . . . . . . . 32Figure 3.1 Demonstration of LiDAR points projection to the RGB image. 42Figure 3.2 The architecture of the proposed RGGNet. . . . . . . . . . . 44Figure 3.3 The architecture of the Tolerance Regularizer. . . . . . . . . . 46Figure 3.4 Examples of RGGNet predictions. . . . . . . . . . . . . . . . 53Figure 3.5 Examples of a bad prediction. . . . . . . . . . . . . . . . . . 54Figure 3.6 Histogram of SEE for RGGNet. . . . . . . . . . . . . . . . . 55Figure 4.1 Conventional synchronization pipeline vs proposed pipeline. . 59Figure 4.2 Illustration of synchronization task definition. . . . . . . . . . 65Figure 4.3 Demonstration of the losses between RGB image candidatesand the depth map. . . . . . . . . . . . . . . . . . . . . . . . 71Figure 4.4 Overview of the proposed LiCaS3 framework. . . . . . . . . . 74Figure 4.5 Strategy of training data generation. . . . . . . . . . . . . . . 78Figure 4.6 Example of Newer College manual annotations. . . . . . . . . 80Figure 4.7 Example good prediction of LiCaS3-h on KITTI. . . . . . . . 84Figure 4.8 Example degenerate prediction of LiCaS3-h on KITTI. . . . . 85Figure 4.9 Online synchronization test runs of two KITTI drives. . . . . 86Figure 4.10 Example good prediction of LiCaS3-h on Newer College. . . 87Figure 4.11 Illustrations of LIMO benchmarks. . . . . . . . . . . . . . . . 92Figure 5.1 Architecture of IMUDB. . . . . . . . . . . . . . . . . . . . . 103Figure 5.2 The future-aware uncertainty and inference process. . . . . . 108Figure 5.3 Example of EuRoC experiments. . . . . . . . . . . . . . . . . 115xviList of AbbreviationsAPE Average Pose ErrorBERT Bidirectional Encoder Representations from TransformersC-3DST Conditional 3D Spatial TransformerCCA Canonical Correlation AnalysisCV Computer VisionCNN Convolutional Neural NetworkCPU Central Processing UnitCRF Conditional Random FieldsDBP Deep Boltsmann MachineDCCA Deep Canonical Correlation AnalysisDL Deep LearningDNN Deep Neural NetworkDOF Degree of FreedomEKF Extended Kalman FilterELBO Evidence Lower BoundFOV Field-of-ViewxviiGAN Generative Adversarial NetworkGDL Geometric Deep LearningGPU Graphics Processing UnitICP Iterative Closest PointIMU Inertial Measurement UnitLIDAR Light Detection and RangingLIMO LiDAR-Monocular Visual OdometryMEMS Microelectromechanical systemsMFI Multi-sensor Fusion and IntegrationML Machine LearningMLM Masked Language ModelMLP Multilayer PerceptronMRR Mean Re-calibration RateMSE Mean Square ErrorMSEE Mean se(3) ErrorNLP Natural Language ProcessingOGM Occupancy Grid MapROS Robot Operating SystemPTP Precision Time ProtocolRMSE Root Means Square ErrorSEE se(3) ErrorSLAM Simultaneous Localization and MappingxviiiSPA Sense-Plan-ActSSIM Structural Similarity Index MeasureSSL Self-supervised LearningSTN Spatial Transformer NetworksTR Tolerance RegularizerVAE Variational AutoEncoderVO Visual OdometryxixAcknowledgmentsFirst and foremost, I shall express the deepest and most sincere thanks for my PhDadviser, Prof. Z.Jane Wang for being such a great mentor, both academically andpersonally. This thesis would never be possible without her continuous support overthe years. Particularly, I owe innumerable thanks to Jane providing me the freedomto explore research topics of my own interests.Second, I would also like to appreciate my wonderful collaborators and co-authors, especially, Dr. Zhenyu Guo, who had lots of insightful discussion withme.Next, I would like to thank my beloved wife, who really helped me a lot onbalancing life and work, and making my boring life colorful. Without her assistingme proof-reading every single paper, this thesis is also not possible.At last, many thanks to my mom, who had been encouraging me to go throughthe PhD training eventually since my childhood. It is her whom provided me asteady life and urged me to never stop learning. My heart was broken when shepassed away in my second year of PhD and she could never make it to my graduationceremony. However, her spirit and inculcation will never fade, which is deeplybaked into this thesis.xxChapter 1IntroductionRecent advances of computing power, sensing technologies and artificial intelli-gence help empower the robots to conduct complicated tasks, such as autonomousdriving [19, 149], robot-assisted surgery and so on. In order to achieve a robustsensing of the surrounding environment, robots are usually armed with multipletypes of sensors. In fact, many complex tasks [2, 52] were reported to yield betterperformance with using multi-modal sensors than using a single sensor. Never-theless, for various multi-modal tasks at the application layer, the sensor layer isexpected to provide accurate calibration, synchronization and noise-free data. Suchtasks at the robot sensor layer are critical pre-requisites for the downstream applica-tions. In the following subsections, we first briefly review the multi-sensor fusionand integration topic; we then systematically discuss the three core concerns at thesensor layer; next, limitations of the current dominant methods and the challengesin several potential directions are described; and at last, the research objectives andcontributions of this thesis are summarized, followed with the thesis outline.1.1 Multisensor Fusion and IntegrationRobots, as a sub-category of the general mechatronics system, are composed ofsensors, actuators, communication systems and compute units. For a robot, sensorsplay a key role in observing and digitizing the surrounding environment.Generally speaking, every sensor has its own limitations and can fail or degrade1in certain conditions during operation. For example, camera sensors might not beuseful when heavy snow blocks the lens. Relying on a single sensor or a singletype of sensors could limit the applications of certain systems and potentially causesafety and reliability concerns.In order to reduce the risk of task failure and provide information redundancy,a common practice is to employ various types of sensors and conduct multisensorfusion and integration (MFI) [100]. For example, autonomous vehicle platforms areoften equipped with different types of sensors, such as Waymo [149] and KITTI[46], which are armed with the camera and Light Detection and Ranging (LiDAR)sensors. When the lighting condition becomes poor, the cameras can fail to detectthe cyclist in a dark tunnel, leading to possible hazard consequences. However,when the LiDAR is added, the robust 3D point clouds of the cyclist can be obtained,regardless of lighting conditions. In contrast, a mirror would mislead the LiDAR toperceive ghost points whereas the camera can better semantically detect it.1.2 Three Core Concerns at the Sensor LayerMFI methods can be achieved at different levels, e.g, estimation methods, classifi-cation methods and inference methods [100]. However, what is in common is thatthere are three core concerns at the sensor layer, as illustrated in Fig. 1.1:• spatial relation between two sensors: the coordinate systems of any twosensors should be spatially correlated so that their signals can be transformedbetween the coordinate systems. The task to derive such relations is calledextrinsic calibration;• temporal relation between two sensors: for the respective signals from twosensors with the same timestamp, the signals should be sampled at the samemoment of the environment. Therefore, such synchronization tasks are ofgreat importance;• noise relation for each sensor itself: the sensor signals should accurately andconsistently reflect the state or state changes of the agent or the underlyingenvironment. Denoising tasks are thus necessary to be studied.2Sensor 1 Sensor 2Spatial RelationTemporal RelationNoise RelationNoise RelationFigure 1.1: Three core concerns at the sensor layerFor any robotics system, the above three relations among all sensors at thesensor layer are important for the downstream application layers. Inaccurate ex-trinsic transformation, mis-synchronized data or noisy signals could jeopardize thedownstream task results and potentially cause fatal consequences. Below Fig. 1.2illustrates one instance of LiDAR point clouds projected to the camera image. Itcan be seen that the truck observations on the camera and depth map do not align.Any of the following may cause this misalignment: 1) the extrinsic calibrationbetween LiDAR and camera is not accurate, thus resulting in wrong projection; 2)the LiDAR may have a latency and the overlay-ed two modalities are not from thesame timestamp; and 3) the LiDAR data is noisy and the packets shift.Figure 1.2: LiDAR point clouds projected to the camera RGB image, withmiss-aligned data of the truck. It’s a KITTI dataset example [46]. Theyellow dashed line bounding box contains the truck on the camera RGBimage. The green dashed line bounding box contains the truck on theLiDAR projected depth map.3In reality, the above three relations will never be perfect and the goal is to reducethe errors to a tolerable range. Preparing calibrated, synchronized, and denoisedmulti-sensor data within the sensor layer will be a pre-requisite for any multi-modalmulti-task robot. These three core relations at the sensor layer are the main focus ofthis thesis.1.3 Limitations of Currently Dominant MethodsFor each relation at the sensor layer, there are different methods to reduce thecalibration, synchronization and denoising errors. What is in common is thatcurrently the dominant approaches heavily depend on expensive hardware.First, to achieve the satisfactory accuracy and precision, one straightforwardapproach is to employ high-end and expensive sensors. For example, many au-tonomous vehicles [46] are equipped with high-cost units, such as high-beamLiDARs or high-end IMUs. These requirements fundamentally limit the cost effi-ciency and discourage the wide deployment of affordable robots.In addition, specific offline and online hardware setups are usually required,such as calibration rooms [46] and hardware synchronization interfaces and cabling[31]. These setups are expensive and also introduce reliability risks. For example,even thought it might be debatable, when a cable wears out over time, which couldlead to degraded hardware synchronization, it would be non-trivial to diagnose andmonitor the situation.Last but not the least, many hardware setups contain bidirectional communica-tion among the compute unit and the sensors, such as the widely adopted PrecisionTime Protocol (PTP) [31] synchronization method. In another word, the processcan only happen online. For users who do not have access to the robotic systemwhich adopts the PTP but still yields large time off, the synchronization can hardlybe achieved offline afterwards.In summary, currently the dominant methods to tackle the major tasks at therobot sensor layer heavily rely on hardware. It is worth exploring more efficientapproaches that could reduce the hardware dependency.41.4 Challenges of Hardware-less Solutions1.4.1 Correlating High Dimensional Data SpacesOne underlying reason why sensor layer data processing relies on the hardwareis that sensor data is usually high dimensional and directly correlating two highdimensional data spaces (of two sensors) could be challenging.Taking Fig. 1.2 as an example, let us assume that the temporal synchronizationerror is the only concern. Solving this problem with using certain hardware canbe straightforward, as what the KITTI [46] platform actually did: every 360◦LiDAR sweep will trigger the camera exposure. Besides the high cost disadvantageas reviewed above, such hardware synchronization methods also yield a reducedsampling rate of the camera. In the meanwhile, without the help of the hardwaresetup, the time offsets need to be derived from the sensor data afterwards. However,there is no universal mechanism or theorem for such a mapping from the two sensordata to the time offset. Similarly, there is no universal theorem for any mappingsfrom sensors to the relations.Therefore, analytically correlating the high dimensional sensor data to derivethe calibration, synchronization or noise relations is challenging.1.4.2 Hand-Crafted FeaturesIn the literature, there are indeed hardware-less attempts for certain problems at therobot sensor layer. For example, [114] assumes that the reflectivity from the LiDARand the intensity from the camera are correlated and the correlation corresponds tothe calibration parameters. With such an assumption, the extrinsic calibration canbe formulated as an optimization process of maximizing the reflectivity-intensityentropy. Another example is matching line features [67, 85]. In [67, 85], it assumesthat the line is a very general feature, which always exists in the operating environ-ment. Therefore, after extracting the lines from each sensor, the parameters, whichlead to the best line matching, are the results. The underlying mechanisms of theseattempts are that, with certain assumptions, hand-crafted features can be adoptedto derive the relation parameters. Though in some scenarios, specific hand-craftedfeatures can achieve good results, such practices may not be feasible and do not5generalize in all scenarios (e.g., valid hand-crafted features might be missing).Common limitations of hand-crafted features have been addressed and tackledby the recent effort in computer vision and deep learning. With the data-drivenmanner, it is possible to automatically learn promising features for certain tasks,without making strong assumptions. Even though numerous studies have beenconducted in deep learning, the majority of the work on robotics have focused onthe application layer tasks. To our knowledge, only a few effort [64, 136] havefocused on employing deep learning on the sensor layer tasks, however they havenot achieved satisfactory results yet. As elaborated in later chapters, we believethe underlying cause is that all these work simply treat the sensor layer task as theapplication layer and utilize very naive approaches. This motivates us to explorethis research gap and develop advanced deep learning methods to tackle the sensorlayer tasks. To get started, in the following subsection, we describe three differentrepresentations of the relations at the sensor layer.1.5 Represent Relations in Three Ways1.5.1 Represent Relations as OutputThe most straightforward way to represent a core relation at the sensor layer is tomodel it as the output, i.e., feeding sensor data or sensors features into a model andyielding the relation representation (as the output), as illustrated in Fig. 1.3:SensorDataSensorRelationModelInput OutputFigure 1.3: Model the relation as the output. When the data from differentsensors sample the same physical world property, they are fed into onemodel and output the relation.6Taking the LiDAR-camera calibration problem as an example for illustration:a specific model, taking in the point clouds and the RGB image as the input, willoutput the extrinsic calibration matrix. In this case, the extrinsic calibration, i.e., thespatial relation between the LiDAR and camera sensors, is represented as the output.Mathematically it is formulated as follows:T = Q(Xl,Xc,ω) (1.1)where the input Xc and Xl are camera and LiDAR data, the output T representsthe extrinsic calibration matrix and the model Q(,) represents the model withparameters ω .1.5.2 Represent Relations as ModelIn some scenarios, representing a core relation as a model itself could be moreconvenient.Sensor DataSensor DataModelInput OutputFigure 1.4: Represent a relation as a model. The input of the model is one sen-sor data and the output is the data of another sensor, which is embeddedwith the relation information.For example, we may develop a specific model which takes in the raw IMU dataand output the noise or the denoised data. In reality, the noise itself is often of lowinterest and outputting the denoised IMU data is more preferred:7Xd = f (Xr,θ) (1.2)where the input Xr and the output Xd represent the IMU raw data and denoisedIMU data respectively, and the model f (,) represents the denoising model withparameters θ .When there are multiple models, Eqn. (1.2) can be generalized as follows:Xd = f1(Xr,θ1)◦ f2(Xr,θ2) · · · ◦ fn(Xr,θn) (1.3)where each fi(,) represents a different model or task and ◦ represents the composi-tion of different functions.1.5.3 Hybrid RepresentationIn fact, the above model and output representations are not conflicting and they canbe combined. Taking the LiDAR-camera synchronization task as an example, firstthe synchronization can be represented as a model which takes in the camera dataXc as the input and outputs the corresponding synchronized LiDAR representationXl:Xl = g(Xc,Θ) (1.4)where g(,) represents the synchronization model parameterized with Θ.In the meanwhile, the synchronization time offset can also be represented as theoutput:S = h(Xl,Xc,Ω) (1.5)whereas the model h(,) represents the function with parameters Ω, which takes in8the LiDAR data Xl and camera data Xc as the input and output the offset variable S.As long as there is a way to connect S and g(,), the above two modellings canbe conducted jointly. We refer such a modelling as hybrid representation.1.6 Challenges of Supervised LearningRegardless of the relation representations, as long as the ground truth is available, itis straightforward to model such representations with deep neural networks (DNNs)under the supervised learning framework. Unfortunately, in practice such groundtruth information is not always available and supervised learning may not be realisticat the sensor layer.At the application layer, the majority of tasks are modelled as in Eqn. (1.1)whereas the input is the fused sensor data and the output is the semantic task space,such as the 3D object bounding box space, the semantic segmentation space and soon. One major difference is that the output of the application layer tasks does notrepresent the relations between the inputs. Another fact worthy being noted is thatthe sensor relations generally are significantly less interpretive for human eyes. As aconsequence, manually annotating the sensor relations, such as extrinsic calibration,can be hard in practice.Below Fig. 1.5 demonstrates one example. It can be seen that visually it can behard to tell which one is synchronized. For the high-frequency IMU data, it is evenmore difficult to visualize and annotate.Figure 1.5: Overlay of the LiDAR projected depth map over the camera image,an example from the Newer College dataset [125]. Three images areconstructed with the same depth map, where the camera data is from acontinuous sequence.9Due to the challenge of obtaining accurate manual annotations, there are barelyany public dataset on learning based calibration, synchronization or denoising, withaccurate human labels. This makes the well-studied supervised learning methodshardly usable in sensor layer tasks, and leaves the self-supervised learning (SSL)as one of the few possible options (for instance unsupervised learning and transferlearning), to continue on the data-driven route.1.7 Challenges of Self-supervised LearningSelf-supervised learning is still a very active research area. Even though tremendousprogress has been made recently in deep learning, SSL is still an open problem inresearch and not widely adopted in industry yet. For example, the state-of-the-artSSL performance [186] on the ImageNet classification benchmark, one of the mostfundamental task in computer vision, are still largely lower than that of supervisedapproaches [96] and can not satisfy the industry needs yet. The trend of using largeSSL models [28] in the natural language processing (NLP) domain is not applicablehere as they are too computationally expensive for the sensor layer, which generallyrequires low latency.In fact, there indeed were a few early attempts [64, 136] of adopting SSLon the LiDAR-camera extrinsic calibration problem. However, these early effortall simply treat the problem the same as application layer tasks. For example,RegNet [136] directly regresses the LiDAR and camera data representation to thecalibration representation, without considering the underneath geometry. In fact, theextrinsic calibration matrix belongs to a known space, which has explicit metricsfor the geodesics. Such a prior of the sensor relation should be used as the objectivefunction. Generally, the underlying differences and challenges of the self-supervisedlearning on sensor layer tasks have not been addressed and studied systematically.We believe this is a major contributor to the satisfactory performance [151, 177] ofthe early attempts.The challenges described above motivate us to delve deeper into this sensorlayer and systematically explore the self-supervised learning direction for tasks atthe sensor layer.101.8 Research ObjectivesLiDAR CameraCalibrationSynchronizationDenoiseIMUSynchronizationSynchronizationCalibrationCalibrationDenoiseDenoiseFigure 1.6: Overview of the research topics and objectives. Green dashedlines indicate the specific relations which this thesis will study.The overall goal of this thesis is to conduct a systematic study of self-supervisedlearning on representative tasks at the robot sensor layer. In our work, we use thepopular LiDAR-Camera-IMU system as the study target. As shown in Fig. 1.6, thereare 9 relations in total. Since IMU sometimes is built-in with LiDAR or camera,in our work, for spatial and temporal relations, we focus on the LiDAR-cameracalibration and synchronization tasks. For the noise relation, as a representativetopic of interest, we focus on the IMU denoising task with the IMU sensor data only,since low-cost IMUs can be really noisy and IMU denoising is of broad interest inpractice.First, we start from the early explored LiDAR-camera extrinsic calibrationtask with using SSL [64, 136]. The focus of this research [177] is to identify theoverlooked issues and validate the hypothesis of why the previous SSL methods11yield unsatisfactory performance. In this task, we represent the spatial calibrationrelation as the output.Next, as a natural extension, the LiDAR-camera temporal synchronizationtask is explored. However, it is later shown in our study that the output spacemodelling SSL as in the calibration task is not sufficient for this more challengingtask. Therefore, the objective of this work is to develop a more sophisticated SSLapproach for the temporal synchronization task, which turns out to be a hybrid spacerepresentation.At last, to provide a comprehensive coverage of both the relation representationsand robot sensor layer tasks, the IMU denoising task is studied. Our proposed SSLmethod represents the noise relation as the model.In summary, we choose representative topics to study all three sensor corerelations and explore three different relation representations.1.9 ContributionsAs a high-level statement, the contributions of this thesis are as follows:• As the first attempt, this thesis systematically discusses the three core relationsat the robot sensor layer and reviews the challenges of the dominant andalternative approaches;• This thesis systematically studies SSL approaches at the robot sensor layerby representing the relations in the output, model or hybrid spaces.Specifically, for each studied task, more specific contributions are described asfollows:• We elaborate on the overlooked geometry considerations in the existing deeplearning based LiDAR-camera calibration methods, and propose the RGGNet[177], the first SSL work at the robot sensor layer that demonstrates competingperformance when compared with the conventional motion-based approach;• We address the challenges of the conventional LiDAR-camera synchroniza-tion pipeline and propose the first deep learning framework, LiCaS3 [178],12which does not rely on hardware synchronization and can achieve superiorperformance;• We propose the first self-supervised learning method for IMU denoising forinertial-aided navigation.1.10 Thesis OutlineThe rest of the thesis starts with Chapter 2, which gives a comprehensive overviewof the robot sensor layer and self-supervised learning, the two key context conceptsin this thesis.Next, in Chapter 3, RGGNet, a tolerance aware LiDAR-camera online calibra-tion method, is introduced. In Chapter 4, LiCaS3, the first deep learning frameworkfor the LiDAR-camera synchronization task, is described. In Chapter 5,IMUDB,the first self-supervised learning medthod on IMU denoising for inertial-aidednavigation, is presented. At last, in Chapter 6, we summarize this thesis.13Deep LearningSelf-supervised LearningSpatial Temporal NoiseOutput Hybrid: Output & ModelModelFirst SSL LiDAR-Camera Calibration Achieving Parity Performances as Conventional Methods(Chapter 3)Sensor Relation TypeSensor Relation Representation Method & Contribution First DL & SSL LiDAR-Camera Temporal Synchronization Method(Chapter 4)First SSL IMU Denoising Method for Navigation(Chapter 5)LiDAR Camera IMUSensor TypeDeep Learning Challenge: Lack of Efficient Human Annotations for Dominant Supervised Learning(Chapter 1)Self-Supervised Learning at Robot Sensor LayerConventional Approach Challenge: Expensive Hardware Dependencies(Chapter 1)ChallengesFigure 1.7: The research roadmap of this thesis. This thesis focuses on tasks at the robot sensor layer with exploringself-supervised learning. Relations at the sensor layer have been studied with various relation representations.14Chapter 2Background2.1 Robotic SystemsA robot is an elevated class of mechatronics and a collection of sensors, actuators,communication systems and compute units that can interact with the environmentfor performing various tasks [79]. The history of robotics can be dated back to theShakey robot at the Stanford University in the late 1960s [109], as shown in Fig. 2.1.Shakey consists of three layers, called the sense-plan-act (SPA) paradigm [110]:• sense: build an internal world model from the camera;• plan: create a goal and a series of actions to achieve the goal;• act: execute the actionsSPA was the earliest robotic architecture which the community had adoptedin the following years. The main drawback of SPA is that the information is keptabstracted in one direction, from the upper stream sensing to the downstream actu-ation: there is neither a feedback loop nor a residual connection directly betweensensors and actuators. In early 1980s, when the compute capability was still limited,the planning step could be time consuming, which would lead the SPA robot intoconstantly stopping and potentially dangerous situations. To address this limita-tion, one classic solution called the Subsumption architecture (also known as the15behavior-based architecture) was proposed by Brooks in late 1980s [14], with finite-state machines which directly connect the sensors to the actuators, and it showedsuccess in many robots. However Subsumption robots struggled with achievinglong-term goals, since such a design was essentially missing. Besides, it was alsoproven that achieving the optimized behavior under this architecture is almost im-possible [79]. This eventually led to the development of the layered robot controlarchitecture that is modularized and allows the individual layers communicate witheach other [7, 75, 123]. For example, ROS (Robot Operating System) [123], themost widely adopted open-source middleware in the robotics community today,has been empowering diverse robots in executing various complicates task, such asautonomous driving [152], and helped accelerate the development of modern robots[180].2.2 Sensor Layer in RoboticsFig. 2.2 illustrates one modern robotic system architecture [152], including thehardware layer and the software layer. At the software layer, operating systemallows each sub-layer in software programmable; the middleware layer is responsi-ble for communication between each module in the application layer; the sensorlayer transmits and prepares various sensor data for applications; and finally theapplication layer utilizes the well-calibrated, synchronized and clean sensor data forall types of complicated tasks. This thesis focuses on the robot sensor layer, whichbridges the hardware and software layers.In the following subsections, we first briefly introduce the most commonlyemployed sensors for modern robots, and then introduce the three key tasks at thesensor layer: calibration, synchronization and denoising.16Figure 2.1: Shakey Robot [109]Robotics SystemHardware LayerSoftware LayerSensor LayerOperating SystemApplication LayerMiddleware LayerPerception Planning Control ...CalibrationSynchronizationDenoising...Figure 2.2: A popular robotic system architecture of modern autonomousrobots. The figure is adapted from the popular open-source autonomousvehicle architecture, Project Apollo [152].172.2.1 SensorsThe sensors on the robots can be grouped into two major categories: exteroceptivesensors and proprioceptive sensors. Exteroceptive sensors obtain signals from theenvironment, including camera, Light Detection and Ranging sensor (LiDAR),RADAR, ultrasonics and so on. Proprioceptive sensors measure the states of therobot itself, including IMU, wheel encoder, GPS and so on. Below we briefly reviewthe three types of sensors which we study in this thesis.CameraCamera is a widely deployed sensor for autonomous robots [18, 19, 46, 152],which can yield dense colored images at a high sampling frequency. From drones[18] to autonomous vehicles [19, 46, 152], camera plays an important role inthe perception system. Common perception tasks include classification [80], 2Dobject detection [130], 2D image segmentation [57], and so on. Since 2012, whenAlexNet [80] demonstrated significant improvements on the ImageNet Large ScaleVisual Recognition Challenge [80], deep learning has been shown to have close-to-human performance in the camera image space on various tasks. The recentrapid development of graphics processing units (GPUs), which demonstrates thesuperior computing on deep learning optimization process than the traditional centralprocessing units (CPUs), accelerates the employment of camera based perceptionon more robots.In this thesis, we focus on the monocular camera. While such a camera sensorcan be of low cost, the main challenges are as follows:• the camera data quality is sensitive to light conditions;• the camera lacks physical measurements, such as depth;• the FOV of the camera is relatively small.Therefore, camera based perception might not be ideal for path planning in thelonger range and the false results could be vital. In addition, the camera hardwaremay also get degraded in raining or foggy days.18LiDARLiDAR directly shoots pulsed laser to collect the feedback to measure the distancebetween the sensor and obstacles. Laser range finder is another commonly usedname in early days [153]. While LiDAR also has a long history, whose earlyapplications can be dated back to the Apollo 15 mission in 1970s, what really makesLiDAR start drawing public attention on autonomous robots is Stanley [153], whichis the first vehicle ever completing the DARPA Grand Challenge in 2005.While there are various kinds of LiDARs [124] and this industry is still underactive research and development, rotational scanning stays the most prominentmechanism for the commercial deployed LiDARs on autonomous vehicles [149,152]. As shown in Fig. 2.3, the mirror inside rotates 360◦ and yields streamingpackets of point clouds. Compared with the camera data, on one hand, LiDARsees further than cameras and yields direct range measurements, which allows thesystem to be able to make earlier predictions and to directly sense obstacles withoutsemantic understanding. On the other hand, the sampling frequency of the LiDARpoint clouds data is usually lower, such as 10 Hz [46], and the data volume issignificantly larger. [119] has a great summary of distinguished properties of pointclouds, different from the images: 1) unordered; 2) interactions among points; and3) invariance under transformations. In addition, the LiDAR point clouds are alsovery sparse and generally the cost of a LiDAR is significantly higher than that of acamera.Even though the above challenges persist, many breakthroughs have been madeand LiDAR has demonstrated tremendous values on perception and simultaneouslocalization and mapping (SLAM) tasks. For example, [179] demonstrated thata visual-LiDAR complementary system can easily outperform vision-only frame-works. Various point clouds representations [81, 90, 119, 190] have been proposedto yield promising performance on various perception tasks, such as 3D objectdetection [90, 141, 168, 190] and 3D semantic segmentation [165]. It is also shownthat the LiDAR-camera based 3D perception tasks are more robust to adversarialattacks than the vision only [2, 3].In summary, nowadays LiDAR has been one of the key sensors on most au-tonomous vehicles and the related research areas have been active.19Figure 2.3: Illustration of a rotational LiDARIMUIMU has a long history, dating back to 1930s [5]. Fundamentally, IMU is composedof gyroscope and accelerometer. Gyroscope measures the three-axis angular veloci-ties, and accelerometer yields the three-axis linear accelerations. In addition, someIMUs also contain magnetometers, which can measure the magnetic field of Earth.An IMU, which include gyroscope, acclerometer and magnetometer, is called 9-axisIMU and it is the most common type applied to robotics navigation.20IMUxyzFigure 2.4: Illustration of a 9-axis IMU. ωx, ωy and ωz represent angular veloc-ity on each axis respectively; ax, ay and az represent linear accelerationon each axis; gx, gy and gz represent gravity vector on each axis.Even though the cost of modern IMUs varies, with the developments of themicro-fabrication technology, IMU sensor can be of low cost as well as small size.Such advantages empower the possibility of packaging IMU together with othersensors such as LiDAR or cameras. For example, both Ouster OS1 series LiDARand Intel Realsense depth cameras have built-in IMU [125], which comes with arelatively accurate factory calibration and rigidity during operation. Consideringthis, extra extrinsic calibrations between IMU and LiDAR or camera might notbe needed in many platforms with these built-in IMU cameras or LiDARs. Themain challenge of IMU usage is thereby the accumulated errors, making IMUdenoising fundamental and important. This is especially true for low-cost Micro-electromechanical systems (MEMS) based IMUs, which is subject to noises fromboth manufacturing or operations [71]. For example, as demonstrated in [71], alow-cost Crossbow IMU300CC IMU can yield a 3 to 5 meters root mean square(RMS) differences due to noises.2.2.2 Extrinsic CalibrationExtrinsic calibration is to obtain the spatial transformation matrix to describe therigid relations between the coordinate systems. For any arbitrary two sensors, S1 and21S2, in the same robotic system, the homogeneous transformation can be representedas a 4× 4 matrix T ∈ SE(3) (Special Euclidean Group 3D). T is composed ofthe rotation matrix R with size 3× 3 and the translation vector t with size 3× 1:[R t0 1].Figure 2.5: Extrinsic transformation between two sensor coordinate systems.With the extrinsic calibration matrix between two sensors, any observed signal,signal representation or perception task output can be transformed from one toanother for fusion.Ideally, the extrinsic calibration between two sensors should be fixed. In reality,during the operation, the extrinsic calibration parameters can shift due to variousreasons such as vibration or thermal-strains. Therefore, for a robot, conductingextrinsic calibration online is necessary.2.2.3 SynchronizationIn a complex robotic system, the data from various sensors should be stamped withthe correct time. Otherwise, the errors will aggregate all the way to the actuation andpotentially result in fatal consequences. Below we briefly review two conventional22synchronization methods.The most straightforward synchronization method is to fire each sensor at thesame time with the same frequency. Such a process is called hardware synchro-nization. Popular open source autonomous vehicle datasets such as KITTI [46]and NuScene [19] all implement this hardware synchronization method. Morespecifically, the camera exposure will be triggered every 360◦ LiDAR sweep, asshown in Fig. 2.6. First, not all sensor supports the trigger control. Second, suchan approach limits the sensor final yielded frequency to the lowest of both. Forexample, KITTI platform uses PointGray Flea2 color cameras (FL2-14S3C-C),which is capable of running at 15 Hz. However, due to the exposure control, thecamera eventually yields 10 Hz to synchronize with the LiDAR frequency.LiDARFigure 2.6: Illustration of the LiDAR-camera hardware synchronization.Another alternative strategy is to establish a protocol for a group of sensors andconduct bidirectional communication to synchronize the timestamps afterwards.A typical protocol is the Precision Time Protocol (PTP) [31], as illustrated in Fig.2.7, which dynamically establishes a communication between primary sensorsand secondary sensors. Similarly, such a protocol is not always supported by allinterfaces.23Primary SensorSecondary SensorPrimary TimeSecondary TimeSyncFollow UpDelay RequestDelay ResponseFigure 2.7: Illustration of the PTP.Overall, conventional methods directly synchronize the sensors without leverag-ing the contents of sensor data, and thus they heavily depend on hardware.2.2.4 DenoisingAny sensor has its designed operating conditions and noise is inevitable. Gener-ally when designing application algorithms, noises should be considered; whilemeanwhile it is desired that downstream applications are robust to noises.Image and point clouds denoising are both classic topics that have been studiedfor a long time [37, 154, 188]. Regardless of specific methods, denoising approachescan be broadly grouped into hand-crafted feature based and automated feature based(where automated features are obtained via machine learning). In fact, currently, thedominant deep learning methods are relatively more robust to noises, since it is acommon practice to add random noises to the data for data augmentation in trainingor even to learn from noises [11].Compared with LiDAR and camera data, IMU denoising could be more challeng-ing. Besides the significantly higher sampling frequency, IMU is one proprioceptivetype of sensor, which measures the states of the system. Namely, for state estimationtasks, such as odometry estimation, the algorithm performance could be subjectto IMU data precision. In addition, IMU noise is not as visible as in LiDAR orcamera data which can use the real-world objects as ground truth. Last but not theleast, currently the conventional IMU denoising methods are hand-crafted feature orheuristics based and deep learning has not yet been popular in most IMU-involvedrobotics tasks, such as in the localization or odometry estimation tasks. For example,24one straightforward method to denoise IMU data is low-pass filtering [38], whichhowever could lead to signal distortion and loss of information. Another way tocompensate the impacts of the noise at the task level [47] is using Kalman Filteror its variants [87]. For example, Open-VINS [47], one popular visual odometryestimation baseline method adopts Extended Kalman Filter (EKF).Therefore, exploring the direction of deep learning based IMU denoising is ofgreat interest and can be promising.2.3 Self-Supervised LearningWhen a large amount of labels are available, supervised learning can achieve reallyimpressive performance on many tasks [57, 80, 81, 130]. However, annotations canbe expensive and in many scenarios can be hardly obtained. Unfortunately, learningfrom unlabelled data, i.e., unsupervised learning, is challenging and usually yieldsless satisfactory performance. One promising solution to this dilemma is learningfrom free labels, namely, self-supervised learning [83].In recent years, self-supervised learning (SSL) has achieved great success onnatural language processing (NLP), such as Bidirectional Encoder Representationsfrom Transformers (BERT) [28] and Embeddings from Language Models (ELMo)[117]. Inspired by that, computer vision (CV) is catching up with recent efforts onSSL, such as Masked Autoencoders [58], which explores transformer architecturesfor CV self-supervised learning.In robotics, popular SSL tasks include the future occupancy grid map (OGM)prediction [62, 105], monocular depth estimation [49, 69, 189], terrain mapping[147], etc..In addition to the common open problems as in supervised learning, such astheory foundation, another obvious challenge of SSL [95] is that free labels are notavailable for all tasks. In another word, formulating a specific problem of interestas a self-supervised learning task is challenging. For example, 3D object detectionrequires semantic annotations of 3D bounding boxes [81] which cannot be obtainedfor free. Even though there are early explorations, such as using geometry heuristicsto generate pseudo labels [89], their performance are unsatisfactory and the learnedrepresentation is usually used as a pre-trained model for supervised fine-tuning.25Based on how the free labels are generated and where the priors come from,below we review several common SSL methods.2.3.1 Data AugmentationData augmentation is a common practice used in training neural networks. Such atechnique can also be adopted to self-supervised learning when the relation betweenthe input and the output is explicit. One example task is image denoising or recon-struction [159]: we input an noisy or corrupted image and output a reconstructedor denoised image. During training, a random crop operation can be applied toeach input example and the original image will be used as the ground truth tosupervise the training. Fig. 2.8 demonstrates one specific example with a squaremask cropping the input image, yielding a reconstructed or denoised image. Theloss is computed with the predicted image and the original uncropped image.EncoderDecoderInput Output Ground TruthLossFigure 2.8: Illustration of the image reconstruction / denoising task.The above learning schema can be extended to other modalities and in fact it hasachieved great success in NLP. For example, the masked language model (MLM) isone significant sub-task in BERT [28]. The idea of MLM is similar as in Fig. 2.8:applying a mask to the tokenized sentence and predicting the full sentence, withusing the original sentence as the free label.In addition to the modality, such a learning scheme can also be generalized totransform operations, such as rotation [78], distortion [8], adding Gaussian noises,etc., as long as the operation is explicit.The input and the output of the above example belong to the same space ormodality, such as the rotated image versus the original image, and the maskedsentence versus the original sentence. The major difference is the domain andthe domain gap can be explicitly described as an operation. Even when the input26and output modalities are different, free labels can still be generated via dataaugmentation. Fig. 2.9 illustrates one example task: first evenly cropping the inputinto four patches and tokenizing the top left, top right, bottom left and bottomright patch as 1, 2, 3 and 4; then a random shuffle being applied to four patches,e.g., resulting in the [1,4,3,2] order; then the order shuffled patches being fedinto a neural network, outputting a vector indicating the predicted order; the order[1,4,3,2] being used as the free label.Output Ground TruthLossInputCrop into Patches Shuffle24311 23 4123414321432Figure 2.9: Illustration of the task of predicting the order of the patches.Fig. 2.9 can be further extended to the case with multiple inputs. As shown inFig. 2.10, the inputs are the original image and the rotated image; the output is theangle difference between the two images; the label is the known 180◦. In fact, therotated image can be the data from a different modality, such as point clouds or athermal map.27Output Ground TruthLossInputs170 180Figure 2.10: Illustration of the task of predicting the angle difference betweentwo images.In summary, as long as the relation between input(s) and output spaces can bedescribed as an explicit operation, data augmentation technique can be employedto generate free labels. In this thesis, such a technique has been applied to theLiDAR-camera calibration task (Chapter 3) and IMU denoising task (Chapter 5).2.3.2 Pseudo LabelsWhen the relation between input(s) and output space cannot be described as anexplicit operation, data augmentation would not be feasible. For such cases, onefeasible approach is to leverage pseudo labels [9, 36, 54, 155]. Pseudo labels can beviewed as ground truth labels with noises.For example, [155] proposed a self-supervised 2D object detection method.Firstly, LiDAR data is segmented with a fast heuristic point clouds segmentationmethod [13]. Next, the segmentation results are projected to the camera RGBimage. Last, both the LiDAR data and 2D RGB image with the projected boundingboxes (derived from the 3D segmentation task) are fed into a neural network foran iterative segmentation self-labelling process. Another example is deep pointclouds clustering [9, 155], which incorporates the traditional k-means to providesupervision signals.28Since pseudo labels have natural limitations, such a self-supervised learningtask is usually jointly trained with other supervised tasks [36, 155] to learn betterrepresentations. In this thesis, such a technique has been employed in the LiDAR-camera synchronization task (Chapter 4).2.3.3 Sequential Self-labelsFor certain specific tasks whose input or output are sequential data, such a sequentialrelation can be used as the self-supervision signal. Fig. 2.11 illustrates a futurevideo prediction task [142], and, as we can see, the sequence itself can easily playthe role of the free ground truth. This learning schema can also be applied toother modalities, such as LiDAR point clouds [102] and the occupancy grid map[62, 105].29Now FuturePastLossInput OutputGround TruthFigure 2.11: An example of using the sequential RGB images for future videoprediction task. t0 represents the current frame, corresponding to oneRGB image. [t−3, t−2, t−1] represent the past three frames respectively.[t1, t2, t3] represent the future three frames respectively. The model takesin the past three frames and the current frame, and outputs future threeframes.In practice, the sequential data is often discrete and has a certain resolution.For example, the seven camera frames in Fig. 2.11 were taken at 10 HZ, i.e., thetime interval between each two neighbour frames is 100 ms. Sometimes, it is ofinterest to predict missing frames within the 100 ms duration for a higher temporalresolution. It should be noted that, for this task, the exact ground truth does notexist and it is less straightforward than the future prediction task. The sequentialrelation can still be used to construct the temporal consistency objective functions[127] to self-supervise the training. The sequential consistency self-supervision canalso be applied to other modalities, such as LiDAR point clouds [29].In this thesis, such a technique has been employed in both the LiDAR-camera30synchronization task (Chapter 4) and IMU denoising task (Chapter 5).2.3.4 Cross ModalityIn certain systems when multiple sensors are well calibrated, synchronized anddenoised, one sensor data can be used as free labels for another sensor.Monocular depth estimation [49, 69, 189], which is an important and relatedtopic in Chapter 4, is one typical task that can be cross-modal self-supervised. Aslong as the LiDAR and the camera has overlapped field-of-view (FOV), the LiDARprojected depth map can serve as the free labels. Since 2014 [32], self-supervisedmonocular depth estimation has been actively researched. Chapter 4 will providea review with more details. Besides, there are also research to do the reverse:generating pseudo LiDAR point clouds from camera images [164]. Theoretically,any two sensors can adopt such a learning scheme, as long as they satisfy the aboveconditions.More generally, it is not necessary that the supervision signal must be sensor rawdata. A specific task output of one sensor can also be the supervision signal for atask of another sensor. In fact, the second example in subsection 2.3.2, which adoptsthe LiDAR segmentation task results to supervise the camera object detection task,with heuristics [13], is also a cross modal self-supervised method. In the featurespace, there are efforts made on adopting cross modal self-supervised learning onthe visual speech recognition task [25].In this thesis, such a cross modal SSL approach has been employed in theLiDAR-camera synchronization task (Chapter 4).2.3.5 SummaryHere we have reviewed four common SSL approaches from the perspective of thelabel source. These four approaches are not mutual-exclusive. In fact, two of thethree tasks, which are studied in this thesis, leverage multiple SSL methods, asillustrated in Fig. 2.12.31LiDAR-Camera Calibration(Chapter 3)LiDAR-Camera Synchronization(Chapter 4)IMU Denoising(Chapter 5)Data AugmentationTaskApproach Pseudo Labels Sequential Self-labels Cross ModalityFigure 2.12: Overview of the relationship between three tasks studied in thisthesis and the supervised learning approaches.Self-supervised learning has been an active research area [83] and there willbe new approaches or perspectives. This section aims at reviewing the commonapproaches related to our work. Defining an efficient self-supervised learning taskis the key. In the following chapters, we demonstrate various innovative ways toformulate the problems at the robot sensor layer as SSL tasks.32Chapter 3RGGNet: Tolerance AwareLiDAR-Camera OnlineCalibration with Geometric DeepLearning and Generative ModelLiDAR-camera extrinsic calibration is a fundamental task at the robot sensor layer.Conventional methods heavily rely on the hardware or hand-crafted feature engi-neering. With the increasing popularity of deep learning (DL), a few recent efforthave demonstrated the advantages of DL on this task. These early attempts are alsothe pioneering work of applying self-supervised learning (SSL) at the robot sensorlayer. Unfortunately, their performance are not satisfactory yet and significantlyworse than that of the conventional methods. In this chapter, we first identify andaddress the performance gap between SSL and the conventional methods; and wethen propose the RGGNet, a new SSL approach, to bridge the gap. As our firstattempt of applying SSL to the robot sensor layer in this thesis, the proposed methodrepresents the calibration spatial relation as the output. The proposed RGGNethas been evaluated fairly with other competing methods on the popular KITTIbenchmark dataset, and it achieves promising results. In fact, the proposed RGGNetis the first SSL based method which achieves equivalent performance as in the33conventional methods at the robot sensor layer, while with the advantages of beingmore generalized.3.1 INTRODUCTIONCombination of 3D LiDARs and cameras is a common setup for modern autonomousvehicles/robots. The camera can provide short-range dense 2D color RGB images,whereas the 3D LiDAR can capture longer-range sparse data as point clouds. Thefusion of point clouds and RGB images is essential for fundamental tasks such as 3Dobject detection[141, 169], and semantic 3D mapping[66]. One critical prerequisitefor data fusion is the accurate extrinsic calibration of the transformation (i.e.,rotationand translation) between the LiDAR and camera in all 6 degrees of freedom (DOF).Extrinsic calibration problems can be grouped into offline and online cali-bration tasks. As the commercial 3D LiDAR became popular after 2005, theoffline calibration problem has been well studied. Classic offline calibrationmethods[40, 45, 51, 104, 113, 156, 158, 183, 187] can achieve satisfactory accuracywith controlled environments and various dedicated reference objects (i.e.,targets).However, in real-world applications, the calibration parameters may keep chang-ing due to various factors (e.g., semi-rigid arrangements of sensors, vibrationsin operation, and thermal strains). Therefore online calibration is equivalentlysignificant.Several effort have been made to tackle the online calibration challenge [12,24, 26, 67, 85, 114, 115, 135, 150, 151, 162]. However, those approaches generallyutilize hand-crafted features and can be vulnerable to the dynamic operating environ-ment. For example, features, such as lines[67] and planar regions[150], may not beeasily extracted in the uneven agriculture terrain scenario. Making such assumptionson environments may put both robots and people nearby under hazardous situations.Recently, a few deep learning-based approaches[64, 136] were proposed toovercome the obstacle. Those methods demonstrate the advantages of DL in termsof the efficiency of feature extraction and capability of generalization. As earlyattempts, they generally treat the deep neural networks (DNN) as a magic black boxregressor and may need a better problem formulation. The calibration performanceof those methods on public datasets are relatively not satisfying [64, 136], and34the trained backbone models require further cumbersome fine-tuning, which limittheir real-world deployment. We believe that problem formulation with missing orimproper geometry consideration is one reason behind it.On the other hand, Riemannian geometry has long been applied in the roboticsarea[98, 116, 145]. In the DL domain, the geometric deep learning (GDL), such asequipping DNNs with the Riemannian geometry [55, 61, 73, 97, 139], is still underthe early exploration. From the statistic machine learning perspective, the outputspace of this LiDAR-camera extrinsic calibration is on the 3D Special EuclideanGroup (i.e., SE(3)), which is well studied. Incorporating the SE(3) output spacenature and leveraging the Riemannian metrics into our problem formulation seemsa natural choice, although this is not explored yet with the DNN for the extrinsiccalibration task.In addition, the extrinsic calibration error is inevitable in practice. The tolerancefor each axis can be dynamic with respect to the environment and could not bearbitrarily defined. Is a 0.2◦ extrinsic calibration error in the yaw axis always worsethan the same amount of error in the roll axis? This question could be hard to answerand may not have explicit criteria. In the meanwhile, the calibration parametersare mostly used to fuse the point clouds and the RGB image data. It is fair toconsider the fused data quality as one metric to reflect the tolerance. To be morespecific, the re-calibrated results should yield similar fused data representation asto the ground-truth. To our knowledge, none of the previous methods addressedthis concern. Without considering the error tolerance, the estimated calibrationparameters may yield a small error numerically, but poor LiDAR-camera fused datarepresentation.To bridge the above research gap, we propose a tolerance aware LiDAR-cameraonline extrinsic calibration method. A deep neural network is adopted to learn amodel that estimates the extrinsic calibration parameters. Riemannian geometry isconsidered while formulating the problem. A deep generative model is also utilizedto regularize the training to model the error tolerance implicitly. Our method iscapable of estimating a wide range of miscalibration using a single frame RGBimage and point cloud data as the inputs. For a fair comparison, we evaluateour method on KITTI benchmark datasets[46], and compare it with the results ofstate-of-the-art methods.353.2 RELATED WORKSGenerally, methods of extrinsic calibrating LiDAR and camera are categorized intotarget-based and targetless approaches, depending on whether a dedicated artificialtarget is used. Target-based methods are limited to offline usage while targetlessapproaches can be used online.3.2.1 Target-based MethodsThe fundamental target-based work is by Zhang [183], which formulates the trans-formation estimation as a nonlinear least-squares iterative minimization problemwith geometric constraints. By using a planar checkerboard as a reference target,grid corners are detected by the camera to estimate the camera pose with respectto the checkerboard, which is further used to correlate the calibration parameterswith laser scanner distance readings. This 2D laser scanner based work is furthergeneralized to 3D LiDARs with similar problem formulations, including variants byUnnikrishnan and Hebert[156], Pandey et al.[113] and Geiger et al.[45]. The abovework assumes that the LiDAR is intrinsically calibrated, which in practice is notnecessarily true and addressed by Mirzaei et al.[104] via incorporating a LiDARsensor analytic model. Later target-based work is focusing on exploring differentfeatures for geometric constraints such as Zhou et al.[187] using line and plane cor-respondences and Verma et al.[158] with point and plan correspondences. Besidescheckerboards or fiducial marker[104], Gong et al. [51] and Florez et al.[40] alsoutilize customized targets instead. Regardless of the targets, it is common that alltarget-based approaches extract geometric features such as points, lines, corners,planes and distances from two sensors, using the known geometric measurements ofthe reference target to formulate an optimization problem. Limited by the artificialtarget, those methods can only be used offline. Scaramuzza et al.[135] propose amethod to get rid of the target by having specific user inputs instead, to correspondthe points in images and LiDAR scans. However, it is still limited to offline usage.3.2.2 Targetless MethodsIn order to extend the calibration to online applications, several targetless methodswere proposed. In the study by Levinson and Thrun [85], the edge feature is ex-36tracted from sensors to minimize an objective function that describes the summeddepth discontinuities. Pandey et al.[114] proposed a mutual information (MI) frame-work that correlates the laser reflectivity to the omnidirectional camera intensity andformulates an objective function with the joint entropy. Another similar but indepen-dent work is by Wang et al. [162]. Tamas and Kato [150] proposed a method usingvisible planar regions in the sensor data to correlate with the calibration parameters.Chien et al.[24] disclose that the mutual features such as MI metrics [114] or edgefeatures could fail the task due to occlusions and shadows. A visual odometry (VO)method is described in their work [24] to estimate ego-motion and find optimalextrinsic parameters together, the optimization of which, however, can get stuckin local minima. One classic motion-based method is by Taylor and Nieto [151],extending the hand-eye calibration method employed with the uncertainty as a three-stage motion-based extrinsic and timing offset estimation pipeline. Later variantsbased on this idea[151] include work by Corte et al.[26], Noel et al[191] and Park etal.[115], which, however, are all based on the availability of high quality odometryestimations. In practice, the odometry estimation, especially in the urban outdoorscenario, is another challenging task. Therefore, from the odometry assumptionperspective, [151] has high reference values to compare with our method. Regard-less of the frameworks, for all the above approaches, the features are hand-craftedand naive so that they can be directly used to formulate and be part of the objectivefunction. The main drawback of those methods is the difficulty in generalization.The hyper-parameters of those methods are carefully selected and tuned, and thosefeatures may fail the task in a different environment.3.2.3 Deep Learning MethodsIt is inspiring to note that the machine learning community has made a breakthroughwith automatic feature engineering via deep neural networks. To our knowledge,a few early attempts leverage the advantages of DNNs to tackle this problem.RegNet[136], the first DNN based method, uses Network in Network (NIN) [92]for feature extraction to directly regress the single frame sensor data to the extrinsicparameters. The application of the DNN automates the feature engineering anddemonstrates the efficiency of the data-driven manner. The major concern for37RegNet is that by using the Euclidean loss to measure the outputs, it overlooksthe output space SE(3) geometry nature. Arguably, one inherent consequence isthat the backbone forward networks struggle in a very short range. Iyer et al. [64]propose the CalibNet to address this issue by introducing a differentiable 3D spatialtransformer networks (3D STN)[53, 65] into the neural networks. By doing that, theestimated calibration offset can be used to re-project the point cloud to formulatea distance loss to the ground-truth, which indirectly considers the geometry. Thegeometric loss enhances the supervision of the training and achieves promisingresults in a larger range. However, their performance are still unsatisfying andworse than non-DL methods. Besides, to achieve reasonable results, their backbonemodels all require cumbersome fine-tuning process or iterative refinements. Onepotential cause could be that CalibNet utilizes the Euclidean loss to measure thedistance between the estimated depth map and the ground-truth. The Euclideandistance between the two depth maps does not reflect the geodesic distance betweenthe two corresponding transformations. Besides, due to the attention mechanism ofthe 3D STN, their pipeline also brings an extra scale factor noise. We believe anadequate problem formulation from the geometry perspective could improve theperformance of the DL-based methods.3.2.4 Spatial Transform NetworksSpatial Transform Networks (STN) is proposed by Jaderberg et al. [65]. With STN,we are able to train Convolutional Neural Networks (CNN) end-to-end with spatialtransformation injected. STN is defined as the combination of a localisation networkand sampling mechanism. Input feature map firstly is passed to localisation network,regressing to transformation parameters, which are then applied to a spatial grid.Lastly, a sampler will be used against the spatial grid to generate warped outputfeature maps. The main focus of STN is to empower CNNs on 2D-image tasksto actively learn transformation-invariance. Many recent work leveraging STNhas achieved effective performance on various tasks, such as [91] integrating STNwith Generative Adversarial Networks (GAN) to learn geometric corrections andgenerating warped images in a selective space, PointNet[119] using an extended andsimplified version for points feature alignment. Later on, Ankur et al.[53] introduced38GVNN, a library implementation for various geometric computer vision operations.In GVNN, a 3D-STN has been presented, which utilizes the same localisation-sampling framework however modularized into several layers. As described in thework of GVNN[53], the 3D-STN is modeling a SIM3 layer instead of SE(3) layer,which means the module will learn 6 DOF transformation parameters with an extrascalar factor, which will benefit many perception tasks as a self-attention pluginmodule for transformation-invariance. However, for 6-DOF extrinsic calibrationtasks, this 3D-STN is not ideal. On one hand, in practice, scalar factor is difficultto measure; on the other hand, the problem formulation will be different and morevague if the output space is SIM3, since with the supervised learning manner, thedependent scalar factor and rotational components are not easy to distinguish.3.3 PROBLEM FORMULATIONIn this chapter, we consider a mobile robot platform with one camera and oneLiDAR, subject to the following assumptions: i) the field of view (FOV) of twosensors have overlaps; ii) the coarse initial extrinsic calibration parameters Yinit andthe camera intrinsic calibration K are available.The camera data is in the RGB image format, represented by Xcam with a sizeof (H,W,3), where H and W means the height and width of the image respectively.The LiDAR sensor data is in the format of point cloud, Xpts with the size of(N,4), where N represents the number of points and 4 represents the Cartesiancoordinates (px, py, pz) and the intensity I of each point. For any frame, there existsa 4×4 extrinsic calibration matrix Y ∈ SE(3). Y consists of a 3 DOF translationalcomponent t with size 3×1 and a 3 DOF rotational component R with size 3×3,namely,[R t0 1]. When the robot is in operation, Y will keep changing with theunknown mis-calibration ∆Y corresponding to the known initial calibration Yinit :Y = ∆YYinit (3.1)Our task is to estimate ∆Y for each frame, given the sensor data (Xpts andXcam) and the camera calibration matrix K. It should be noted that ∆Y could berandom and independent for each frame and utilizing a sequence data stream will39not necessarily simplify this problem.The depth map is a powerful and common [64, 136, 165] feature representationwhich fuses the sensor data and calibration parameters all together. A depth mapXdm can be obtained with:zuv1= K[R t]pxpypz1 ,Xdm[int(u), int(v)] = z(3.2)where K is the 3×3 camera intrinsic matrix; (px, py, pz) is the Cartesian coordinateof each LiDAR point; [int(u), int(v)] is the corresponding depth map (Xdm) coor-dinate; z is the projected depth in the camera plane. It can be seen that within thesensors overlapped FOV, each point in Xdm corresponds to an RGB point of Xcamfor the same real-world physical point.Modeling this correlation is one key step in the extrinsic calibration task. Tra-ditional methods [114, 162] usually assume that this correspondence is an explicitfunction of the extrinsic calibration parameters. However, this strong assumptionwas proven to yield the generalization issue [24]. In our work, we instead adoptthis correlation to regularize the parameters of the estimation model so that theestimated offset ∆Yˆ should also yield a similar fused data representation. In thisway, implicit and dynamic tolerance is modeled. Therefore, we refer to it as theTolerance Regularizer (TR).To be more specific:∆Yˆ = f (Θ,Xdm,Xcam),s.t.,d(∆Yˆ,∆Y)< ε1,s(Xˆdm,Xdm,Xcam)< ε2(3.3)where f (,) is the extrinsic calibration model with unknown parameters Θ; ∆Yˆ isthe estimated calibration offset; d(,) represents the distance between the estimatedoffset and the ground-truth; s(,) represents the Tolerance Regularizer; Xˆdm is theestimated depth map; ε1 and ε2 mean the convergence error bounds. Xdm and Xˆdm40are obtained from Yinit and Yˆ, where Yˆ equals to ∆YˆYinit .With the given n examples of (Xdm,Xcam), this problem can be further formu-lated as a supervised learning task:Θ= argminΘ1nn∑i=1d(∆Yi,∆Yˆi)◦ s(Xˆdm,Xdm,Xcam) (3.4)where ◦ means the composition of two constraints.Clearly d(,) and s(,) are critical in our problem formulation. For the two mostrelated papers [64, 136], although not specifically stated, their underneath problemformulations can be considered as special cases of ours in Eqn. (3.4). The RegNet[136] defines d(,) as the Euclidean distance of the dual-quaternion representationbetween ∆Yˆ and ∆Y while treating s(,) as a constant. The CalibNet [64] defines d(,)as the distance between the estimated point clouds and the ground-truth while havings(,) as the Euclidean loss between the estimated depth maps and the ground-truth.Since ∆Y or Y is not in the Euclidean space, introducing the geodesic distancefor d(,) and incorporating the Riemannian geometry consideration is more desired.Similarly, for (Xdm, Xcam), utilizing the likelihood (of the estimated depth map inthe distribution of the ground-truth depth map, with the same camera image) todefine s(,) will be more adequate. The details of the two terms will be elaborated inSection 3.4.3.4 METHODSIn this section, we first give a brief description of the proposed method. We thenelaborate on the feature representation, the RGGNet architecture, the distance onSE(3) and the Tolerance Regularizer.3.4.1 OverviewThe above problem formulation includes several key components: 1) the onlinecalibration model f (,); 2) the distance between two SE(3) elements d(,); 3) s(,),the similarity between the estimated depth map and the ground-truth.We use a deep neural network to model f (,), as it is well-known that multi-layerfeed-forward networks are one kind of universal approximators [60]. A successive41Figure 3.1: The intensity depth map Xidm and the point map Xpm are projectedby the point clouds Xpts with the calibration parameters (K, Y ). Eachfeature map corresponds to an RGB image Xcam.feed-forward process transforms the coordinate system of the input data space intoan output space coordinate system, whose classes are linearly separable by hyper-planes [84]. From the Riemannian geometry perspective, recent work [55] impliesthat the learning process can be modeled as sequences of left/right Lie Group actions.Therefore, in our modeling, f (,) learns a sequence of transformations from the inputsensor data space to SE(3). It is also shown[55] that the residual neural network(ResNet) [56] can approximate the coordinate system transformation in a dynamicmanner, whereas ordinary networks, such as the multi-layer perception (MLP), can42only model the static system. In addition, ReLU, a common activation function,is not a proper coordinate transformation in terms of the bijection property, unlessbeing used in ResNets[55]. Therefore, considering the above principles, RGGNet isdesigned using ResNets for feature extraction and MLP with leaky ReLU or tanhfor regression. Due to the page limit and scope of the work, we could not elaboratemore and recommend interested readers refer to [55] for more details.Regarding d(,) and s(,), we specify a loss function for Eqn. (3.4) to trainRGGNet:Loss = d(∆Yi,∆Yˆi)+λ ·T R (3.5)where λ is a constant to bring the two components to the same scale and TR rep-resents the Tolerance Regularizer. In our work, we use a SE(3) geodesic distanceequipped with left-invariant Riemannian metrics to construct d(,). The ToleranceRegularizer TR is modeled by a Variational AutoEncoder (VAE)[77], a deep gener-ative model. The VAE is trained with the ground-truth (intensity) depth map andcamera data to maximizes the evidence lower bound (ELBO). During the back-propagation of RGGNet training, the parameters of this VAE are not updated. Withthe estimated (intensity) depth map and camera data as inputs, the ELBO output isused as the TR.3.4.2 Intensity Depth Map and Point MapUsing the Eqn. (3.2), a single channel depth map Xdm can be obtained. To takeadvantage of all information available from the LiDAR sensor, we further encodethe intensity I of the corresponding LiDAR point into the Xdm as well. To avoidconfusion, this two-channel feature map is named as the intensity depth map Xidm,as shown in Fig. 3.1. (Xidm, Xcam) will be the inputs of RGGNet.In order to compute the TR, a re-calibrated intensity depth map needs to becomputed, and a 3D spatial transformation is also required. To improve the pointlook-up efficiency, we also construct a feature representation called "point map".The point map is obtained by further encoding the Cartesian coordinates of thecorresponding LiDAR point (i.e. (px, py, pz)) into the intensity depth map. By doing43Figure 3.2: The architecture of the proposed RGGNet.this, we retain the rich information into a single encapsulated tensor, as shown inFig. 3.1.3.4.3 RGGNet ArchitectureAs shown in Fig. 3.2, first, a single camera image Xcam, and an intensity depthmap Xidm are fed into two ResNet-50 [56] respectively. The two 2048-D outputfeatures are concatenated into a 4096-D vector before being further fed into a three-layer MLP. During training, the ImageNet[27] pre-trained weights are loaded toinitialize the camera ResNet-50. The MLP includes three layers, each of whichis using leaky ReLU as the activation function and follows with a dropout layer.The last layer of MLP regresses to the rotational and translational vectors via twoseparate streams. The rotational stream is activated by the tanh function. Since thetranslational stream is not bounded, no activation is applied to the output. The twostreams outputs are concatenated finally into a 6D vector in se(3), representing as{r1,r2,r3, t1, t2, t3} ∈ R6. Lastly, the se(3) representation is exponential mapped toSE(3). Therefore, the backbone forward RGGNet takes in a single frame of RGBimage and intensity depth map, to output the estimated re-calibration offset ∆Yˆin the form of SE(3). We use yˆ and y to represent the se(3) form output and theground-truth to distinguish Yˆ and Y in the SE(3) form.443.4.4 Distance on SE(3)To accurately describe the distance between two elements in SE(3) (i.e. d(∆Yi,∆Yˆi)in Eqn. (3.5)), the geodesic distance is a natural choice. Geodesics on SE(3) canbe defined with a Riemannian metric g(p), an inner product on the tangent spaceof each point p on the manifoldM (i.e. SE(3)). Since SE(3) has no bi-invariantmetrics [98], we use the left translations [21]:<U,V >p=< D(Lp−1)pU,D(Lp−1)pV )>,s.t.∀p ∈M ,U,V ∈TpM(3.6)where D(Lp−1)p represents the differential of the left translation at the element p;TpM is the tangent space at p on the manifoldM .For a single point V on SE(3) with a Lie algebra parameterized vector v ∈ R6,we will further have a norm defined (omitting the identity subscript e) [21]:∥V∥2p =∥∥D(Lp−1)pV∥∥2 = vT ·JTp−1 ·g(e) ·Jp−1 ·v (3.7)where J represents Jacobian matrix. To be more specific, for the estimation ∆Yˆ andground-truth ∆Y, we define an inner product g(∆Yˆ) at the identity, ∆Y:d(∆Y,∆Yˆ) = ∆YˆT ·JT∆Yˆ−1 ·g(∆Y) ·J∆Yˆ−1 ·∆Yˆ (3.8)The notations above are aligned with [21], and details of the Riemanniangeometry concept can be found in the Chapter 1.2 of [21].3.4.5 Tolerance RegularizerThe architecture of the Tolerance Regularizer is shown in Fig. 3.3. During theback-propagation of RGGNet training, a re-calibrated point map, Xˆpm, is firstlycomputed by a 3D spatial transformer (i.e. the C-3DST in the chart, representing theconditional 3D spatial transformer). The C-3DST takes in the RGGNet prediction45Figure 3.3: The architecture of the Tolerance Regularizer. VAE encoders con-tain four Conv-ReLU-BN layers with depths as 64/128/64/32, followingwith a FC-ReLU-BN layer with 1024 units and one FC layer with 200units; decoders contain two FC-ReLU-BN FC layers with units as 200and (H×W )8 , three Deconv-ReLU-BN layers with depths as 32/64/128 andoutput Deconv-Sigmoid layer with 5 units. Kernel size of 4 and stridesize of 2 are set for all convolution and deconvolution layers.∆Yˆ, the camera matrix K, and the miscalibrated point map Xpm. Then the re-calibrated intensity depth map Xˆidm can be obtained by simply slicing the last twochannels of Xˆpm. Next, the Xˆidm is concatenated with the RGB camera data of theframe. This feature is first down-sampled to half the size and then fed into the VAEto compute the ELBO. Finally, the ELBO is appended to the loss as the TR. Twokey factors here are the weights in the VAE and the C-3DST, which are elaboratedbelow.In terms of the weights in VAE, the ground-truth intensity depth map and cameradata are used to train a VAE beforehand. Then during the training of RGGNet, thepre-trained VAE weights are loaded and frozen to calculate the ELBO.In order to obtain a re-calibrated point map, a differentiable 3D spatial trans-former is required. The commonly used 3D spatial transformer network (3D STN)[53, 65] is essentially a self-attention module and is not specifically designed forthis purpose. Directly applying the 3D STN as in the CalibNet[64] will introducean extra scalar factor. Therefore, we introduce a conditional 3D spatial transformer(C-3DST) module for this operation, as shown in Fig. 3.3. The idea is as follows:46In order to guarantee the 3D to 2D one-to-one mapping for being differentiable, there-projection is conditioned by the values of feature channels of the point map ateach pixel, instead of bi-linear sampling approaches[53, 64].To be more specific, each pixel with a coordinate (hi,wi) on the input point mapXpm can be transformed with (∆Rˆ ∆tˆ) of ∆Yˆ to obtain an updated point map Xˆpm:z′iu′iv′i1= K[∆Rˆ ∆tˆ]px,ipy,ipz,i1 ;(u′i,v′i) ∈ R2,Xpm[hi,wi] = (px,i, py,i, pz,i, Ii,zi);(hi,wi) ∈ Z2,Xˆpm[h′i,w′i] = (px,i, py,i, pz,i, Ii,z′i);(h′i,w′i) ∈ Z2(3.9)where (u′i,v′i) is the coordinates on the re-calibrated point map Xˆpm coordinatingsystem for the LiDAR point (px,i, py,i, pz,i); z′i is the updated camera plane depth. Inorder to construct an updated point map Xˆpm, each (u′i,v′i) ∈ R2 shall be mapped toan updated image 2D coordinate (h′i,w′i) ∈ Z2. However, due to the transformationhappens in 3D, multiple (u′i,v′i) can be mapped to the same (h′i,w′i), which is finefor the forward but would break the back-propagation. In order to keep this processdifferentiable, we condition the R2 → Z2 mapping with the LiDAR point intensity:(h′j,w′j) = argmax(u′j,v′j)I j (3.10)The updated intensity depth map Xˆidm can be derived by slicing the last twochannels of Xˆpm. Finally, we can compute the ELBO with the VAE for the TR:T R = ELBO(Xˆidm,Xcam |Θv) (3.11)where Θv represents the pre-trained VAE weights.473.5 EXPERIMENTS3.5.1 Data PreparationWe test on KITTI benchmark [46] recordings, which provide RGB images and pointclouds from Velodyne HDL-64E LiDAR, at various scenes and at different days.We follow the same strategy as in [64, 136] to prepare our training and testing data.For each frame, firstly, a transformation matrix, Yrdm, is generated for everydata pair, with uniformly sampling each translation axis from the range of ±ψ mand each rotation axis from the range of±φ ◦. Then the inverse of Yrdm is applied toground-truth calibration parameters Ykitti (provided by KITTI) to compute updatedinitial calibration parameters Yinit . By doing this, Yrdm becomes our label Ygt . Next,we use Yinit together with the raw point cloud and camera matrix K to constructthe initial mis-calibrated point map Xpm. Thus each training pair includes inputs of(Xpm, Xcam) and a label of Ygt , where Xcam is the RGB image.With the above procedure, a training dataset is created from all (except 0005and 0070) drives in 09/26/2011 sequence, including:• 24000 examples with the off-range as (±0.2m, ±15◦);• 4000 examples with the off-range as (±0.3m, 0◦);• 4000 examples with the off-range as (0 m, ± 20◦);For testing, we prepare 4 different testing datasets, each of which has 2000examples, with the following test name configurations:• T1 0005/0070 drives in 09/26/2011 sequence, with off-range as (±0.2m,±15◦);• T2a all drives in 09/26/2011 sequence (excluding 0005/0070), with off-rangeas (±0.2m, ±10◦);• T2b 0005/0070 drives in 09/26/2011 sequence, with off-range as (±0.2m,±10◦);• T3 0027 drive in 10/30/2011 sequence, with off-range as (±0.3m, ±2◦).48T2a and T2b are with the same drives and off-range as evaluated by CalibNet[64]while T3 is used to compare with the non-DL method[151]. In order to obtain thedenser representation, the point map and RGB image are pre-processed by applyingan average pooling with stride size 2 and 2×2 kernel.3.5.2 RGGNet TrainingAll work in this chapter is implemented in Tensorflow 1.14 [1] and trained in asingle NVIDIA GeForce 2080Ti GPU.For both the VAE pre-training and RGGNet training, the Adam optimizer[76] isapplied with the initial learning rate of 1e−4. Learning rates are decayed with thecosine decay schedule with restarts[99] using the decay rate as 0.96 and the decaystep as 2000. λ for the VAE regularizer is set as 1e−9. The batch size for VAE is setas 32 and RGGNet as 8, due to the GPU memory limit.The RGGNet is firstly trained with 30 epochs without the Tolerance Regularizer,which is marked as α-RGGNet. The VAE is trained with 30 epochs with ground-truth intensity depth map concatenated with the RGB image. For the formal RGGNettraining, the weights of the pre-trained α-RGGNet are loaded as the initializationparameters, and the VAE regularizer is applied. In addition, to demonstrate theadvantages of this data-driven manner, we also train an extra model, RGGNet+,which is fine-tuned from the fully trained RGGNet with extra 2000 frames trainingdata from the 10/30/2011 0027 drive (which is separately generated and differentfrom T3) with the off-range as (±0.3m, ±2◦).Since the RegNet[136] does not report numerical test results with the samesetting, we also trained a baseline model using the RGGNet neural network ar-chitecture but with the same dual quaternion Euclidean loss as in [136]. Exceptfor the loss, all training settings are the same. We consider this as our RegNetimplementation, marked as β -RegNet.In order to numerically verify the efficiency of C-3DST, we also trained aRGGNet by swapping the C-3DST with the exact same 3D-STN implementedby CalibNet. We mark this model as RGGNet-3dstn. In order to explore theperformance on sparser point clouds, we also trained a 16 lines LiDAR versionby down sampling 64 rings to 16 uniformly, for each neural network. The models49include α-RGGNet-16, RGGNet-16, RGGNet-16+, RGGNet-3dstn-16.3.5.3 MetricsPrevious work [64, 136, 151] mainly use the mean square errors (MSE) with Eulerangle representation as the metric. However, considering the geometry, MSE can bemisleading and it is more reasonable to use the se(3) error (SEE) instead:E = ||∆Yi−∆Yˆi|| (3.12)Based on the se3 error, with n testing examples, we derive two metrics toevaluate the models:• Mean se3 error (MSEE):MSEE =1nn∑iEi (3.13)• Mean re-calibration rate (MRR):MRR =1nn∑i|ηi−Eiηi| (3.14)where ηi represents the mis-calibration noise of the ith example. MRR reflectshow much noise is compensated by the re-calibration.3.6 EVALUATION AND DISCUSSIONThe quantitative evaluation results are shown in Table 3.1 and Table 3.2. It canbe seen that both RGGNet and α-RGGNet outperform the existing DNN basedmethods [64, 136] in terms of the MSEE. The heavily dropped MRR of β -RegNetfrom T1 to T2b suggests that, without considering the geometry, the model cannotbe generalized to different ranges. This conclusion aligns with what the originalRegNet paper [136] has reported. Compared with CalibNet[64], our backboneRGGNet also yields significantly smaller MSEE. As reported in [64], CalibNetwould fail when the translation off-range is beyond ±0.25m, even utilizing aniterative refinement methodology. In contrast, as shown in Table 3.2, our method50Table 3.1: PERFORMANCE COMPARISONS WITH DNN BASED METH-ODSMSEE (MRR) T1 T2a T2bCalibNet[64] - 0.022 (-) 0.022 (-)β -RegNet 0.048 (53.23%) 0.044 (37.08%) 0.046 (34.14%)RGGNet-3dstn-16 0.034 (67.85%) 0.031 (56.81%) 0.031 (57.25%)RGGNet-3dstn 0.027 (72.91%) 0.020 (70.22%) 0.022 (67.85%)α-RGGNet-16 0.023 (78.21%) 0.020 (72.27%) 0.020 (72.20%)α-RGGNet 0.027 (73.04%) 0.016 (73.74%) 0.018 (70.28%)RGGNet-16 0.024 (77.17%) 0.021 (70.86%) 0.022 (70.44%)RGGNet 0.021 (78.40%) 0.014 (75.61%) 0.017 (72.64%)Table 3.2: PERFORMANCE COMPARISONS WITH A NON-DNN BASEDMETHODMSEE (MRR) T3TAYLOR[151] 0.010 (-)β -RegNet 0.092 (-1.89%)RGGNet-3dstn / RGGNet-3dstn-16 0.025 (68.82%) / 0.0575 (37.39%)α-RGGNet / α-RGGNet-16 0.023 (65.00%) / 0.0362 (59.83%)RGGNet / RGGNet-16 0.022 (67.50%) / 0.039 (57.39%)RGGNet+ / RGGNet-16+ 0.010 (83.22%) / 0.012 (78.52%)is capable of not only estimating accurate offsets when the translation off-rangeis ±0.3m, but also maintaining a low MSEE even in a totally new environment.Besides, the performance of RGGNet-3dstn is slightly better than CalibNet butworse than RGGNet. This numerically verifies the efficiency of C-3DST over thenaive 3D-STN. Therefore, RGGNet outperforms both RegNet and CalibNet.With the same off-range settings, each test example in T2b is a new sceneto the model, whereas T2a contains scenes exposed in training. It can be seenthat for RGGNet, the MRRs/MSEEs from T2a and T2b are both close and high,which demonstrates good generalization capability. In addition, the RGGNet-1651also achieves good performance, which demonstrates the capability of generalizingto sparser point clouds.However, for more different scenes in T3 test, our models show a slight degra-dation in terms of MRRs. This is due to the tremendous scene differences between10/30/2011 and 09/26/2011. By including a small number of data from 10/30/2011into the training (the model is referred to as RGGNet+), the MSEE (MRR) isboosted from 0.022(67.50%) to 0.010(83.22%) respectively. This shows the ad-vantage of the data-driven methods. The motion-based method[151] is selectedfor comparison because it is the most recent work that provides KITTI benchmarkresults for the same task in such an end to end manner. Table 3.2 shows that theMSEEs of the motion-based model and RGGNet+ are equal to 0.010. It is worthemphasizing that, compared with [151], our method is more generalized, withoutrequiring hand-crafted feature engineering, expertise hyper-parameter tuning, theextra IMU sensor, or sequential data processing. In addition, the reported MRR isas high as 83.22%, which means that the de-calibration noise is well compensated.52Figure 3.4: Examples of RGGNet predictions. Reference objects for calibration are shown in bounding boxes.53Figure 3.5: One bad prediction example with the same input from T2b test. a: initial de-calibrated input; b: theground-truth; c: the prediction from α-RGGNet, yielding a 0.0550 se3 error; d: the prediction from RGGNet,yielding a 0.060 se3 error. Images are cropped for demonstration.54Fig. 3.4 shows two examples of RGGNet predictions. It can be seen that thedepth map and camera data of the reference objects within the bounding boxes alignaccurately.Figure 3.6: Histogram of SEE for RGGNet on T1, T2a, T2b, T3 tests. The histogramis shown in green and 50-th percentile is shown in blue.Fig. 3.5 demonstrates a bad prediction example from RGGNet and α-RGGNetthat both have a relatively large se3 error. More specifically, α-RGGNet yieldsa 0.0550 se3 error, which is very close but slightly smaller than that of RGGNet(0.060 se3 error). However, from the sub-figures c and d, we can see RGGNetresults are more acceptable and similar to the calibrated projection since the pointclouds do fall on the vehicle with slightly roll axis off. In contrast, the projectionby α-RGGNet is visually off, resulting in a worse-quality fused depth map. This55is a typical example: Without considering the error tolerance, a small error mayyield a much worse depth map. This example implies that, by introducing theTolerance Regularizer, the RGGNet learns to yield an output that is more similarto the ground-truth depth map, within an error bound. Besides, the slightly lowerMSEE and higher MRR in all tests also demonstrate the advantages of the proposedTR. Fig. 3.6 shows the histogram of SEE for RGGNet on various tests.Lastly, the inference speed can achieve 41 fps on a single NVIDIA GeForce2080Ti GPU using naive TensorFlow 1.14 frozen protobuf serving, with batch sizeset as 1. This suggests that the model could be deployed to run in a real-timemanner.3.7 CONCLUSIONIn this chapter, we present the RGGNet pipeline for 3D LiDAR-camera extrinsiccalibration. We formulate the calibration task as a geodesic distance optimizationproblem that is regularized by the likelihood bound learned from a generative model.Our experiments demonstrate the efficiency and superiority of the proposed modeland the Tolerance Regularizer. We believe this current work could inspire manyfuture research in this area.56Chapter 4LiCaS3: A SimpleLiDAR-Camera Self-SupervisedSynchronization MethodThe LiDAR-camera temporal synchronization task is a natural extension fromthe LiDAR-camera spatial calibration task in Chapter 3. However, the temporalsynchronization task deals with the sequential data, which can be more complicated.It should also be noted that the RGGNet estimates the calibration difference betweenan artificial de-calibrated system and the given system. This is because the factorycalibration is usually available and this task definition can still be valuable. However,this setting does not apply to the LiDAR-camera synchronization task. It motivatesus to explore a more advanced and powerful SSL approach.It is worth mentioning that the conventional methods were dominant and therehas been no research on applying SSL or even DL on such a synchronization taskwhen we published the related work. In this chapter, we first address the limitationsof the conventional methods; further, we propose the LiCaS3, the first DL andSSL based LiDAR-camera synchronization framework. LiCaS3 consists of twoneural networks, which represents the temporal relation as the output and the modelrespectively. In another word, our proposed method examines the hybrid sensorrelation representation. LiCaS3 has been intensively tested on two datasets and ithas been integrated with an off-the-shelf odometry estimation method. The final57reported performance exceed the conventional one and demonstrate significantimprovements in the end-to-end test. This work fills the research gap of applyingDL / SSL on LiDAR-camera temporal synchronization and serves as a pioneeringwork for future research.4.1 INTRODUCTIONLiDAR and camera are two widely used sensor types in modern autonomous roboticssystems. By fusing the long-range 3D point clouds from LiDAR and the short-rangedense camera images together, many 3D perception tasks have achieved superiorperformance than the single modality approaches. However, such a promisingsensor fusion requires accurate LiDAR-camera spatial calibration and temporal datasynchronization. The LiDAR-camera calibration problem has been an active andrelatively well studied topic during the past years [26, 64, 136, 151, 177]. However,less research have been done for LiDAR-camera temporal synchronization andcertain challenges have not been addressed yet.As illustrated in Fig. 4.1, the conventional LiDAR-camera synchronizationpipeline generally consists two parts [129] : clock synchronization and temporalsynchronization. Clock synchronization is the process of aligning the clocks of allsensors to the same reference clock [31, 46, 144], and thus it usually requires certainhardware setups. Temporal synchronization is the process of estimating the constanttime offsets between sensor measurements [26, 151]. While such a pipeline canprovide accurate synchronization, there are several potential limitations:1. The clock synchronization requires certain hardware setup, which is expensiveand may not be possible in some cases [20, 125]. The influences of clockdrift may be minor on certain platforms, such as drones of small size[108].However, for many robots, such as the ones with long cables, the clock driftconsequence could be significant. For systems that do not support hardwaresynchronization, the time offsets between two sensor measurements are notnecessary constant [125].2. Currently dominant temporal synchronization methods are motion-based[26, 42, 43, 151], all requiring the precise inertial measurement unit (IMU)58Black-box ProcessClock Sync Temporal SyncData SyncConventional PipelineProposed PipelineTimestamp as Feature Motion as FeatureDeep FeatureCamera RawLiDAR RawIMU RawProcessed DataSynchronized DataSynchronized DataProcessed DataReference TimestampFigure 4.1: (Top) The conventional LiDAR-camera synchronization pipeline:the timestamps associated with raw observations are first aligned to thereference timestamp (clock synchronization); the processed data andthe reference timestamp are then used to extract co-observed motionfeatures to estimate a constant offset between two sensors. (Bottom) Theproposed pipeline: it directly uses deep neural networks to extract deepfeatures from processed data and estimate the dynamic offset for eachframe.sensor measurements, which can be obtained from high-end IMU sensor orthrough advanced data processing. High-end IMU sensors can be expensiveand unavailable on low-cost platforms. IMU data processing could be sophis-ticated since there are many noise and artifact factors (such as bias, thermalstrain, and so on) and would introduce latency.3. For offline data recording users, who do not own the system, they may nothave a way to synchronize the LiDAR and camera data themselves afterwards[19, 46]. The offline data tasks could cover a wide range of applications, suchas 3D object detection [112], scene flow estimation [132], mapping [34] andso on.With the increasing popularity of LiDAR-camera systems in various applica-tions, the above concerns need to be addressed for the potential wide adoption oflow-cost platforms. As demonstrated in Fig. 4.11 and Table 4.6, the LiDAR-Camera59100ms upper bound mis-synchronization can lead to a significant pose error, whencompared to the baseline of LiDAR-monocular visual odometry (LIMO) [52]. Forresearchers without access to robotics systems, unsynchronized datasets may bedifficult to process or even useless, which is especially a concern in the machinelearning community.As illustrated in Fig. 4.1, regardless of pipelines, the common objective is toobtain synchronized LiDAR and camera data sequences. The nature of the LiDAR-camera synchronization problem is to identify which RGB image frame from thecamera corresponds to a given frame of point clouds from the LiDAR or the otherway around. In recent years, deep neural networks (DNNs) have been successfullyapplied to LiDAR-camera systems. Encouraged by such effort, it is possible that wecould directly work on the matching for each frame of sensor data with an end-to-end and data-driven manner to avoid the cumbersome and expensive conventionalcascaded steps. Especially, recent effort of applying deep learning (DL) on LiDAR-camera spatial calibration [64, 136, 177] shed lights on employing DNNs to tackleour synchronization problem. However, to our knowledge, leveraging deep learningon LiDAR-camera synchronization has not been explored yet.If sufficient annotations were provided, such a matching problem would bea well-explored supervised learning task, which is not much different from taskssuch as the MNIST classification [82]. Unfortunately, in reality, acquiring accuratepoint clouds and RGB image sequential pairing annotations can be both difficultand expensive. This probably can partially explain why there has no publishedstudy on using DNNs for the LiDAR-camera synchronization task yet. There-fore, due to the lack of annotations in practice, in terms of machine learning, oursynchronization problem has to be formulated as an self-supervised learning oncross-modal retrieval [68] task. It is worth mentioning that self-supervised learn-ing on cross-modal retrieval [68] is still a very challenging problem in machinelearning. Even though there are related effort in computer vision, such as audio-visual synchronization [25, 118] and image captioning [50], which aim to solvesimilar underlying learning problems, all those methods cannot be directly appliedto our LiDAR-camera synchronization problem. This is because that the underlyingcross-modal correspondences and constraints between vision and other modalitiesare all regarding semantics instead of geometry. For example, in the overlapping60area of a tightly-coupled LiDAR-camera sensor rig, a group of 3D points from theLiDAR will geometrically correspond to deterministic pixels on the RGB image,while the correlations between audio, text and image data are more vague and lessdeterministic: different people may pronounce the same word with different lipmovements. Typical assumptions in those work, such as the latent space of eachmodality can be embedded in an Euclidean space [25], are inappropriate for ourproblem of interest. In addition, the dominant work of cross-modal self-supervisedlearning on LiDAR-camera systems nowadays mainly focus on downstream appli-cations, such as depth completion and monocular depth estimation [49, 69, 189]instead of matching or retrieval, except for a few recent work on spatial calibration[64, 136, 177]. This motivates us, as the first attempt, to explore self-supervisedlearning to tackle the LiDAR-camera synchronization problem.To bridge the above research gap, we propose an innovative LiDAR-cameraself-supervised synchronization pipeline, named as the LiCaS3, as illustrated in Fig.4.1 and Fig. 4.4. Instead of relying on clock synchronization with using the IMU,LiCaS3 directly estimates the offsets of the measurements, without assuming thatthe offset is a constant. The only loose constraint we assume is that the upper boundε of the offsets is known, which in practice is easy to estimate. Since LiDAR sensorsusually yield equal [46] or lower [125] sampling frequency but larger data volumethan the cameras do, in our work, we focus on finding the corresponding RGBimage for each given frame of point clouds. As stated in [144], mixing the spatialcalibration and temporal offset estimation together will make the problem morecomplicated. Therefore, in our work, we focus on the temporal synchronizationproblem, assuming that spatial calibration is solved separately. It is worth notingthat our proposed pipeline is compatible with several recent proposed DNN-basedspatial calibration frameworks [64, 136, 177]. LiCaS3 can also be deployed alongwith the conventional synchronization pipeline.The workflow of the proposed self-supervised learning pipeline, as illustratedin Fig. 4.4, is as follows: First, the point clouds are represented as depth mapwith spatial calibrations. Then each RGB image candidate within the latency upperbound of the targeting point clouds (or depth map) is used to predict a depth map,with the targeting depth map as the self-label, via a weights-sharing neural network.The inverse of the losses (between the predictions and the targeting depth map) are61used as the self-supervision signal to jointly train another classifier to predict thetime offset. By doing so, without external annotations during the training, we canobtain an offset estimator which can be deployed both online and offline.The reminder of this chapter is organized as follows. In Section 4.2, we discussrelated work on LiDAR-camera synchronization, cross-modal self-supervised learn-ing and monocular depth estimation. Our methods are described in Section 4.3. InSection 4.4, we present our experimental results on KITTI [46] and Newer Collegedataset [125]. We also generate a small amount of synchronization annotations ontop of the Newer College datasets, which will be publicly available together withour source codes. Finally, we conclude this chapter in Section 4.5.4.2 RELATED WORK4.2.1 LiDAR-camera SynchronizationModern autonomous driving vehicles, such as KITTI [46] and NuScene [19] plat-forms, are reported to employ the hardware synchronization by triggering thecamera exposure with every LiDAR 360◦ spin. For low-cost platforms such as mo-bile robots [20] or LiDAR-camera rigs [125], the hardware trigger setups are usuallynot available and alternative clock synchronization approaches include passive clocksynchronization [111], PTP [31] and NTP [103]. However, the clock synchroniza-tion processes are complex and difficult to model; and the uncertainty of the clockrelated methods makes such processes comparatively black-box [129]. A series ofearly work to address such challenges were done by Furgale et al. [42, 43] andRehder et al. [129], proposing an offline target-based method (temporal synchro-nization) to jointly estimate spatial calibration and temporal offset between multiplesensors. The time offset is estimated within a rigorous theoretical framework ofthe maximum likelihood estimation. Kalibr [43, 128], a toolbox that is developedbased on the above work has been widely used [125] in both industry and academiatoday. An alternative framework TD-ICP was introduced by Kelly and Sukhatme[74], formulating the synchronization as a registration problem. While the abovetwo frameworks demonstrate accurate offline synchronization performance, they allrequire a checkerboard, which limits the online usage.62Recently, plenty of methods have been proposed to tackle the LiDAR-cameraextrinsic calibration jointly with temporal offset estimation for online usage. Taylorand Nieto [151] proposed a motion-based method under the hand-eye calibration[148] framework. One issue of this and previous methods in practice is that theycannot handle long-term operation, which is addressed by Corte et al. [26] with asolution to heuristically discard outlier portions. Sommer et al. [144] argue that thespatial-temporal joint optimization makes the overall problem more complicatedand fragile [144]. To address this issue, they proposed a simple LED device as thealternative for the high-cost hardware synchronization setup [144]. More recently,Qiu et al. [122] introduce an IMU-centric scheme to explore the temporal-rotationalcalibration problem for heterogeneous sensor sets. This method is reported toyield close and accurate performance as Kalibr [43, 128] on time offset estimation.However, these methods require precise IMU measurements in the loop to estimatea constant time offset, whereas our method can handle dynamic time offsets withoutIMU. In addition, previous methods rely on hand-crafted feature engineering toextract motion features, whereas our methods leverage the advantages of automaticfeature extraction of deep learning.4.2.2 Self-supervised Cross-modal RetrievalSelf-supervised learning on cross-modal retrieval has been a hot topic in computervision. Cross-modal retrieval is defined as taking one type of data as the query toretrieve relevant data of another type [161]. The cross-modal retrieval approachescan be grouped into two: real-valued representation learning and cross-modalhashing. Regardless of the methods, the idea is similar that they all transformvarious modalities (e.g., image, text, audio) into one or more common spaces [161]through different methods. Below we focus on recent deep self-supervised learningbased work. One pioneering work is done by Ngiam et al. [107] to use a deepautoencoder [59] to learn a sharing representation space between speech audio andthe corresponding videos of the lips. Srivastava and Salakhutdinov [146] arguethat the statistical properties of each modality are very different, which shouldbe learned with individual models. It is shown that training a dedicated DeepBoltsmann Machine (DBM) [134] for each modality and then fusing them into a63joint representation can achieve better performance on the same task as well asthe image-text retrieval task. Inspired by those two, a few approaches [39, 172]extend the classic Canonical Correlation Analysis (CCA) [126] into Deep CanonicalCorrelation Analysis (DCCA), which have been examined on acoustic-articulatoryand image-text retrieval tasks. Later variants focus on improving the cross-modalsupervisions [25, 39, 118], adding more tasks [171] and leveraging traditionalmachine learning methods [10]. Since the above real-valued representation learningmethods in run-time may be too slow for applications on large scale data. Anincreasing number of methods [121] on cross-modal hashing have been proposedthat various modalities data are transformed into a common Hamming space forfaster search speed. Nevertheless, since the correlations among vision, text, audioand language are just semantics, the above methods do not work for our problem.The multi-way classification learning strategy proposed by Chung et al. [25]is the most similar work to our proposed self-supervised learning scheme. We alluse the inverse distances between each instance of one modality and the targetingmodal instance as the self-supervision label to train another classifier. However, thismethod will not work for our problem since it assumes the two modalities can beembedded into an Euclidean space. Instead, our proposed scheme parameterizesthe distance between two modalities with DNNs, via a monocular depth estimationtask.4.2.3 Monocular Depth EstimationIn the important task of monocular depth estimation, with a given monocular cameraRGB image, each pixel is mapped to a depth value. Eigen et al. [32] are the firstto adopt deep neural networks on depth estimation from a single image. Later on,several effort improve the performance with the combination of the ConditionalRandom Fields (CRF) [86, 93]. All those early DNN adoptions achieve significantimprovements over traditional hand-crafted feature engineering approaches [72].With the introduction of ResNet [56] and UNet [133], the residual or skip-connectionbased neural network structure has becoming a common backbone for such task[44]. It should be noted that the monocular depth estimation is a notoriously ill-posed problem [32] since the color-depth mapping can be ambiguous without any64. . .. . .. . .. . ..................Figure 4.2: Our synchronization task focuses on the processed LiDAR dataXp and camera data Xc. For each targeting frame of point clouds xpq ,within a search window of length l, the task is to identify the relativeframe number yq, the corresponding RGB image of which shares thesame or closest observations as xpq . The search window l starts fromq− ls and ends at q+ le.constraints. Since 2016, numerous methods strengthen the constraints from variousperspectives. Garg et al. [44] explore geometry constraints by leveraging the stereo-pair camera as stronger supervision. Godard [48] found out the weighted structuralsimilarity index measure (SSIM) and L1 loss [185], which is a loss designed basedon human perception, achieves superior performance with stereo-pair consistencyenforcement. Such loss has gained popularity in almost all later work. Inspired bythe spatial transformer networks (STN) [65], which utilizes a differentiable bi-linearup-sampling to empower neural networks to conduct affine transformation withgradients, Zhou [189] regresses two neighbour camera images into the SE(3) spaceto jointly learn the motion of the camera as an implicit constraint. More recent workfocus on joining other tasks such as optical flow [175], edge detection [174], andsemantic segmentation [23]; adding direct constraints such as surface normal [120]and visual odometry [173]; leveraging extra information such as 3D sensor [63] andsequential frames [163]. Even though the performance of the depth estimation is notthe focus of our work, all previous work shed lights on our method that leveragesthe depth estimation for a different purpose.654.3 METHODS4.3.1 Problem FormulationLet us consider the system with a single LiDAR and a single camera. We assumethat the extrinsic calibration T and the camera matrix K are available, and the twosensors have overlapping area. As mentioned in Section 4.1, our work focus onsynchronizing the camera data to the LiDAR (the slower sensor) data. We assumethat the upper bound of the time offsets ε is provided. Pose information is notneeded in our method.As shown in Fig. 4.2, let {xp1 , ...,xpn} be a sequence of LiDAR data and{xc1, ...,xcm} be a sequence of camera data. Since the two sequences are not nec-essarily synchronized, the q-th (q ∈ {1, ...,n}) frame of point clouds and of RGBimage do not necessarily correspond to the same real-world observation. Let l bea search window with ls past frames and le future frames: l = ls + le + 1. Here lsatisfies ε ·Fc ≤ l, where Fc is the frequency of the camera sensor. The LiDAR-camera synchronization task is essentially to identify the relative frame numberyq ∈ {0, ..., l−1}, the corresponding RGB image xcyq+1−ls of which shares the sameor closest real-world observations as xpq . Let h(,) be a parameterized synchronizationmodel. Mathematically, such a problem can be represented as follows:yˆq = h(xpq ,xcq−ls , ...,xcq, ...,xcq+le |ω) (4.1)where yˆq is the predicted relative frame index in the RGB image search window forxpq and ω represents the parameters of model h(,).4.3.2 Comparisons with Motion-Based MethodsIn the meanwhile, as illustrated in Fig. 4.1, the problem formulation of motion-basedLiDAR-camera temporal synchronization methods [26, 151] can be represented asfollows:∆tc = ψ(Xpk ,Xck,Xik, tr) (4.2)66where Xpk ,Xck,Xik are processed and clock synchronized sequences of point clouds,RGB images and IMU data respectively; tr is the reference timestamp; ψ(,) repre-sents the temporal synchronization method and ∆tc is the estimated constant timeoffset between LiDAR and camera.Besides the dependencies on IMU data, reference timestamp and constant timeoffset assumption, the main difference of such problem formulation in Eqn. (4.2)is that ψ(,) is not a parametric model. Instead, ψ(,) relies on hand-crafted featureextraction such as Harris corner detection [151] and iterative closest point (ICP) toassociate point clouds between frames. Such an approach may lead to installing onsub-optimal local minima, expensive complexity and lack of generalizability. Onthe other hand, by parameterizing the model h(,) with ω , our problem formulationnot only can get rid of numerous dependencies, but also can simplify the inferenceand handle dynamic time offsets for individual frames.While our formulation Eqn. (4.1) is more generic, it is also challenging to learnh(,) in a realistic way. In the following subsections, we present different learningschemes to approach this problem and finally how to model h(,) in an innovativeself-supervised learning manner.4.3.3 Supervised Learningh(,) in Eqn. (4.1) can be modeled in a supervised multi-class classification learningframework, as long as the ground truth is available:ωˆ = argminωn∑i=1Lmc(yˆi,ygti ) (4.3)where ygti represents the ground truth supervision signal for yˆi, ωˆ denotes theestimated parameters of h(,) and Lmc(,) represents the multi-class classificationobjective function, such as the Euclidean loss or cross-entropy loss.However, in practice, the ground truth supervision signals are difficult and expen-sive to obtain. This is especially true, considering the high frequency of the sensors,sparsity of point clouds and the similarity of nearby frames. Therefore, practically,our problem cannot be solved solely with the supervised learning framework.674.3.4 Self-supervised Cross-modal RetrievalAn alternative approach to the direct supervised learning method is the self-supervisedcross-modal retrieval:yˆq = argmini{D(xci ,xpq)}, i ∈ {q− ls, ...,q+1e} (4.4)where D(,) means the distance between each RGB image in the search window andthe targeting point clouds.However explicitly defining D(,) is challenging, since the RGB image xr ∈RH×W×3 and the LiDAR point clouds xp ∈RN×3 are essentially different modalities.In the meanwhile, the recent progress of deep learning based monocular depthestimation is inspiring, which 1) uses the projected depth map as an efficientrepresentation of the point clouds, and 2) learns the mapping from RGB imagespace to the depth map space. The monocular depth estimation model g(,) and itsloss function d(,) together can be a good candidate to model D(,) implicitly.To be more specific, let xdm be the depth map representation of point cloudsxp with the same height and width (H,W ) as the RGB image. Each pixel on thedepth map and RGB image of the same coordinate should correspond to the samereal-world object, within the overlapping field-of-view (FOV). Such fact will onlystay true for synchronized point clouds and RGB image pair, the mechanism ofwhich can be used in return to distinguish which pair is the synchronized one.Depth Map ProjectionFirst of all, the depth map can be obtained via projecting point clouds as follows:zuv1= K[R t]pxpypz1s.t. u≤W,v≤ H(4.5)68where (u,v) is the depth map pixel coordinate of the LiDAR point (px, py, pz) in thecamera overlapping area; R and t represent the rotation and translation componentsof the extrinsics T respectively; K is the camera matrix; z is the depth value that isencoded on (u,v).Monocular Depth EstimationConsidering the same data in Fig. 4.2, the depth estimation model can be trained asthe follows:θˆ = argminθm∑q=1q+le∑i=q−lsd(g(θ ,xci ),xdmq ) (4.6)where θˆ represents the trained parameters of model g(,).Running the inference of this model on each RGB image in the search windowand the one yielding the minimal error is the synchronized RGB image. Namely,the Eqn. 4.4 can be further specified as:yˆgq = argmini{d(g(θˆ ,xci ),xdmq )}, i ∈ {q− ls, ...,q+1e} (4.7)where yˆgq is the predicted relative frame number in the RGB image searchsequence with g(,) model.UncertaintyLet us consider the targeting depth map xdmq and two RGB image frames in thesearch window, with xcs as the synchronized frame and xcu as one unsynchronized.Fus denotes the warping from xcs to xcu.From Eqn. (4.6), it can be seen that both d(g(θ ,xcs),xdmq ) and d(g(θ ,Fus xcs)),xdmq )(i.e., d(g(θ ,xcu),xdmq )) are being minimized. Since there could be dynamic objectsand different scenes, Fus contains high uncertainty. If we view d(g(θ ,xcs),xdmq ) as aspecial case when each element in the Fus tensor equals to 1 (i.e., no transformation),the forward could be viewed as two cascading steps: 1) apply a transformation Fisto any input xci to warp it to xcs ; 2) feed xcs into g(,). Sharing the spirit of the spatialself-attention [65], the step one can also be modelled in a parameterized way f (,):69g(θ ,xcs) = g(θ ,Fus xci )) = g(θ , f (θ′,xci ))) (4.8)where θ ′ represents the parameters of f .With this equivalent modelling, each frame of the RGB image can be viewed asthe "normalized" element in the nearby domain. If there are enough samples fromthis domain for individual RGB image, the dynamic transformation Fis can be fittedwell with deep neural networks.70Figure 4.3: An illustration that the synchronized RGB image and the depth map yield the minimal loss. The left imageshows a continuous 6-frame RGB image sequence, from top to bottom, with the targeting depth map overlay. Theright chart shows the losses decrease first and then increase, whose minimal matches the ground truth synchronizedframe index. This illustration is from the inference results of LiCaS3-g model with l = 6 and s = 2.71However, in our case, such equivalent single layer cannot be well trained due tothat only a few (l) nearby frames in the search window are being observed by thecamera and used for training. As a consequence, f (,) brings in noise for each inputexcept for the synchronized frame. Such noise or uncertainty will get aggregatedthrough g(,) and yield a bad prediction results. In other words, the synchronizedRGB image will yield the best depth map prediction and minimal loss, as illustratedin Fig. 4.3. This makes it promising to leverage the outputs of a sequence of depthestimation to conduct the synchronization task in an self-supervised way. With thisproblem formulation, no external annotations are required.ChallengesHowever such a self-supervised learning inference model could yield poor general-ization performance, since the monocular depth estimation is an ill-posed problem[32]. As reviewed in 4.2.3, in order to achieve good generalization, various con-straints are introduced to the monocular depth estimation model, which wouldconflict with the synchronization task.More specifically, in our work, the depth estimation model is only used forthe mapping between the raw RGB and depth map pairs. Each RGB image is notnecessarily trained with the actual ground truth depth map, as described in Eqn.(4.6). Therefore, we are not optimizing the accuracy of the depth estimation, whileinstead leveraging the depth estimation model with the help of the inference modelin Eqn. (4.7) for classification. Over-optimizing the performance of the depthestimator could lead to poor generalization of the classifier.4.3.5 Self-supervised Learning with Pseudo LabelsTo compensate the weaknesses of the above direct supervised learning and self-supervised cross-modal retrieval methods, we propose an innovative approach bycombining them together: using the output of self-supervised model g(,) as thenoisy self-labels to train a classifier h(,), jointly. By doing so, g(,) only needs toperform well on the training data and h(,) is able to get trained with the pseudolabels. As we can see, such a process of h(,) is also a self-supervised training, asg(,), without external labels.72To be more specific, with Eqns. (4.6) and (4.7), the model h(,) can be trainedin the following way:ωˆ = argminωn∑i=1Lmc(yˆi,ygi ) (4.9)where ygi is the prediction results with the self-supervised model g(,).73......... ... ... ...M CSingle Frame Camera DataSingle Frame Camera DataPredicted Depth MapPredicted Depth MapTargeting Depth MapPrediction ErrorPrediction ErrorTargeting Depth MapAll Frames Camera Data in Search WindowMask ConcatCross-Entropy LossFigure 4.4: Overview of the proposed LiCaS3 framework. The upper part of the figure demonstrates the self-supervisedlearning part: all RGB image candidates are fed into the a sharing weight UNet [133] g(,) and supervised by thesame targeting depth map xdmq . The losses are further used as the self-label for the bottom supervised learning part.The details are elaborated in Section 4.3.6.744.3.6 Deep Neural NetworksThe self-supervised learning scheme introduced above can be implemented withvarious deep neural networks and loss functions. Below, we demonstrate oneimplementation that we have experimented in this article.Referring to recent monocular depth estimation methods [49, 69, 189], a nativeUNet [133] is employed to model g(,). The input is the RGB image and the outputis the depth map. Except that each kernel depth is reduced to half of the originaldesign for the sake of model size, everything else stays the same as the originalUNet implementation [133].For the the LiDAR data {xp1 , ...,xpn} and camera data {xc1, ...,xcm}, each trainingexample is prepared as (xcq−ls , ...,xcq, ...,xcq+le ;xdmq ), q∈ {ls+1, ...,m− le}. As shownin Fig. 4.4 top, each camera frame xci , i ∈ {q− ls, ...,q+1e} is fed into the weights-sharing UNet g(,) to yield a predicted depth map xˆdmi7→q, which is used to calculatethe loss with the targeting depth map xdmq .As suggested by several work [16, 17, 131], the structural similarity indexmeasure (SSIM) not only demonstrates good performance in comparing two depthmaps but also can derive geodesics as a metric. Inspired by this, we choose theweighted SSIM and L1 loss [185] as the d(,) in Eqn. (4.7), which has also beenused in the most recent depth estimation work [48, 49]:d(xˆdmi 7→q,xdmq ) =α2(1−SSIM(xˆdmi 7→q,xdmq ))+(1−α)||xˆdmi7→q− xdmq ||1(4.10)where α is fixed as 0.85 as suggested in the original paper [185]. The predictedsynchronized RGB image frame relative index yˆgq can be obtained from Eqn. (4.7).As demonstrated in Fig. 4.4 bottom, for h(,), the training example is(xcq−ls , ...,xcq, ...,xcq+le ,xdmq , yˆgq) with yˆgq as the prediction results from g(,) for xdmq .Firstly, each xci is pixel-wise multiplied by xdmq so that the pixels out of the LiDARFOV are zeroed-out (masked); then the masked l RGB images and xdmq are concate-nated along the channel axis to construct a (H,W, l) tensor. Then such a tensor isforwarded to a ResNet-50 [56] and output a 2048-D vector. This vector is furtherflattened and passed through three dense layers, as shown in Table 4.1, to output75Table 4.1: LAST THREE LAYERS OF h(,)number of units activation function1 512 Leaky Relu2 128 Leaky Relu3 l Sigmoidthe predicted index yˆhq. The cross entropy loss Lce is employed on one-hot encodedprediction and the supervision signal. Since the specific choice of the neural networkarchitecture is not the focus of our work, for h(,), we employ the popular ResNet-50backbone.Technically g(,) should be separately trained before h(,) training. In practice,we found that jointly training g(,) and h(,) does not reduce the accuracy, whichmakes the training end-to-end. Therefore, our training is a single stage with thefollowing loss:L = d+β ·Lce (4.11)where β is a constant to weight the two losses so that they are in the same scale. Inpractice, we found the training is not sensitive to this value and β = 0.01 has beenadopted across all experiments.4.4 EXPERIMENTS AND RESULTSIn this section, we first describe the experimental datasets, training details and eval-uation metrics from Section 4.4.1 to 4.4.3. Next from Section 4.4.4 to 4.4.9, intenseevaluations are presented, including comparisons with hardware synchronization,comparisons with software synchronization, evaluations in an offline synchroniza-tion setting, comparisons with supervised methods, end-to-end LiDAR monocularcamera odometry integration test and evaluations in the lower beams LiDAR system.At last from Section 4.4.10 to 4.4.12, we show several ablation studies, inferencespeed profiling and discuss the limitations of our methods.76Table 4.2: KITTI AND NEWER COLLEGE DATASET SPECSItem KITTI [46] Newer College [125]LiDAR Type Velodyne HDL-64E Ouster OS1-64LiDAR Frequency 10 Hz 10 HzLiDAR Range 120 m 120 mCamera Type Point Grey Flea 2 Intel RealSense D435iCamera Frequency 10 Hz 30 HzSynchronization Hardware Trigger PTP [31] + Kalibr [43]4.4.1 DatasetWe evaluate the proposed method exhaustively on two real-world dataset: KITTI[46] and Newer College [125], as described in Table 4.2, which include bothhardware synchronization and network/software synchronization.In KITTI [46], the Velodyne LiDAR and cameras are hardware synchronized sothat each spin of the LiDAR triggers the cameras exposure, sampling at the same 10Hz automatically. In this dataset, the RGB images and point clouds are one-to-onepaired. In our experiments, we utilize the point clouds and left camera data fromall 09_26 sequences for training, except preserving randomly selected 5 drives(0005, 0015, 0023, 0036 and 0093) as the testing data. The ratio of testing overtraining data is around 15%. All point clouds are projected into depth maps withthe calibration parameters. To construct the unsynchronized examples, a slidingwindow of size l (i.e., ls+ le+1 in Fig. 4.2) is employed to the synchronized KITTIsequences ({(xc1,xdm1 ), ...,(xcn,xdmn )} to generate pairs with dynamic offsets. Fig.4.5 illustrates one example of how the training data is generated and how such astrategy can create equivalent real-world latency scenario. Our approach is similarto the one in [167]. In our work, we evaluate different values of l ∈ {6,8,11} withstride s = 1. To further validate our method on low frequency camera scenario, wealso prepare dataset with sliding windows whose camera sequences are sampledwith a stride 2. Such setting is equivalent to the case that the camera runs at 5Hz while the LiDAR runs at 10 Hz. The sliding window or the maximum offset lis selected based on the assumption that the latency upper bounds are larger thanε ∈ {0.5,0.7,1.0,2.0,1.4,4.0} seconds for each setting, which we believe covers awide range. In terms of data processing, the depth maps are max pooled with kernel77Sliding WindowCamera SequenceTargeting Depth Map. . . . . .. . .Figure 4.5: An example demonstration of the sliding window strategy tocreate unsychronized LiDAR and camera data, with dynamic offsets.{xc1, ...,xc21} represent 11 adjacent RGB images in sequence and xdm11represents the targeting depth map. A sliding window of size l = 6is employed to sample adjacent RGB images with a stride size s = 2,yielding 6 training examples, each of which contains 6 RGB images forthe targeting depth map. The green colored squares correspond to thedata that share the same or closest real-world observations. However,such correspondence (i.e., the ground truth ygt) is not used during thetraining and only leveraged in the testing.size as 8 and stride as 4 for denser representation [177]. The LiDAR sensor range120m is utilized to clip and normalize the depth map in the range of [0,1]. Thenboth the RGB images and depth maps are central cropped and resized to (32,256)for efficient GPU kernel usage. Considering the large data size, the training data isuniformly downsampled to only keep half. In total, six LiCaS3 models have beentrained on KITTI dataset to compare with the hardware synchronization groundtruth.The Newer College [125] dataset were collected using a handheld rig withan Ouster OS1-64 LiDAR and an Intel RealSense D435i global shutter camera.Without hardware synchronizations, the Newer College team employed a widelyused toolbox Kalibr [43] to estimate the time offsets between the point cloudsand RGB images, with the help of two IMUs embedded in both sensors. In ourexperiments, we use the short experiment run, with the first 90% frames for trainingand the rest 10% for testing. Similarly, the depth map is max pooled, clipped and78normalized using same settings. Then both the RGB images and depth maps arecentral cropped and resized to (64,128). The training data generation strategy is thesame as in KITTI, with the only difference that the sliding window sampling strideis 5 and 10. This is because the Kalibr synchronization results yield noticeablemis-synchronisation when the stride is set to 1 (i.e., iterating every point cloudsframe) at the frame granularity. This is also one motivation to apply deep learningon such a task. During our data inspection, we found when the stride is set to 5and 10, the synchronization results can still serve as reasonable references. Sinceour training is self-supervised and does not require annotations, this does not forma blocker for training but makes it tricky to evaluate the performance. In order tostill compare our methods with the widely used Kalibr against the common groundtruth, we also manually annotated 10% of the Newer College testing data for thecase of sliding window as 6 and stride as 1. Fig. 4.6 demonstrates one example ofthe annotated dataset. The annotating strategy is as follows:1. Each testing example is annotated by 2 independent labelers.2. Each testing example can be annotated with more than one offset or "unsure".3. The final testing ground truth is constructed by those labels that all labelersyield the same annotation. For examples that are marked with "unsure" ordisagreed are discarded.It should be noted that during the annotation, we found that, due to the highsampling frequency and slow movements of the sensor rig, the adjacent two frameslook very close and the labelling is challenging. As a consequence, only 36.79%labels are agreed and the ratio of unsure agreed examples over total agreed examplesis 5.26%. This justifies our previous claim that a self-supervised method is necessarysince the annotations are very tedious and challenging. When the sampling strideis set to 5 and 10, we still leverage the Kalibr results as an relative comparison.In such a setting, the latency upper bounds for Newer College are consideredas ε ∈ {0.83,1.16,1.66,2.33,3.33} seconds. The training data is also uniformlydownsampled by half. In total, six LiCaS3 models have been trained on NewerCollege dataset to compare with the psudo ground truth and one model has beentrained to compare with the manual labelled ground truth.79Figure 4.6: An example sequence of Newer College [125] testing data thatwe manually annotated, with l = 6 and s = 1 sampling. Each image isoverlayed with the same targeting depth map, where the number on thetop left is the sequence number. The bottom middle camera image (i.e.,image sequence 4) is the paired one to the targeting point clouds from thebaseline method [43]. However, by taking a close look at the highlightedarea, it can be seen that the last camera image matches the depth mapthe best, which is also what all our labelers agree.4.4.2 Training DetailsOur methods are implemented in Tensorflow-GPU 1.15 and trained on a singleNVIDIA Geforce 2080ti GPU. During training, the batch size is fixed to 32 and theADAM optimizer [76] is used with a fixed learning rate of 10−4 for 15 epochs andthen 10−5 for another 5 epochs, which is exactly the same as the settings of relateddepth estimation work [69]. The only data augmentation we applied is randomhorizontal flip during training.4.4.3 Evaluation MetricsClassification AccuracySince the formulated problem is essentially a classification task, we utilize thewidely used classification accuracy score [80] to measure the performance:A =NtN(4.12)80where Nt is the number of true predictions (i.e., when yˆ = ygt in Eqn. (4.9)) and Nis the number of test examples.Top-2 AccuracyTop-N accuracy is another widely used metric to evaluate the performance of aclassifier [80].In our case, when the prediction index is only one frame away, such as ygt = 2and yˆ = 1, the result should be more preferred than the case of more frames away,such as ygt = 2, yˆ = 6. Therefore, we define the top-2 accuracy At2 as the follows:At2 =Nt +Nd1N(4.13)where Nd1 is the number of predictions that are one frame off, i.e., |yˆ− ygt |= 1.Smoothed AccuracyIf we apply a weak assumption that the offset stays static in adjacent b frames, nowour prediction can be obtained from a naive majority voting method. Here we definethe smoothed-b accuracy Asmb as follows:Asmb =NsmtNsm(4.14)where Nsmt is the number of true predictions (from majority voting) and Nsm is thenumber of test sequence examples. The single frame classification accuracy score Acan be viewed as a special case of Asmb with b = 1. We use Asmt2,b to denote the top-2accuracy for Asmb .81Table 4.3: CLASSIFICATION PERFORMANCES OF SELF-SUPERVISED LiCaS3 AND SUPERVISED LEARNINGONLINE SYNCHRONIZATION# Model Datasets Ground Truth Sampling A (At2) Asm10 (Asmt2,10) Asm20 (Asmt2,20)1 LiCaS3-h KITTI[46] Hardware Synchronization l=6,s=1 73.62% (92.90%) 86.93% (97.94%) 92.65%(99.07%)2 LiCaS3-h KITTI[46] Hardware Synchronization l=8,s=1 68.39%(90.83%) 86.13%(97.98%) 92.31%(99.56%)3 LiCaS3-h KITTI[46] Hardware Synchronization l=11,s=1 62.91%(90.58%) 78.15%(97.09%) 85.68%(98.66%))4 LiCaS3-h KITTI[46] Hardware Synchronization l=6,s=2 78.48%(91.53%) 95.39%(98.27%) 98.81% (99.64%)5 LiCaS3-h KITTI[46] Hardware Synchronization l=8,s=2 69.60%(87.33%) 90.11%(96.28%) 98.16%(99.26%)6 LiCaS3-h KITTI[46] Hardware Synchronization l=11,s=2 63.76%(86.03%) 83.40%(96.20%) 91.14%(98.66%)7 SL-h[133] KITTI[46] Hardware Synchronization l=6,s=1 89.45%(97.37%) 97.80%(99.62%) 98.63%(99.95%)8 SL-h[133] KITTI[46] Hardware Synchronization l=8,s=1 86.95%(96.04%) 96.97%(99.62%) 97.93%(99.85%)9 SL-h[133] KITTI[46] Hardware Synchronization l=11,s=1 89.94%(96.88%) 97.33%(99.47%) 99.00%(100%)10 SL-h[133] KITTI[46] Hardware Synchronization l=6,s=2 94.22%(98.01%) 99.21%(100%) 99.99%(100%)11 SL-h[133] KITTI[46] Hardware Synchronization l=8,s=2 92.64%(96.60%) 98.85%(99.95%) 99.97%(100%)12 SL-h[133] KITTI[46] Hardware Synchronization l=11,s=2 92.27%(96.16%) 99.18%(99.54%) 99.95%(100%)13 LiCaS3-h Newer College[125] PTP[31] + Kalibr[43] l=6,s=5 74.13%(91.12%) 96.84%(99.44%) 100%(100%)14 LiCaS3-h Newer College[125] PTP[31] + Kalibr[43] l=8,s=5 65.77%(86.10%) 94.05%(97.17%) 99.57%(99.93%)15 LiCaS3-h Newer College[125] PTP[31] + Kalibr[43] l=11,s=5 55.68%(79.17%) 86.84%(94.10%) 95.87%(98.70%)16 LiCaS3-h Newer College[125] PTP[31] + Kalibr[43] l=6,s=10 76.23%(92.84%) 97.34%(99.50%) 100%(100%)17 LiCaS3-h Newer College[125] PTP[31] + Kalibr[43] l=8,s=10 66.25%(87.46%) 94.47%(99.19%) 99.78%(100%)18 LiCaS3-h Newer College[125] PTP[31] + Kalibr[43] l=11,s=10 53.54%(83.11%) 80.81%(96.94%) 90.72%(98.47%)19 SL-h[133] Newer College[125] PTP[31] + Kalibr[43] l=6,s=5 83.57%(93.08%) 99.86%(99.93%) 100%(100%)20 SL-h[133] Newer College[125] PTP[31] + Kalibr[43] l=8,s=5 82.36%(93.79%) 99.58%(99.93%) 100%(100%)21 SL-h[133] Newer College[125] PTP[31] + Kalibr[43] l=11,s=5 82.84%(92.64%) 99.43%(100%) 100%(100%)22 SL-h[133] Newer College[125] PTP[31] + Kalibr[43] l=6,s=10 90.75%(96.49%) 99.93%(99.93%) 100%(100%)23 SL-h[133] Newer College[125] PTP[31] + Kalibr[43] l=8,s=10 89.82%(96.71%) 100%(100%) 100%(100%)24 SL-h[133] Newer College[125] PTP[31] + Kalibr[43] l=11,s=10 86.00%(95.49%) 99.85%(100%) 100%(100%)82Table 4.4: CLASSIFICATION PERFORMANCES OF SELF-SUPERVISEDLiCaS3 AND SOFTWARE SYNCHRONIZATION WITH NEWER COL-LEGE MANUAL ANNOTATIONS# Model Sampling A At21 PTP[31] + Kalibr[43] l=6,s=1 36.33% 49.50%2 LiCaS3-h l=6,s=1 70.63% 90.10%Table 4.5: KITTI: CLASSIFICATION PERFORMANCES OF SELF-SUPERVISED LiCaS3 OFFLINE SYNCHRONIZATION (ON TRAIN-ING SEQUENCES)# Model Sampling A (At2) Asm10 (Asmt2,10) Asm20 (Asmt2,20)1 LiCaS3-g l=6,s=1 75.05% (98.19% ) 91.29%(99.25%) 96.38%(99.66%)2 LiCaS3-g l=8,s=1 66.07% ( 96.58% ) 84.25%(99.07%) 92.82%(99.64%)3 LiCaS3-g l=11,s=1 56.81% ( 93.10%) 73.66%(98.19%) 83.96%(99.18%)4 LiCaS3-g l=6,s=2 78.10% ( 97.93%) 94.51%(99.54%) 98.16%(99.86%)5 LiCaS3-g l=8,s=2 67.76% (95.98% ) 88.66% (99.26%) 95.22%(99.74%)6 LiCaS3-g l=11,s=2 53.41% ( 89.00% ) 73.27%(96.50%) 84.09%(98.82%)7 LiCaS3-h l=6,s=1 79.03% (97.43% ) 92.65%(98.81%) 95.50%(99.04%)8 LiCaS3-h l=8,s=1 73.99% (97.14%) 90.48%(98.85%) 95.86%(99.23%)9 LiCaS3-h l=11,s=1 65.78% ( 91.98%) 80.17%(95.98%) 85.73%(97.22%)10 LiCaS3-h l=6,s=2 83.90% (98.00% ) 96.92%(99.25%) 98.69%(99.54%)11 LiCaS3-h l=8,s=2 74.55% (96.57%) 92.57%(99.36%) 97.37%(99.58%)12 LiCaS3-h l=11,s=2 70.87% ( 93.85% ) 87.06%(98.22%) 92.90%(99.10%)83102345 (Current)Pred (5)GT (5)102345 (Current)Pred (2)GT (2)102345 (Current)Pred (0)GT (4)(a) (b) (c)Figure 4.7: KITTI: Demonstration of example prediction results of the LiCaS3-h model (l=6 and s=2). For each example,0 represents the oldest frame, i.e., the frame 1000ms old while 5 represents the current frame, and so on. The RGBimages of 0 to 5 in each example are the inputs to the LiCaS3-h neural networks and the output is a single number.The dashed green bounding box shows the matched patterns on the depth map and RGB image overlay. The lastrow of each column is the ground truth and the second last row is the predicted frame.84102345 (Current)Pred (3)GT (5)102345 (Current)Pred (4)GT (3)102345 (Current)Pred (4)GT (3)(a) (b) (c)Figure 4.8: KITTI: Demonstration of degenerate prediction results of the LiCaS3-h model (l=6 and s=2). The dashedgreen bounding box on ground truth frame shows the obvious visual features that should have been matched. Thered ones on the prediction frame are what are actually matched. Sub-figure (a) represents a stationary scenariowhile (b) and (c) are from a moving vehicle.85Figure 4.9: Online synchronization test runs of two KITTI drives.8610 2 3 4 5 (Current) Pred (3) Kalibr (2)10 2 3 4 5 (Current) Kalibr (2)Pred (3)10 2 3 4 5 (Current) Kalibr (3)Pred (4)Figure 4.10: Newer College: Demonstration of example prediction results of the LiCaS3-h model (l=6 and s=1). Foreach example, 0 represents the oldest frame, i.e., the frame 1000ms old while 5 represents the current frame, andso on. The RGB images of 0 to 5 in each example are the inputs to the LiCaS3-h neural networks and the outputis a single number. The dashed bounding box shows a visual recognizable differences on the depth map and RGBimage overlay. The red and green colors are used to distinguish the matched and mismatched frames. The lastcolumn of each row is the Kalibr baseline [43] and the second last column is the predicted frame. It can be seenthat the Kalibr yield poor results that can be visually told while LiCaS3 predicts the accurate synchronized frame.874.4.4 Comparisons with Hardware SynchronizationFor the KITTI dataset, the hardware synchronization results are used as the vali-dation ground truth. The rows 1-6 in Table 4.3 demonstrate the performance ofLiCaS3-h model with various settings. The "-h" is used to distinguish the "-g" modelwhich we will also evaluate in later subsection. The SL-h represents the model thatis trained in supervised learning, which will be discussed in later sections as well.It can be seen that the single frame accuracy is very promising, compared withwhat hardware synchronization achieves. All models achieve smoothed-20 top-2accuracy values higher than 98%, which means less than 2% of predictions are faraway from 2 frames, under a weak assumption of 20 frames offsets unchanged.From the table, we can see that the larger the s and smaller the l, the higheraccuracy the model can achieve. While the impacts of the sampling settings on theclassification accuracy is large, the top-2 accuracies are all very high with smallvariations. This explains why for all models the smoothed-20 top-2 accuracy valuesare high. Particularly, for KITTI, when l = 6,s = 2, the performance are the best,achieving single frame classification accuracy 78.48% and top-2 accuracy 91.53%in a totally self-supervised manner. The smoothed-20 top-2 accuracy is as high as99.64%. Fig. 4.7 demonstrates three examples of predictions with l = 6 and s = 2.Qualitatively, Fig. 4.7 shows two good (i.e., (a) and (b)) and one bad (i.e., (c))prediction examples. For (a) and (b), we can see that the LiCaS3-h model predictsthe correct synchronized frame while making a right turn (in (a)) or running at highspeed (in (b)). We can also see that the positions of surrounding vehicles couldbe very different even within one frame mis-synchronization. This could lead toconfusions for downstream perception detectors without synchronization.Fig. 4.8 and Fig. 4.7.(c) demonstrate degenerate estimations. We observe twofailure reasons: 1) The failure reason of Fig. 4.8.(a) and Fig. 4.7.(c) is that theenvironment keeps static and the vehicle is not moving or moving very slowly.Since the input RGB images are almost identical, we argue that in such scenarios,the synchronization is of less interest and the error within the search window sizeframe is tolerable. 2) Fig. 4.8.(b) and Fig. 4.8.(c) show a different failure reason.For Fig. 4.8.(b), it can be seen that the edge of the building should have beenmatched in the ground truth frame (as shown in the green bounding box) whereas88the predicted results match the edge of the window between the camera and depthmap. Similarly, the prediction in Fig. 4.8.(c) mis-matches the pattern on differentcars. The root cause of such a failure could be the max-pooling pre-processing thatis applied on the depth map right after the point clouds projection. Even though themax-pooling is proven very useful in various deep learning based LiDAR-Cameraextrinsic calibration methods [64, 177], when two strong features are spatially close,such an operation may introduce errors. We suggest that augmenting the trainingwith different max-pooling kernel sizes could be one cost effective way to improvein future work.In the above benchmark, no sequential constraints has been added and for eachpoint clouds frame, the paired RGB image could be any past frame in the searchwindow. However, in practice, the search window could be further constrained bythe prediction results of the last frame. For example, if the last frame synchronizedresults show latency of 3 frames, the current frame search window could startfrom the last 3 frames instead of last 6 frames. With such setting, two simulationshave been done and the results are shown in Fig. 4.9. It can be seen that for bothdrives, the single frame A and A2 achieves high score while the offset for eachframe stays dynamic. Especially for drive 0005, only one frame (28th) is wrong butwithin one frame away, yielding 92.31% accuracy and 100% top-2 accuracy. Thisdemonstrates that our methods can achieve comparable performance even withoutweak assumptions of offsets being constant in 20 frames. The videos are providedin the supplementary material for more intuitive demonstration.4.4.5 Comparisons with Software SynchronizationWhen ground truth is unavailable, it is widely used to take Kalibr’s results as areference [122, 125]. The Newer College [125] dataset also utilize the Kalibr[43, 128] with PTP [31] assistance for the baseline synchronization results.The row 13-18 in Table 4.3 demonstrate the relative performance of LiCaS3-hmodel with various settings, comparing with the baseline. Both the impacts ofsampling settings and relative performance align with ones from KITTI very well.This demonstrates the generalization of our methods.As we can see from the table, when l = 11,s= 5, the single frame top-1 accuracy89is less than 60%, even though the smoothed accuracy is still very high. We arguethat this could result from that the ground truth that we try to compare with isonly psudo ground truth, provided by Kalibr [43, 128] with PTP [31]. To furthercompare LiCas3 with Kalibr [43, 128] + PTP [31], a benchmark has been run withl = 6 and s = 1 on our manual annotated dataset mentioned in subsection 4.4.1.The results are shown in Table 4.4. Qualitatively, it can be seen that our methodsyield significant better performance than Kalibr [43, 128] + PTP [31]. Fig. 4.10demonstrates three examples of predictions of such benchmark as well. It can beseen from all three images that the baseline methods yield noticeable mis-alignmentbetween the projected depth map and the RGB image. In the meanwhile our LiCaS3yields the best aligned pair. The above results show the superiority of our proposedmethods over the widely used Kalibr [43, 128] even with the help of PTP [31].4.4.6 Evaluation of Offline SynchronizationSince LiCas3 does not require annotations, this method can also be used to syn-chronize a long sequence for offline synchronization usage. To evaluate that, thesix LiCaS3 models with various sampling settings have been tested on the KITTItraining data. As demonstrated in Table 4.5, both LiCaS3-g and LiCaS3-h modelsyield accurate synchronization results. The impact of l and s is consistent as theonline results shown in Table 4.3. One interesting discovery is that h(,) models ingeneral all slightly outperform g(,) model even though h(,) is supervised by theoutputs of model g(,). In other words, the testing errors of h(,) are even smallerthan the pseudo label errors, demonstrating the superior robustness of deep neuralnetworks.4.4.7 Comparisons with Supervised MethodsTo better understand the performance gap between self-supervised and supervisedlearning, another twelve supervised learning models (denoted as SL-h) have beentrained, as illustrated in rows 7-12 and 19-24 in Table 4.3. For KITTI and NewerCollege, the ground truth or psudo ground truth has been used as the supervisionsignal to train the model h(,), with other settings exact the same. By comparingthe models with the same dataset and same sampling settings, we can see that the90supervised learning models yield noticeable better performance.For KITTI, when l = 6,s = 2, the single frame classification accuracy achieves94.22% and the smoothed-20 accuracy achieves 99.99%. This implies that deeplearning methods are capable of achieving competing performance as the hardwaresynchronization methods, without any hardware setup. The performance gap be-tween supervised and self-supervised learning could potentially be further reducedvia more sophisticated neural network architecture and fine-tuning in future.Even though, the major drawback of expensive annotations of the supervisedmethod persists. Considering the convenience of not requiring extra annotations andthe comparable smoothed accuracy, the self-supervised LiCaS3 is a more efficientmethod.4.4.8 Evaluation with LIMO91Figure 4.11: Illustrations of LIMO benchmarks. Left: 0036 sequence; right: 0093 sequence. Each sequence has threeruns with a zoomed-in screenshot shown in the dashed bounding box.92Table 4.6: APE RMSE OF BASELINE, MIS-SYNCHRONIZED RUN ANDLICAS3 RUN FOR SELECTED KITTI TEST SEQUENCESTesting Sequence 0093 0036Mis-Synchronized Run 1.474 m 4.336 mLiCaS3 Run 1.226 m 2.959 mTable 4.7: PERFORMANCES OF LiCaS3 ON 32 BEAMS LIDARModel A (At2) Asm10 (Asmt2,10) Asm20 (Asmt2,20)64 Lines 73.62% (92.90%) 86.93% (97.94%) 92.65% (99.07%)32 Lines 60.21% (86.45%) 76.43% (95.51%) 84.24% (97.90%)In order to demonstrate the how LiCas3 can benefit real-world robotics appli-cations, we have further conducted evaluations to integrate with one recent opensourced LiDAR-monocular visual odometry (LIMO) [52] method.The two longest sequences with turns from our KITTI testing datasets, whichLiCaS3 is not used for training, are selected to run the experiments. For eachsequence, we use the default settings of LIMO as the baseline. Since KITTI ishardware synchronized, in order to evaluate the impact of mis-synchronization, wesynthesize a latency with a upper bound of 100 ms. With such, a mis-synchronizedrun can be obtained for each sequence. At last, we apply LiCaS3-h model tothe mis-synchronized data for a compensated run. In terms of metric, we use thestandard root means square error (RMSE) of average pose error (APE) for numericalbenchmarks.Qualitatively, as shown in Fig. 4.11, the mis-synchronization leads to noticeableoff along the trajectory while the LiCaS3 compensated trajectory is very close tothe baseline for both sequences. Quantitatively, as illustrated in Table 4.6, in bothsequences, the LiCaS3 demonstrates significant improvements on the average poseerrors over the mis-synchronized cases.934.4.9 Lower Beams LiDARsIn order to evaluate the generalization of LiCas3 on lower beams LiDAR, we furthertrain one LiCaS3 model with LiDAR downsampled KITTI datasets. We use theexact same LiDAR beam down sample implementation as the RGGNet[177]1. TheLiCaS3-h 32-line model is trained with the same settings as in the 64-line one, withthe window and stride size fixed as l = 6,s = 1. As shown in Table 4.7, we can seethat the performance of the 32-line case degrade, which is expected. However, thesmoothed-20 accuracy is still quite promising and demonstrates that the proposedLiCaS3 can generalize to low-beam LiDAR. It is also worth mentioning that theperformance for 32-line or lower beams could be further improved via tuninghyper-parameters.4.4.10 Ablation StudiesBesides the above evaluations, the impacts of choice of loss d(,), joint trainingstrategy and whether the learning scheme of Chung et al. [25] can be directlyapplied to our task, have been examined.As a result, one model has been trained under the setting of l = 6,s= 1 in KITTI,with the L2 loss. During our experiments, we found that the training diverges aftera few iterations even with a very small learning rate. This could be caused by thefact that the depth map space is not Euclidean [16, 17, 131], i.e., the inner productscannot be used as a mathematically strict metric. Therefore, even though L2 satisfiesidentity of indiscernibles and symmetry axioms in this space, the triangle inequalityis not satisfied. When the training starts from a black image, the L2 loss could leadthe training into a very bad local minima, therefore diverging the training.To study the effects of the joint training, another model is trained under thesetting of l = 6,s = 1 in KITTI with the two stage strategy: g(,) is first trained for10 epochs and then the weights are loaded and frozen to train h(,) for another 6epochs. The learning rate change stays the same as in 4.4.2. The accuracy scoresare almost identical. This can result from that the training of h(,) is robust to noisylabels in the first few epochs of the joint training. The fact that the LiCaS-h modelsoutperform the LiCaS3-g models in the offline synchronization evaluation provides1https://github.com/KleinYuan/RGGNet94one proof of such hypothesis. Therefore, for having efficient experiments, thisobservation motivates us to conduct joint training for all models reported in thischapter.At last, in order to validate our previous claim in subsection 4.2.2 that thelearning scheme of Chung et al. [25] does not work for our task, another modelhas been trained under the setting of l = 6,s = 1 in KITTI, with two ResNet-50as the encoder to embed the two modalities into Euclidean space. We view thisas our implementation of Chung et al. [25]. With everything else stays the same,the model diverges quickly and we are not able to train a converged model withvarious starting learning rates. We believe this may result from the inappropriateassumption that the depth map and RGB image can be embedded into the Euclideanspace without direct supervision. Regardless, leveraging a depth estimation task is amore direct way to parameterize the distance between those two modalities.4.4.11 Inference Speed ProfilingOn KITTI, for the LiCaS3-h model with l = 6 and the input size as (32,256), theinference time spent on a NVIDIA GeForce 2080ti for a single frame is 19 ms. OnNewer College, for the LiCaS3-h model with l = 6 and input size as (64,128), theinference time on the same platform setting is 25 ms. It can be seen that the modelalone can run at significantly higher frequency than the LiDAR sensor and thus canbe deployed online. In practice, the inference speed can be further increased, withmore hardware optimizations such as TensorRT2.4.4.12 LimitationsEven though our methods achieve good performance on various benchmarks, thereare several limitations:• our approach does data level synchronization instead of a continuous realvalue offset estimation. There could be applications that require a more strictand fine-grained time-offset;• our methods accept a fixed number of frames as the input instead of a dynamic2https://developer.nvidia.com/tensorrt95one. Multiple neural networks need to be trained for different search windowsize configurations;• in our work, we focus on single frame synchronization while in practiceadjacent frames might be correlated and such correlation information can befurther explored. As shown in our experiments, naively smoothing severalsequential predictions can achieve better results.However, we believe the above limitations do not prevent from applying our methodsto many applications, such as the demonstrated LIMO task. In many deep learning-based perception, prediction, planning, or odometry estimation tasks, the inputs areall at the data level. We argue that such a constraint is not of large concern. Besides,the second and third limitations can be addressed by replacing the CNN with a moreadvanced recursive neural network (RNN) or transformer structure. We will exploresuch ideas in future work.4.4.13 Failed AttemptsAs we can see, the proposed LiCaS3 is essentially representing the temporal syn-chronization as both the output and the model, i.e., a hybrid representation. In fact,we have indeed explored the straightforward output representation, which utilizesthe exact same framework as the RGGNet [177]. However, the results are notsatisfactory and the model can hardly converge. We believe one potential cause isthat the synchronization task is more complex than the extrinsic calibration task,since sequences of data have been used.4.5 CONCLUSIONIn this chapter, we first address several potential limitations of the conventionalLiDAR-camera synchronization pipeline; we then present a simple self-supervisedpipeline, named LiCaS3. The synchronization task is formulated as a classificationproblem. The deep learning based framework is implemented, where the trainingcan be achieved in an end-to-end manner without manual annotations, while theinference is also simple and real-time.The proposed methods have been intensely evaluated on two benchmark datasets96to show the generalization of LiCaS3. Our experiments demonstrate the efficiencyand superiority of the proposed model over the commonly used Kalibr method [43]and comparable performance over the hardware synchronization methods. Whenthere is no high-end IMU available, no hardware triggering setup or no access tothe robotics system, our proposed approach provides an alternative solution forlow-cost platforms and for researchers who only have access to recorded LiDARand camera datasets for the synchronization task. We believe this work could inspiremany future research in the related area.97Chapter 5IMUDB: Self-Supervised IMUDenoising For Inertial AidedNavigationIMU plays an important role in inertial aided navigation on robots. However, rawIMU data could be noisy, especially for low-cost IMUs, and thus requires effi-cient pre-processing or denoising before applying further navigation algorithms.Conventional IMU denoising approaches are mostly hand-crafted and may faceconcerns such as sensor modelling errors and generalization issues. Several recentwork leverage deep neural networks to tackle this problem and achieve promisingresults. However, the currently reported deep learning methods are mainly basedon supervised learning, and thus require sufficient and accurate annotations. Whilein real-world applications, such annotations can be expensive or unavailable, mak-ing these methods not practical. To address the above research gap, we proposeincorporating self-supervised learning and future-aware inference into IMU denois-ing. The end-to-end navigation evaluation results on EuRoC and TUM-VI datasetsare promising. Open-VINS [47] has been adopted as the baseline method, whichhas equipped with Extended Kalman Filter (EKF) to compensate some noise [87].We have demonstrated that our methods can work together with EKF and yieldedbetter performance. In the previous chapters, spatial and temporal relations havebeen explored between the LiDAR and camera sensors, and we reported promising98results. The sensor relations have been represented as the output and the hybrid wayrespectively. In this chapter, the proposed SSL method for the IMU denoising taskrepresents the noise relation as the model.5.1 INTRODUCTIONA modern Inertial Measurement Unit (IMU) consists of a three-axis gyroscope,which measures angular velocities, and a three-axis accelerometer, which yieldsgravity-compensated linear accelerations. Together with other devices, such asGlobal Positioning System (GPS), camera or Light Detection and Ranging sensor(LiDAR), navigation can be conducted accurately. However, in real-world applica-tions, IMU data noise can be generated from both the manufacturing process andthe operational environment, which can largely degrade the performance of InertialAided navigation [38]. Therefore, efficient IMU data denoising is important [6].Traditional denoising methods include low-pass filtering, wavelet-based denois-ing [33, 70], Savitzky-Golay filtering [71] and so on. Even though those methods arewell-studied, they are all based on hand-crafted sensor or noise modeling and lack ofgeneralization [181]. To address this concern, a few recent work [15, 181] leverageautomatic feature engineering with supervised deep learning to achieve promisingresults. However, in practice, the ground truth data for supervised learning can beexpensive or unavailable [18], which limits the wide adoption of these methods.For example, EuRoC [18] micro aerial vehicle (MAV) employs an accurate lasertracker system for obtaining the ground truth poses indoor. Such a setup may notbe applicable for urban city delivery drones or autonomous vehicles (AVs) outdoor[137].Therefore, it is desirable to explore unsupervised or self-supervised methodsfor IMU denoising, to further bring deep learning into practice. In fact, denoisingautoencoder [159], a self-supervised framework, has been shown to be powerfulon learning general noise-free features. Application-wise, masked autoencoder, amore general form, has achieved tremendous successes in both natural languageprocessing (NLP) [28] and computer vision (CV) [58] domains. While traditionalmethods formulate denoising as a filtering process, one very promising alternativeis to view denoising as a self-reconstruction process. In our case, IMU data itself99can be considered as the features. However, this has not been explored for IMUdenoising yet.In this chapter, to address the above research gap, the problem formulationof self-supervised learning for IMU denoising is firstly presented, and the pro-posed method is referred as IMUDB, which stands for Inertial Measurement UnitDenoising BERT. IMUDB can be viewed as a more general form of conventional fil-ter based methods, including three self-supervised tasks: 1) Masked IMU Modeling(MIM), 2) IMU Denoising Modeling (IDM) and 3) Future IMU Forecasting (FIF).The entire training process is self-supervised and does not require any externalground truth poses [182] or assumed noise-free IMU data. Further, we define afuture-aware uncertainty (FAU) and the corresponding inference to only trust thedenoised output when the uncertainty is tolerable. Further, the proposed IMUDB isdirectly evaluated via integrating with off-the-shelf inertial aided odometry methodson both EuRoC MAV and TUM-VI datasets. An ablation study on KITTI for [46]high-end IMUs is also conducted.The main contributions of our work are as follows:• As the first attempt, we explore self-supervised learning on IMU denoising forinertial-aided navigation and formulate the IMU denoising as a reconstructionprocess;• An innovative framework, IMUDB, is proposed with three different tasksfor IMU denoising and the importance and necessity of the future predictionsub-task is demonstrated;• An uncertainty-aware inference method is proposed and evaluated.5.2 RELATED WORKS5.2.1 Conventional IMU DenoisingTraditional low-pass/high-pass filtering (LPF/HPF) is one straightforward approachfor IMU denoising [70, 88], removing certain frequencies with appropriate thresh-olds. As real IMU data is time-varying, to reduce possible signal distortion and to100address the trade-off between the time and frequency resolutions, wavelet-baseddenoising technique [33, 70, 106] has been proposed, which can deal with multiplescale frequency ranges. Kang et al. [70] demonstrated a 19.72% performanceimprovement using wavelet-based denoising when compared with that of LPF. Nas-sar and El-Sheimy [106] showed wavelet denoised results are 46% to 63% betteron an INS/GPS integrated system application, respectively. While wavelet-baseddenoising has been widely adopted, Karaim et al. [71] argued that it may not besuitable for real-time implementations and explored using Savitzky-Golay (S-G)filter to achieve both faster speed and better accuracy.5.2.2 Supervised Learning on IMU Denoising For NavigationInspired by recent progresses in deep learning, as the first attempt, Esfahani et al.[35] proposed OriNet, a deep learning based 3-D orientation estimation method.OriNet significantly outperforms existing solutions, demonstrating the potential ofleveraging DNNs on IMU processing. While OriNet is based on recurrent neuralnetworks (RNNs), Brossard et al. [15] showed a simpler supervised learning methodwith convolutions neural networks (CNNs). Despite of the constraints to SO(3),the above methods regress the IMU data to the poses, instead of focusing on theIMU noise model. To address the above limitation, Zhang [181] proposed a newproblem formulation to learn the derived observable IMU integration terms. Theabove work can all be viewed as implicit [35] or explicit [181] IMU denoisingmethods. Nevertheless, all these methods are based on supervised learning andrequire annotated (ground truth) poses for training. In practice, such ground truthposes are expensive to obtain, or the annotation accuracy highly depends on thequality of IMU sensors.5.2.3 Self-supervised Learning on IMUSelf-supervised learning on IMU data processing is an exciting but new effort.Bidirectional Encoder Representations from Transformers (BERT), is a pioneeringself-supervised learning framework in the NLP domain [28]. Inspired by BERT, Xuet al. [170] proposed LIMU-BERT, adapting BERT to IMU representation. Theirmain adaptations and enhancements include data fusion and normalization, the101masking strategy and its light-weighted design. While inspiring and pioneering, thismethod [170] focuses on IMU representation instead of explicit denoising, therefore[170] does not necessarily yield good denoising performance. The demonstratedapplications in [170] are classification tasks of Human Activity Recognition (HAR)and Device Placement Classification (DPC), on mobile devices, which overallare still supervised learning tasks. More importantly, in [170], the downstreamclassification tasks need to be jointly trained with LIMU-BERT, while dominantinertial aided odometry / navigation algorithms are generally not differentiable andthus cannot be jointly trained. Instead, our work focuses on explicit IMU denoisingby exploring self-supervised learning, which does not require jointly training with adifferentiable odometry estimation algorithm. Since LIMU-BERT is the state-of-the-art (SOTA) auto-encoder of IMU representation learning methods, in our work,it is adopted as the auto-encoder backbone. However, it should be noted that anyalternative general masked auto-encoder can be employed instead in our proposedmethod.5.3 METHODS5.3.1 Problem FormulationConsider the IMU data sequence X ∈ RL×6 with sequence length L. We haveXˆ = ψ(X) (5.1)where ψ(,) means the denoising process and Xˆ is the denoised IMU sequence.Conventional filter-based methods, such as low-pass filter, can be represented as:Xˆ = B ·X (5.2)where B is the corresponding hand-crafted matrix and agnostic to the distributionof X . In our work, we formulate the IMU denoising problem as a learning processinstead:θˆ = argminθd(Q(X ,θ),Y ) (5.3)102Future IMU Forecasting(FIF)IMU Denoising Modeling(IDM)Masked IMU Modeling(MIM)..............................MaskIMU Normalized Sequences6D IMU Normalized Data PointShared WeightsShared WeightsSelf-Supervised by Past and Future Sensor DataFigure 5.1: The proposed IMUDB architecture. All terms are defined in Eqn.(5.6). The learning process is composed of three self-supervised learningtasks: Masked IMU Modeling (MIM), Future IMU Forecasting (FIF)and IMU Denoising Modeling (IDM). Green and yellow represent theobservations and predictions correspondingly. The red highlighted boxrepresents the current frame.where Q(,) means a learnable denoising function with parameters θ ; θˆ repre-sents the estimated parameters; Y represents the specific supervision signal (suchas ground truth annotations in supervised learning or observed features in self-supervised learning); d defines the loss/objective function. The problem formulationcan be viewed as a more generalized form of Eqn. (5.2), with extension from linearto nonlinear and the function is learned in a data-driven manner. In the followingsubsection, we further describe how the problem is formulated as a self-supervisedlearning task.5.3.2 Self-supervised LearningAll existing deep learning based IMU denoising methods use similar problemformulations as Eqn. (5.3), with X and Y being different (such as gyroscopeintegrals [15] or derived observable IMU integration terms [181]). Since in practicelabeled (ground truth) data is not always available, we explore a self-supervisedtraining process.Recent dominant masked autoencoders for CV [58] and NLP [28] frameworks103suggest that self-supervised learning should consider two tasks of the optimizationprocess: reconstructing the masked input and forecasting whether a given sequenceis the next sequence. In BERT [28], a popular NLP self-supervised learning frame-work, the first task is called the masked language model (MLM) and the second asthe next sentence prediction (NSP). The BERT learning process can be representedas the follows:< θˆ , θˆ ′ >= argminθ ,θ ′[d( f (XM,θ),XM)+d(g(X ,XK ,θ′),Y F)](5.4)where XM denotes the masked sequence; g(,) means the NSP optimization headwith parameters θ ′ ; Y F represents the binary value, i.e., 1 means that the givenarbitrary sequence XK is the true next sequence.LIMU-BERT [170], the very recent work of adapting BERT into the IMUfeature representation, suggests that only MLM is useful while NSP does not helpfor the IMU sequence. A naive implementation of LIMU-BERT on IMU denosingwould be as follows:θˆ = argminθd( f (XM,θ),XM) (5.5)However, it should be noted that LIMU-BERT [170] is only used to learn a goodIMU presentation instead of explicitly doing IMU denosing. In [170], all evaluatedtasks are trained in a supervised manner. Therefore, the observation of NSP beinguseless in [170] might not be true for our IMU denoising problem.Let xi, i = 1,2, ...,L, be the i-th frame of IMU data in the sequence X . In orderto better systematically study self-supervised learning on IMU denoising, in ourwork, we define a more complete learning process as follows:< θˆ , ωˆ >= argminθ ,ω[d( f (XM,θ),XM)+λ1 · (d(h( f (X ,θ),ω),XF)+d(h(X ,ω),XF))+λ2 ·d( f (X ,θ)L,xL](5.6)104where h(,) represents a function with parameters ω that predicts the future IMUsequence XF , and f (X ,θ)L is the L-th or the last frame of the prediction fromf (X ,θ). λ1 and λ2 are two scaling factors to weight the losses. One deep learningimplementation of modelling and learning f (,) and h(,) will be discussed in 5.3.4.The above optimization problem contains three heads (self-supervised learningtasks):• Masked IMU Modeling (MIM): d( f (XM,θ),XM). This task is to predict amasked portion of an entire IMU sequence;• IMU Denoising Modeling (IDM): d( f (X ,θ)L,xL). This task is to denoise theinput raw IMU sequence;• Future IMU Forecasting (FIF): d(h(X ,ω),XF)+d(h( f (X ,θ),ω),XF). Thistask is to predict the next future IMU sequence with either the input IMUsequence or the output of IDM.It is clear that the naive LIMU-BERT in Eqn. (5.5) can be viewed as a specialcase with λ1 = 0 and λ2 = 0. The FIF task can be viewed as a variant of the NSP taskin the original BERT. The main difference between FIF and NSP is that FIF predictsthe exact entire future or next sequence data while NSP only predicts whether agiven sequence is the next one.The run-time inference is as follows:xˆL = f (X , θˆ)L (5.7)where xˆL means the denoised last (L-th) frame of the IMU data.5.3.3 Future-Aware UncertaintyFor the estimated denoising output, i.e. xˆL, can we trust it to replace the raw datafor downstream tasks? In order to address this, below we propose one formulationof uncertainty by comparing the denoising outputs from IDM and FIF.As demonstrated in Fig.5.2, for an (L+1) length sequence, both the IDM task(i.e. Eqn. (5.7)) and FIF task can yield the denoised IMU at the current frame, asshown in the following:105xˆ′L+1 = h(X , ωˆ)1xˆ′′L+1 = h( f (X , θˆ), ωˆ)1(5.8)where the subscript 1 represents the first element of the predicted sequence fromh(,) and ωˆ means the learned parameters of the model h(,).In the ideal setting when there is no noise and both models f (,) and h(,) yieldzero testing error, the current observation and all predictions, namely, each elementin {xL+1, xˆL+1, xˆ′L+1, xˆ′′L+1}, should be equal to each other. However, in reality, bothtesting errors and noise exist, and the four elements are not necessarily equal. Thelarger the variance, the denoising model is more uncertain.More specifically, in our work, we define uncertainty as the unpredictabilityof the denoising process and the un-trustworthiness of the raw IMU data itself.For example, when the device is operated beyond the designed temperature range,the corresponding IMU data could be completely noise-corrupted due to IMUmalfunction. Since there is no perfect denoising algorithm, the denoising processmay fail or even introduce larger noise under certain situations. It should be notedthat the heuristic definition in this work is not the only way to define or model theuncertainty.Let xgtL+1 be the ground truth noise-free (L+1)-th frame of IMU data. With ourproblem formulation, here we propose representing the uncertainty as a combinationof the IMU noise itself (xgtL+1− xL+1) and the model testing error E f = xgtL+1− xˆL+1,as follows:µ = ||xgtL+1− xL+1||1+ ||xgtL+1− xˆL+1||1 (5.9)where µ means the uncertainty and ||, ||1 represents the L1 norm. However, inpractice, since xgtL+1 is unknown, µ cannot be obtained directly.In our learning process, since h(,) is trained by optimizing both d(h(X ,ω),XF)and d(h( f (X ,θ),ω),XF), the two testing errors of h(,) can be represented asfollows:106Eh,x = xgtL+1− xˆ′L+1Eh, f ,x = xgtL+1− xˆ′′L+1(5.10)where Eh,x means the testing error of h(,) with the raw IMU data as the input andEh, f ,x means the testing error of h(,) with the output of the IDM task as the input.Eh, f ,x essentially consists of both testing errors of f (,) and h(,) corresponding to thesame input. We suggest simply approximating the testing error of f (,) as follows:E f ≈ ||Eh,x−Eh, f ,x||1 = ||xˆ′L+1− xˆ′′L+1||1 (5.11)With Eqns. (5.9) and (5.11), we can have the following:µ ≥ ||xL+1− xˆL+1||12+||xgtL+1− xˆL+1||12≈ ||xL+1− xˆL+1||1+ ||xˆ′L+1− xˆ′′L+1||12(5.12)Further, we consider the uncertainty tolerance into the run-time inference:xL+1 =xL+1, if µ ≥ εxˆL+1, otherwise (5.13)where the threshold ε represents the uncertainty tolerance. Since in practice, ε canbe empirically chosen, we simplify the definition of the uncertainty as:Λ= ||xL+1− xˆL+1||1+ ||xˆ′L+1− xˆ′′L+1||1 (5.14)which is essentially twice the lower bound of the uncertainty µ . Then Eqn. (5.13) isstraightforward to implement by explicitly computing Λ:107... .........FAU orFigure 5.2: The future-aware uncertainty and inference process. Dashedeclipses are the observation and predictions of the current frame IMU,which are used to compute FAU to yield the uncertainty bounded denoiseoutput.xL+1 =xL+1, if Λ≥ εxˆL+1, otherwise (5.15)The interpretation of Eqn. (5.15) is quite intuitive: When the uncertainty lowerbound is too large, the denoised output will not be used to update the raw IMU data.Since Λ contains the items of the future IMU sequence forecast model h(,), wename it as future-aware uncertainty (FAU).5.3.4 Deep Neural NetworksThe above proposed approach can be implemented in different ways. Below wedescribe one specific implementation of deep neural networks, by leveraging aproven efficient light-weighted LIMU-BERT [170] as the architecture to model f (,)and h(,) in Eqn. (5.7) and Eqn. (5.8). It is worth emphasizing that the architecturecan be any kind of autoencoders. Here employing LIMU-BERT will make ourlater comparisons with LIMU-BERT more fair. Besides, it is the SOTA IMUrepresentation learning architecture.The proposed IMUDB architecture is illustrated in Fig.5.1. It consists of two108independent neural networks, each of which has three self-supervised learning tasks(i.e., MIM, FIF and IDM). For the MIM task: a random mask IM is firstly generatedand applied to the normalized IMU sequence X , giving a masked IMU sequenceXM; then X and IM are fed into f (,) for the prediction XˆM , which is self-supervisedby XM. For IDM, X is directly fed into f (,) to obtain a denoised Xˆ , and a lossis computed between the last element of Xˆ and X . For the FIF task, X and Xˆare fed into h(,) for the future IMU sequence prediction respectively, which isself-supervised by the same future IMU sequence. MIM and IDM tasks share thesame f (,). The same masking and normalization strategy of LIMU-BERT is usedin order to have fair comparisons. The L2 loss is employed to model d() in Eqn.(5.6). The whole training is end-to-end in a single stage.For inference, as illustrated in Fig.5.2 and in subsection 5.3.3, the normalizedIMU sequences are fed into two neural networks, obtaining three predictions. Thenthe uncertainty is computed following Eqn. (5.14) and then it is used to comparewith the uncertainty bound to determine whether the IMU data shall be updated.5.4 EXPERIMENTS AND RESULTS5.4.1 DatasetsTo demonstrate the generalization of our proposed method on diverse navigationapplications, we evaluate the proposed IMUDB on EuRoC [18] and TUM-VI [137]datasets, with ablation study on the KITTI [46] dataset.EuRoC [18] consists of images from two global-shutter cameras at 20 Hz andIMU sequences at 200 Hz from one ADIS16448 MEMS IMU sensor. An externalaccurate laser tracker system is used to construct the ground truth of the poses ofthe micro aerial vehicle (MAV). There are eleven different sub-datasets from twodifferent scenes: industrial machine hall and vicon room. Same as in previous work[181], MH{01,03,05}, V1{02} and V2{01,03} are used for training while the restare used for testing.TUM-VI [137] dataset contains sequences of cameras and IMU data fromvarious scenes. More specifically, the IMU is a low cost Bosch BMI160 16-bit3-axis MEMS unit, yielding the data at 200 Hz. An OptiTrack motion capture109system has been used to capture the ground truth poses. In our experiments, we usethe outdoor sequences {01,03,05,07,08} for training and {02,04,06} for testing.KITTI [46] is one of the most popular autonomous driving datasets. Theplatform is armed with four cameras, one Velodyne LiDAR and one GPS/IMUcombined system. The sensors are hardware synchronized, which yields the same10 Hz frequency for the LiDAR and cameras. The IMU sequence is sampled at100 Hz. As serving different AV applications, the IMU on the KITTI platform is ahigh-end OXT RT3000 when compared with the ones on EuRoC MAV or TUMV-VIdatasets. 09_30_0027 and 09_30_0034 sequences are used for testing while the restare used for training.5.4.2 Training DetailsOur methods are implemented in Pytorch-Lightning and trained on a single NVIDIAGeforce 2080ti GPU. During training, the batch size is fixed to 1024 and the ADAMoptimizer [76] is used with a starting learning rate of 1e−3. A cosine annealingwarm restart [99] learning rate strategy is applied with 5000 iterations for the firstrestart and each next restart uses 10 times more iterations. In practices, all modelsare found to converge well at the first restart. Heuristically λ1 is set as 1 and λ2 isset as 10 to balance the numerical scales. For both datasets, we conduct experimentswith the original frequency without downsampling and we apply a fixed sequencelength of 30 (namely, L = 30 in Eqn. (5.6)). ε is empirically selected as 3 times theconverged total loss.5.4.3 Evaluations with Odometry EstimationSetupThe ground truth of noise-free IMU data is not available, which makes the directevaluation of the denoising not feasible in practice. Besides, in practice, IMU data isusually integrated and deeply embedded in odometry estimation process. Therefore,instead, in our experiments, the end-to-end odometry estimation is utilized toevaluate the denoising, leveraging the ground truth poses. More specifically, theproposed IMUDB is tested with off-the-shelf open source visual-inertial odometry110[47], Open-VINS [47] as the baseline method since it is relatively new and popularwith code publicly available. It should be noted that Open-VINS has equipped withExtended Kalman Filter (EKF) which can compensate some noise [87]. Duringthe runtime, we feed a buffer with a queue size of L to IMUDB for the denoisedsequence with the same length. The last element is used as the updated denoisedcurrent IMU data, which is sent to the odometry estimation algorithm, as illustratedin Eqn. (5.7).It is worth emphasizing that there is no reported self-supervised learning methodfor explicit IMU denoising yet. For performance comparisons, for each dataset,besides the proposed IMUDB, another two models are also trained to demonstratethe contribution of each self-supervised learning task:• LIMU-BERT: by setting both λ1 and λ2 as 0, as a special case of IMUDB,we view this model as the equivalent form of LIMU-BERT [170];• IMUDB (without the FIF task): by setting λ1 as 0, the FIF task is turned off.111Table 5.1: EUROC DATASET: the POSITIONAL RMSE (M) OVER THE ENTIRE TRAJECTORY OF TEST SE-QUENCES.Sequence MH_02_easy MH_04_difficult V1_03_difficult V2_02_medium V1_01_easy Average Require Ground TruthTr. [38] 0.185 0.151 0.267 0.111 0.081 0.159 NoTLIO [94] 0.303 0.823 0.653 0.509 0.343 0.526 YesE2E [143] 3.307 5.199 5.329 2.897 8.166 4.980 YesZhang [181] 0.150 0.138 0.147 0.104 0.066 0.121 YesOpen-VINS [47] 0.182 0.370 0.084 0.078 0.066 0.156 NoLIMU-BERT* [170] 0.368 0.443 0.305 0.400 0.355 0.374 NoIMUDB (w/o FIF) 0.190 0.333 0.090 0.073 0.064 0.150 NoIMUDB (w/o FAU) 0.145 0.250 0.079 0.065 0.056 0.119 NoIMUDB 0.124 0.237 0.076 0.065 0.050 0.110 No*LIMU-BERT is not originally designed for IMU denoising.112Table 5.2: TUM-VI DATASET: the POSITIONAL RMSE (M) OVER THEENTIRE TRAJECTORY OF TEST SEQUENCES.Sequence outdoor2 outdoor4 outdoor6 AverageOpen-VINS [47] 25.666 57.386 58.518 47.190IMUDB (w/o FIF) 19.836 43.524 62.744 42.035IMUDB 15.498 15.498 51.475 32.976Comparisons with Competing MethodsAs the positional RMSE over the entire trajectory is widely reported by previouscompeting methods [47, 138, 181] on EuRoC dataset, we also adopt it in ourwork. From Table 5.1, it can be seen that the proposed IMUDB yields the leastpositional RMSE almost in all cases and yields the lowest average one. Eventhough for the MH_04_difficult, IMUDB’s performance is not the best, we stillsee a significant boost from the baseline Open-VINS. The poor performance ofLIMU-BERT support our previous claim that the naive LIMU-BERT may learngood IMU feature representations, but it is not necessarily capable of performing thedenoising task well. By adding the IDM task, the IMUDB (w/o FIF) yields certainimprovements from the baseline, which justifies the necessity of the IDM self-supervised learning task. Further, by comparing IMUDB (w/o FIF) and IMUDB, itcan be seen that the contribution of the FIF task is significant, boosting the averagefrom 0.150 to 0.110. Therefore, for the IMU denoising task, predicting the nextframes is meaningful. Even though the FIF task is different from the NSP taskin NLP, it shares the same spirit of leveraging future information to learn a betterrepresentation. Last but not the least, our proposed methods do not require groundtruth for training, which is more efficient and attractive for real-world deployment.113Table 5.3: KITTI DATASET: absolute pose error (APE) and relative pose error (RPE) results over the entire trajectoryOF TEST SEQUENCES.Sequence LIO-SAM LIMU-BERT [170] IMUDB (w/o FIF) IMUDB09_30_0027 1.099 / 0.766 1.102 / 0.766 1.104 / 0.766 1.082 / 0.76609_30_0034 2.927 / 0.860 2.873 / 0.859 2.837 / 0.859 2.836 / 0.859114Figure 5.3: Example: The EuRoC MH_04_difficult sequence trajectory (x-yplane) results of the ground truth (blue), Open-VINS baseline (red) andOpen-VINS armed with the proposed IMUDB (green).Fig.5.3 illustrates one qualitative example to demonstrate the performance ofIMUDB. In the three highlighted bounding box regions, it can be seen that withIMU denoising, the trajectory is much closer to the ground truth one. A real-time evaluation video is provided in the supplementary material for better intuitivedemonstration.115GeneralizationIn order to demonstrate that our proposed methods can be generalized to variousplatforms, we also test different methods on TUM-VI. From Table 5.2, it can beseen that IMUDB also yields the best performance and demonstrate significantlower RMSE than that of Open-VINS using raw IMU data. It should be noted that,since only indoor ground truth is available, the results reported in Table 5.2 onlycounts the beginning and ending sequences which are all indoor and have the groundtruth. However, since the errors propagate over the entire trip, our reported resultsare overall meaningful and representative. Since LIMU-BERT was not originallydesigned for IMU denoising, it yields very large even diverging errors in the outdoorenvironment. We therefore do not report its results here. Overall, these experimentsshow that the proposed IMUDB is applicable to low-cost IMU devices.Ablation Study on High-end IMUEven though our work mainly focus on low-cost IMUs, it is still of interest toevaluate the denoising impacts on high-end devices. Namely, different modelshave been trained on KITTI datasets. For KITTI, LIO-SAM [138] is employed toevaluate the impact of IMU denoising because it is relatively new with code publiclyavailable. The runtime denoising process is the same as on EuRoC. To use the sameevaluation metrics of the original LIO-SAM, the absolute pose error (APE) andrelative pose error (RPE) over the entire trajectory are used as the metrics for KITTIexperiments.From Table 5.3, it can be seen that IMUDB also yields the best performance.However, the improvements are marginal, likely due to that the original IMU data ismuch less noisy.5.4.4 Latency ProfilingFor the IMUDB model with L = 30 and inference batch size as 1, the averageinference time spent on a NVIDIA GeForce 2080ti is 2.73 ms. Such a latency isequivalent to 366 Hz, higher than the raw frequency in most common IMU devices[18, 46].1165.4.5 Limitations and Future WorksIn addition to demonstrating the efficiency of our proposed IMUDB, below wealso discuss about its limitations: 1) Sequence Size: In our current work, we use afixed sequence size (L) of 30 for the input, which is not necessary the best choicefor all applications. In future, an ablation study will be made to justify a betterchoice of the sequence length. 2) Deep Neural Networks and Hyper-Parameters:The one implementation we described is based on LIMU-BERT [170] and allhyper-parameters are directly inherited from the work without fine-tuning. It ispossible that with better fine-tuned hyper-parameters (such as the mask ratio), theperformance could be further improved.5.5 CONCLUSIONSIn this chapter, we first discuss the limitations of the existing deep learning basedIMU denoising methods; and we then present a self-supervised approach, theIMUDB, for IMU denoising in inertial aided navigation. The denoising task isformulated as a signal reconstruction problem. Further, one deep learning imple-mentation is described. The training can be achieved in an end-to-end mannerwithout requiring annotations. In addition, an innovative uncertainty-aware infer-ence process is proposed and discussed.The proposed IMUDB approach has been evaluated on different datasets toshow the generalization of IMUDB. Our experiments demonstrate the efficiencyand superiority of the proposed IMUDB over the competing methods. We believethis work could inspire many future related research in the navigation area.117Chapter 6Conclusion and Future Directions6.1 ConclusionMulti-sensor fusion and integration has been playing an important role for achievinghigh reliability and efficiency of modern robots. LiDAR, camera and IMU sensorsare three of the most commonly installed sensors on modern autonomous robots.In this thesis, the LiDAR-camera-IMU sensor system has been selected as thestudy target. At the robot sensor layer, there are three core sensor relations: spatialrelation, temporal relation and noise relation, which correspond to the calibration,synchronization and denoising tasks respectively. These tasks are critical pre-requisites for downstream application layer tasks; and currently the dominantconventional methods heavily rely on expensive hardware or hand-crafted features.The high cost of existing solutions could potentially prevent low-cost robots frombeing scalable and widely deployed. Also, the hand-crafted features could limit thegeneralization of the conventional approaches. This motivates us to explore otheralternative approaches. In the meanwhile, deep learning has been rapidly developedin recent years and the data-driven manner is a promising direction. However, thedominant supervised learning could be infeasible at the sensor layer due to thedifficulty and inefficiency of having sufficient human annotations. This leads us toeventually explore the self-supervised learning (SSL) direction, which leveragesfree labels. Currently there exists very few work which employ SSL on tasks at therobot sensor layer. More specifically, for the LiDAR-camera calibration task, the118performance are unsatisfactory and significantly worse than conventional methods.Besides, the unique challenges of the tasks at the robot sensor layer have barelydiscussed and addressed. In order to address such research gaps, this thesis doessystematic studies on applying self-supervised learning on the robot sensor layertasks.In Chapter 1, as the first attempt, we systematically point out the three coreconcerns at the sensor layer and review three different ways to represent the relations.The limitations of conventional dominant methods, challenges of the hardware-lesssolutions, challenges of supervised learning and self-supervised learning are alsodiscussed in details. The above viewpoints and discussion not only provide ourselvesa skeleton to conduct the following studies, but also lay a foundation for futureresearchers working on related topics.In Chapter 3, as the first work of this thesis, we focus on answering why previousfew attempts of applying SSL to the robot sensor layer tasks (more specifically, theLiDAR-camera calibration task) yield poor performance. In this work, we point outtwo potential issues of previous work: 1) the optimization objective function designmay not respect to the underlying geometry; and 2) the error tolerance has never beenconsidered. In order to address these two issues, RGGNet has been proposed, whichdirectly adopts the left-invariant Riemannian metrics loss to reflect the distance inthe output space and leverages a generative model to regularize the error tolerance.The proposed method represents the calibration relation straightforwardly as theoutput. The experimental results validate our hypothesis and the proposed methoddemonstrates the superior performance than previous SSL methods and equivalentperformance as the conventional method. In fact, the proposed RGGNet is the firstwork achieving parity performance as conventional methods among all sensor layertasks using self-supervised learning. This satisfactory performance shows the hugepotential of applying SSL and motivates us continue exploring this SSL direction.This first work suggests that, when the output space is a mathematically well-studiedone, directly using the geodesic distance as the objective function will be moreefficient. For such a realistic problem, the tolerance of calibration errors also playsan important role. More recently, there are various deep learning methods [101, 140,160, 166, 184] proposed for this task, which focus on leveraging iterative refinement,sequential data or better feature extraction neural network architecture. However, it119is worth mentioning that none of these methods reported better performance thanour RGGNet when compared fairly and they did not leverage the loss we proposein this work. Therefore, for future researchers in this specific topic, we suggestleveraging the loss function we proposed together with later verified improvements(i.e. iterative refinement, sequential inputs or more powerful neural networksarchitecture) as the starting point.As a natural extension from Chapter 3, in Chapter 4, the sensor temporalrelation, more specifically the LiDAR-camera synchronization task, has been studied.To get started, we first point out the limitations of the dominant conventionalapproaches and justify the motivations of using SSL on this task. While the twotasks (calibration and synchronization) look similar, the synchronization task ismore complicated than the calibration task, and representing the relation only asthe output is not feasible. As a result, LiCaS3, an innovative SSL framework,has been proposed for the LiDAR-camera synchronization task. The proposedframework represents the relation as both the output and the model, examiningthe hybrid relation representation. The proposed LiCaS3 has been intensivelyevaluated, demonstrating superior performance than conventional approaches andcompeting performance as in expensive hardware synchronization methods. Thissuccess further validates the idea of applying SSL on the robot sensor layer tasks.Since the proposed LiCaS3 is the very first framework that explores the LiDAR-camera synchronization topic and has shown really promising results using SSL,we recommend future researchers leverage our framework as the starting point.At last, in Chapter 5, the IMU denoising task has been studied. This task studiesthe noise sensor relation and represents the relation as a model. We first discussthe limitations of both conventional methods and existing supervised learningbased methods. Next, IMUDB, an innovative SSL method of IMU denoising forrobot navigation, has been proposed. The proposed IMUDB has been intensivelyevaluated on multiple datasets while it is integrated with a popular off-the-shelfodometry estimation method, and IMUDB demonstrates superior performance. Ourproposed method has been the first attempt to explore SSL on such IMU denoisingtopics, and it achieves superior performance. For researchers who are interestedin SSL based IMU denoising, our proposed framework is recommended as a goodstarting point.120In summary, this thesis systematically studies applying self-supervised learningon the robot sensor layer tasks. All our proposed methods have achieved competingperformance as conventional approaches but with the advantage of being lesshardware dependent. We believe that this thesis can inspire many future researchin the related area and provides a good foundation on exploring SSL at the robotsensor layer tasks.6.2 Future DirectionsBased on the accomplished work in this thesis, we would like to point out severaldirections for future research.6.2.1 A General and Unified FrameworkIn our thesis, three types of sensor relations are studied separately. However, theycould be combined as one unified framework, which may be more efficient and savethe computational cost. For example, the proposed RGGNet and LiCaS3 are bothextracting the features from the LiDAR and camera data. Even though the neuralnetworks are trained separately and the exact architectures are not the same, thereis nothing to prevent us from combing them at least at the feature extraction level.In fact, there are research [22], showing that the multi-modal multi-task has thepotential of achieving better performance. This could motivate us to explore fusingthe separate frameworks into one.In addition, in this thesis, only LiDAR, camera and IMU sensors have beenstudied. The sensor technology evolves over time and different types of sensorscould be used by different robots. It will be of great interest to develop a unifiedmulti-task framework that can take the input from different types of sensors.Further extending this idea, in practice, the downstream perception layer mayalready have many deep learning based tasks. The inputs of the perception tasksare basically the outputs of the sensor layer. There may already exist good featureextraction neural networks at the application layer. Joining the three or other sensorlayer tasks with perception tasks, such as object detection or semantic segmentation,is another promising research direction to explore.1216.2.2 Ensemble MethodEven though conventional methods for each task have limitations, in practice, whenthey are available, assembling the conventional methods and the SSL based methodcould yield superior performance than that of the individual methods.Two major common challenges of any deep learning based methods are the lackof interpretation and the long tail. The conventional methods, when available, canprovide good redundancy. The fusion of conventional and SSL methods can happenat different levels: feature level or task level. It is worthy exploring efficient ways toassemble the conventional methods with SSL based methods. For example, eventhough the motion based LiDAR-camera online calibration method is subject to thedrawbacks of hand-crafted features and motion estimation, it is possible to combinethe deep learning based method and the motion based method at different levels:• feature level: utilizing DNNs to estimate the ego motion from each modalityand then fitting it in the motion based framework;• task level: reasoning the output from two methods sequentially and rejectingthe one with larger uncertainty.6.2.3 More Sophisticated LearningAdvanced Neural Network ArchitectureIn our thesis, the main focus for each work is to validate the idea of applying SSLon the specific task. Therefore, we use the most basic neural network backbone forthe implementation, such as ResNet [56], and the latent space fusion strategy usedin this thesis is also pretty naive.The progress of neural network architectures has been rapid. For example,recently, the transformer based architecture [58] and the attention mechanism [157]have been demonstrating better performance. We believe that adopting more ad-vanced deep neural network architectures or fusion strategies could achieve betterperformance. One starting point can be replacing the backbone neural network inour thesis with a transformer based one and adding attention layers.122Online Learning and Life-long LearningIn our work, all training has been offline. In fact, since self-supervised learningmethods can be achieved with free labels, it is also possible to run them online. Itwould be interesting to study the performance when the methods are continuouslytrained online, namely, online life-long learning.6.2.4 Streaming ProcessingThe dominant rotational LiDAR sensor usually takes 100 ms or 50 ms per 360◦sweep. This may form a non-tolerable latency, for some downstream tasks. To tacklethis challenge, recently a few pioneering work proposed the concept of streamingprocess [4, 41]. The main idea is to run inference on each packet of the LiDARpoint clouds with a small FOV. The temporal accumulation can happen at the latentspace. By doing this, the camera data, which usually comes with a higher frequency,can also be fused at a higher frequency [4].In our work, all frameworks still require a full 360◦ sweep of the LiDAR pointclouds data. It would be beneficial if we integrate our proposed methods with thestreaming process, to further reduce the latency.6.2.5 SecuritySecurity is another important aspect, which our thesis has not touched yet. Securityis concerned at different layers. In fact, there are existing work for various appli-cation layer tasks, such as physical attacks on object tracker [30] and attacks onmulti-modal 3D object detector [2, 3].Similarly, when applying deep learning to the sensor layer tasks, security isequally important. For example, a shirt with certain patterns could result in signif-icant calibration estimation errors in RGGNet [177] or synchronization errors inLiCaS3 [178]. In addition, the robustness research to adversarial or physical attacksis also of broad interests. However, such a topic has not been explored yet and canbe an interesting future direction.123Bibliography[1] M. Abadi. Tensorflow: learning functions at scale. In Proceedings of the21st ACM SIGPLAN International Conference on Functional Programming,pages 1–1, 2016.[2] M. Abdelfattah, K. Yuan, Z. J. Wang, and R. Ward. Adversarial attacks oncamera-lidar models for 3d car detection. In 2021 IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS), pages 2189–2194.IEEE, 2021.[3] M. Abdelfattah, K. Yuan, Z. J. Wang, and R. Ward. Towards universalphysical attacks on cascaded camera-lidar 3d object detection models. In2021 IEEE International Conference on Image Processing (ICIP), pages3592–3596. IEEE, 2021.[4] M. Abdelfattah, K. Yuan, Z. J. Wang, and R. Ward. Multi-modal streaming3d object detection. arXiv preprint arXiv:2209.04966, 2022.[5] N. Ahmad, R. A. R. Ghazilla, N. M. Khairi, and V. Kasi. Reviews on variousinertial measurement unit (imu) sensor applications. International Journal ofSignal Processing Systems, 1(2):256–262, 2013.[6] M. Alam and J. Rohac. Adaptive data filtering of inertial sensors withvariable bandwidth. Sensors, 15(2):3282–3298, 2015.[7] R. Alami, R. Chatila, S. Fleury, M. Ghallab, and F. Ingrand. An architecturefor autonomy. The International Journal of Robotics Research, 17(4):315–337, 1998.[8] D. Alexey, P. Fischer, J. Tobias, M. R. Springenberg, and T. Brox.Discriminative unsupervised feature learning with exemplar convolutionalneural networks. IEEE Trans. Pattern Analysis and Machine Intelligence, 99,2015.124[9] E. Aljalbout, V. Golkov, Y. Siddiqui, M. Strobel, and D. Cremers. Clusteringwith deep learning: Taxonomy and new methods. arXiv preprintarXiv:1801.07648, 2018.[10] H. Alwassel, D. Mahajan, B. Korbar, L. Torresani, B. Ghanem, and D. Tran.Self-supervised learning by cross-modal audio-video clustering. Advances inNeural Information Processing Systems, 33, 2020.[11] M. Baradad Jurjo, J. Wulff, T. Wang, P. Isola, and A. Torralba. Learning tosee by looking at noise. Advances in Neural Information Processing Systems,34:2556–2569, 2021.[12] S. Bileschi. Fully automatic calibration of lidar and video streams from avehicle. In 2009 IEEE 12th International Conference on Computer VisionWorkshops, ICCV Workshops, pages 1457–1464. IEEE, 2009.[13] I. Bogoslavskyi and C. Stachniss. Efficient online segmentation for sparse3d laser scans. PFG–Journal of Photogrammetry, Remote Sensing andGeoinformation Science, 85(1):41–52, 2017.[14] R. Brooks. A robust layered control system for a mobile robot. IEEE journalon robotics and automation, 2(1):14–23, 1986.[15] M. Brossard, S. Bonnabel, and A. Barrau. Denoising imu gyroscopes withdeep learning for open-loop attitude estimation. IEEE Robotics andAutomation Letters, 5(3):4796–4803, 2020.[16] D. Brunet. A study of the structural similarity image quality measure withapplications to image processing. 2012.[17] D. Brunet, J. Vass, E. R. Vrscay, and Z. Wang. Geodesics of the structuralsimilarity index. Applied Mathematics Letters, 25(11):1921–1925, 2012.[18] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W.Achtelik, and R. Siegwart. The euroc micro aerial vehicle datasets. TheInternational Journal of Robotics Research, 35(10):1157–1163, 2016.[19] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan,Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset forautonomous driving. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 11621–11631, 2020.[20] N. Carlevaris-Bianco, A. K. Ushani, and R. M. Eustice. University ofmichigan north campus long-term vision and lidar dataset. The InternationalJournal of Robotics Research, 35(9):1023–1035, 2016.125[21] M. P. d. Carmo. Riemannian geometry. Birkhäuser, 1992.[22] S. Casas, A. Sadat, and R. Urtasun. Mp3: A unified model to map, perceive,predict and plan. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition, pages 14403–14412, 2021.[23] V. Casser, S. Pirk, R. Mahjourian, and A. Angelova. Depth predictionwithout the sensors: Leveraging structure for unsupervised learning frommonocular videos. In Thirty-Third AAAI Conference on ArtificialIntelligence (AAAI-19), 2019.[24] H.-J. Chien, R. Klette, N. Schneider, and U. Franke. Visual odometry drivenonline calibration for monocular lidar-camera systems. In 2016 23rdInternational conference on pattern recognition (ICPR), pages 2848–2853.IEEE, 2016.[25] S.-W. Chung, J. S. Chung, and H.-G. Kang. Perfect match: Self-supervisedembeddings for cross-modal retrieval. IEEE Journal of Selected Topics inSignal Processing, 14(3):568–576, 2020.[26] B. Della Corte, H. Andreasson, T. Stoyanov, and G. Grisetti. Unifiedmotion-based calibration of mobile multi-sensor platforms with time delayestimation. IEEE Robotics and Automation Letters, 4(2):902–909, 2019.[27] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: Alarge-scale hierarchical image database. In 2009 IEEE conference oncomputer vision and pattern recognition, pages 248–255. Ieee, 2009.[28] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training ofdeep bidirectional transformers for language understanding.arXiv:1810.04805, 2018.[29] L. Ding and C. Feng. Deepmapping: Unsupervised map estimation frommultiple point clouds. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 8650–8659, 2019.[30] L. Ding, Y. Wang, K. Yuan, M. Jiang, P. Wang, H. Huang, and Z. J. Wang.Towards universal physical attacks on single object tracking. In AAAI, pages1236–1245, 2021.[31] J. C. Eidson, M. Fischer, and J. White. Ieee-1588™ standard for a precisionclock synchronization protocol for networked measurement and controlsystems. In Proceedings of the 34th Annual Precise Time and Time IntervalSystems and Applications Meeting, pages 243–254, 2002.126[32] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a singleimage using a multi-scale deep network. Advances in Neural InformationProcessing Systems, 27:2366–2374, 2014.[33] N. El-Sheimy, S. Nassar, and A. Noureldin. Wavelet de-noising for imualignment. IEEE Aerospace and Electronic Systems Magazine, 19(10):32–39, 2004.[34] M. Elhousni, Y. Lyu, Z. Zhang, and X. Huang. Automatic building andlabeling of hd maps with deep learning. In Proceedings of the AAAIConference on Artificial Intelligence, volume 34, pages 13255–13260, 2020.[35] M. A. Esfahani, H. Wang, K. Wu, and S. Yuan. Orinet: Robust 3-dorientation estimation with a single particular imu. IEEE Robotics andAutomation Letters, 5(2):399–406, 2019.[36] H. Fan, X. Chang, W. Zhang, Y. Cheng, Y. Sun, and M. Kankanhalli.Self-supervised global-local structure modeling for point cloud domainadaptation with reliable voted pseudo labels. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages6377–6386, 2022.[37] L. Fan, F. Zhang, H. Fan, and C. Zhang. Brief review of image denoisingtechniques. Visual Computing for Industry, Biomedicine, and Art, 2(1):1–12,2019.[38] J. Farrell. Aided navigation: GPS with high rate sensors. McGraw-Hill, Inc.,2008.[39] F. Feng, X. Wang, and R. Li. Cross-modal retrieval with correspondenceautoencoder. In Proceedings of the 22nd ACM international conference onMultimedia, pages 7–16, 2014.[40] V. Fremont, P. Bonnifait, et al. Extrinsic calibration between a multi-layerlidar and a camera. In 2008 IEEE International Conference on MultisensorFusion and Integration for Intelligent Systems, pages 214–219. IEEE, 2008.[41] D. Frossard, S. Da Suo, S. Casas, J. Tu, and R. Urtasun. Strobe: Streamingobject detection from lidar packets. In Conference on Robot Learning, pages1174–1183. PMLR, 2021.[42] P. Furgale, T. D. Barfoot, and G. Sibley. Continuous-time batch estimationusing temporal basis functions. In 2012 IEEE International Conference onRobotics and Automation, pages 2088–2095. IEEE, 2012.127[43] P. Furgale, J. Rehder, and R. Siegwart. Unified temporal and spatialcalibration for multi-sensor systems. In 2013 IEEE/RSJ InternationalConference on Intelligent Robots and Systems, pages 1280–1286. IEEE,2013.[44] R. Garg, V. K. Bg, G. Carneiro, and I. Reid. Unsupervised cnn for singleview depth estimation: Geometry to the rescue. In European conference oncomputer vision, pages 740–756. Springer, 2016.[45] A. Geiger, F. Moosmann, Ö. Car, and B. Schuster. Automatic camera andrange sensor calibration using a single shot. In 2012 IEEE InternationalConference on Robotics and Automation, pages 3936–3943. IEEE, 2012.[46] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: Thekitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013.[47] P. Geneva, K. Eckenhoff, W. Lee, Y. Yang, and G. Huang. Openvins: Aresearch platform for visual-inertial estimation. In 2020 IEEE InternationalConference on Robotics and Automation (ICRA), pages 4666–4672. IEEE,2020.[48] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monoculardepth estimation with left-right consistency. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, pages 270–279,2017.[49] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow. Digging intoself-supervised monocular depth estimation. In Proceedings of the IEEEinternational conference on computer vision, pages 3828–3838, 2019.[50] L. Gomez, Y. Patel, M. Rusiñol, D. Karatzas, and C. Jawahar.Self-supervised learning of visual features through embedding images intotext topic spaces. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 4230–4239, 2017.[51] X. Gong, Y. Lin, and J. Liu. 3d lidar-camera extrinsic calibration using anarbitrary trihedron. Sensors, 13(2):1902–1918, 2013.[52] J. Graeter, A. Wilczynski, and M. Lauer. Limo: Lidar-monocular visualodometry. In 2018 IEEE/RSJ international conference on intelligent robotsand systems (IROS), pages 7872–7879. IEEE, 2018.128[53] A. Handa, M. Bloesch, V. Pa˘tra˘ucean, S. Stent, J. McCormac, andA. Davison. gvnn: Neural network library for geometric computer vision. InEuropean Conference on Computer Vision, pages 67–82. Springer, 2016.[54] K. Hassani and M. Haley. Unsupervised multi-task feature learning on pointclouds. In Proceedings of the IEEE/CVF International Conference onComputer Vision, pages 8160–8171, 2019.[55] M. Hauser and A. Ray. Principles of riemannian geometry in neuralnetworks. In Advances in neural information processing systems, pages2807–2816, 2017.[56] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for imagerecognition. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 770–778, 2016.[57] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedingsof the IEEE international conference on computer vision, pages 2961–2969,2017.[58] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Maskedautoencoders are scalable vision learners. arXiv:2111.06377, 2021.[59] G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of datawith neural networks. science, 313(5786):504–507, 2006.[60] K. Hornik, M. Stinchcombe, H. White, et al. Multilayer feedforwardnetworks are universal approximators.[61] B. Hou, N. Miolane, B. Khanal, M. C. Lee, A. Alansary, S. McDonagh, J. V.Hajnal, D. Rueckert, B. Glocker, and B. Kainz. Computing cnn loss andgradients for pose estimation with riemannian geometry. In InternationalConference on Medical Image Computing and Computer-AssistedIntervention, pages 756–764. Springer, 2018.[62] P. Hu, A. Huang, J. Dolan, D. Held, and D. Ramanan. Safe local motionplanning with self-supervised freespace forecasting. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages12732–12741, 2021.[63] L. Huynh, P. Nguyen, J. Matas, E. Rahtu, and J. Heikkila. Boostingmonocular depth estimation with lightweight 3d point fusion. InProceedings of the IEEE/CVF International Conference on Computer Vision,pages 12767–12776, 2021.129[64] G. Iyer, R. K. Ram, J. K. Murthy, and K. M. Krishna. Calibnet:Geometrically supervised extrinsic calibration using 3d spatial transformernetworks. In 2018 IEEE/RSJ International Conference on Intelligent Robotsand Systems (IROS), pages 1110–1117. IEEE, 2018.[65] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformernetworks. In Advances in neural information processing systems, pages2017–2025, 2015.[66] J. Jeong, T. S. Yoon, and J. B. Park. Multimodal sensor-based semantic 3dmapping for a large-scale environment. Expert Systems with Applications,105:1–10, 2018.[67] J. Jiang, P. Xue, S. Chen, Z. Liu, X. Zhang, and N. Zheng. Line featurebased extrinsic calibration of lidar and camera. In 2018 IEEE InternationalConference on Vehicular Electronics and Safety (ICVES), pages 1–6. IEEE,2018.[68] L. Jing and Y. Tian. Self-supervised visual feature learning with deep neuralnetworks: A survey. IEEE Transactions on Pattern Analysis and MachineIntelligence, 2020.[69] A. Johnston and G. Carneiro. Self-supervised monocular trained depthestimation using self-attention and discrete disparity volume. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pages 4756–4765, 2020.[70] C.-H. Kang, S.-Y. Kim, and C.-G. Park. Improvement of a low cost memsinertial-gps integrated system using wavelet denoising techniques.International Journal of Aeronautical and Space Sciences, 12(4):371–378,2011.[71] M. Karaim, A. Noureldin, and T. B. Karamat. Low-cost imu data denoisingusing savitzky-golay filters. In 2019 International Conference onCommunications, Signal Processing, and their Applications (ICCSPA),pages 1–5. IEEE, 2019.[72] K. Karsch, C. Liu, and S. B. Kang. Depth extraction from video usingnon-parametric sampling. In European Conference on Computer Vision,pages 775–788. Springer, 2012.[73] P. Kaul and B. Lall. Riemannian curvature of deep neural networks. IEEEtransactions on neural networks and learning systems, 2019.130[74] J. Kelly and G. S. Sukhatme. A general framework for temporal calibrationof multiple proprioceptive and exteroceptive sensors. In ExperimentalRobotics, pages 195–209. Springer, 2014.[75] D. Kent, T. Galluzzo, P. Bosscher, and W. Bowman. Robotic manipulationand haptic feedback via high speed messaging with the joint architecture forun-manned systems (jaus). In AUVSI Unmanned Systems Conference, 2004.[76] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXivpreprint arXiv:1412.6980, 2014.[77] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXivpreprint arXiv:1312.6114, 2013.[78] N. Komodakis and S. Gidaris. Unsupervised representation learning bypredicting image rotations. In International Conference on LearningRepresentations (ICLR), 2018.[79] D. Kortenkamp, R. Simmons, and D. Brugali. Robotic systems architecturesand programming. In Springer handbook of robotics, pages 283–306.Springer, 2016.[80] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification withdeep convolutional neural networks. In Advances in neural informationprocessing systems, pages 1097–1105, 2012.[81] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom.Pointpillars: Fast encoders for object detection from point clouds. InProceedings of the IEEE/CVF conference on computer vision and patternrecognition, pages 12697–12705, 2019.[82] Y. LeCun. The mnist database of handwritten digits. http://yann. lecun.com/exdb/mnist/, 1998.[83] Y. LeCun and I. Misra. Self-supervised learning: The dark matter ofintelligence. Meta AI, 23, 2021.[84] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553):436–444, 2015.[85] J. Levinson and S. Thrun. Automatic online calibration of cameras andlasers. In Robotics: Science and Systems, volume 2, 2013.131[86] B. Li, C. Shen, Y. Dai, A. Van Den Hengel, and M. He. Depth and surfacenormal estimation from monocular images using regression on deep featuresand hierarchical crfs. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 1119–1127, 2015.[87] M. Li and A. I. Mourikis. Online temporal calibration for camera–imusystems: Theory and algorithms. The International Journal of RoboticsResearch, 33(7):947–964, 2014.[88] Z. Li, J. Wang, J. Gao, B. Li, and F. Zhou. A vondrak low pass filter for imusensor initial alignment on a disturbed base. Sensors, 14(12):23803–23821,2014.[89] H. Liang, C. Jiang, D. Feng, X. Chen, H. Xu, X. Liang, W. Zhang, Z. Li, andL. Van Gool. Exploring geometry-aware contrast and clusteringharmonization for self-supervised 3d object detection. In Proceedings of theIEEE/CVF International Conference on Computer Vision, pages 3293–3302,2021.[90] M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun. Multi-task multi-sensorfusion for 3d object detection. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 7345–7353, 2019.[91] C.-H. Lin, E. Yumer, O. Wang, E. Shechtman, and S. Lucey. St-gan: Spatialtransformer generative adversarial networks for image compositing. InProceedings of the IEEE Conference on Computer Vision and PatternRecognition, pages 9455–9464, 2018.[92] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprintarXiv:1312.4400, 2013.[93] F. Liu, C. Shen, and G. Lin. Deep convolutional neural fields for depthestimation from a single image. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 5162–5170, 2015.[94] W. Liu, D. Caruso, E. Ilg, J. Dong, A. I. Mourikis, K. Daniilidis, V. Kumar,and J. Engel. Tlio: Tight learned inertial odometry. IEEE Robotics andAutomation Letters, 5(4):5653–5660, 2020.[95] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang.Self-supervised learning: Generative or contrastive. IEEE Transactions onKnowledge and Data Engineering, 2021.132[96] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang,L. Dong, et al. Swin transformer v2: Scaling up capacity and resolution. InProceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 12009–12019, 2022.[97] S. Lohit and P. Turaga. Learning invariant riemannian geometricrepresentations using deep nets. In Proceedings of the IEEE InternationalConference on Computer Vision, pages 1329–1338, 2017.[98] J. Loncaric. Geometrical analysis of compliant mechanisms in robotics(euclidean group, elastic systems, generalized springs. 1985.[99] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warmrestarts. arXiv preprint arXiv:1608.03983, 2016.[100] R. C. Luo and C.-C. Chang. Multisensor fusion and integration: A review onapproaches and its applications in mechatronics. IEEE Transactions onIndustrial Informatics, 8(1):49–60, 2011.[101] X. Lv, B. Wang, Z. Dou, D. Ye, and S. Wang. Lccnet: Lidar and cameraself-calibration using cost volume network. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, pages 2894–2901,2021.[102] B. Mersch, X. Chen, J. Behley, and C. Stachniss. Self-supervised point cloudprediction using 3d spatio-temporal convolutional networks. In Conferenceon Robot Learning, pages 1444–1454. PMLR, 2022.[103] D. L. Mills. Internet time synchronization: the network time protocol. IEEETransactions on communications, 39(10):1482–1493, 1991.[104] F. M. Mirzaei, D. G. Kottas, and S. I. Roumeliotis. 3d lidar–camera intrinsicand extrinsic calibration: Identifiability and analytical least-squares-basedinitialization. The International Journal of Robotics Research, 31(4):452–467, 2012.[105] N. Mohajerin and M. Rohani. Multi-step prediction of occupancy grid mapswith recurrent neural networks. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 10600–10608, 2019.[106] S. Nassar and N. El-Sheimy. Wavelet analysis for improving ins and ins/dgpsnavigation accuracy. The Journal of Navigation, 58(1):119–134, 2005.133[107] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodaldeep learning. In ICML, 2011.[108] T.-M. Nguyen, S. Yuan, M. Cao, Y. Lyu, T. H. Nguyen, and L. Xie. Ntuviral: A visual-inertial-ranging-lidar dataset, from an aerial vehicleviewpoint. The International Journal of Robotics Research, page02783649211052312, 2021.[109] N. J. Nilsson. A mobile automaton: An application of artificial intelligencetechniques. Technical report, Sri International Menlo Park Ca ArtificialIntelligence Center, 1969.[110] N. J. Nilsson. Principles of artificial intelligence. Springer Science &Business Media, 1982.[111] E. Olson. A passive solution to the sensor synchronization problem. In 2010IEEE/RSJ International Conference on Intelligent Robots and Systems,pages 1059–1064. IEEE, 2010.[112] A. Paigwar, D. Sierra-Gonzalez, Ö. Erkent, and C. Laugier.Frustum-pointpillars: A multi-stage approach for 3d object detection usingrgb camera and lidar. In Proceedings of the IEEE/CVF InternationalConference on Computer Vision, pages 2926–2933, 2021.[113] G. Pandey, J. McBride, S. Savarese, and R. Eustice. Extrinsic calibration ofa 3d laser scanner and an omnidirectional camera. IFAC ProceedingsVolumes, 43(16):336–341, 2010.[114] G. Pandey, J. R. McBride, S. Savarese, and R. M. Eustice. Automatictargetless extrinsic calibration of a 3d lidar and camera by maximizingmutual information. In Twenty-Sixth AAAI Conference on ArtificialIntelligence, 2012.[115] C. Park, P. Moghadam, S. Kim, S. Sridharan, and C. Fookes. Spatiotemporalcamera-lidar calibration: A targetless and structureless approach. IEEERobotics and Automation Letters, 5(2):1556–1563, 2020.[116] F. Park, A. Murray, and J. McCarthy. Designing mechanisms for workspacefit. In Computational Kinematics, pages 295–306. Springer, 1993.[117] M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, andL. Zettlemoyer. Deep contextualized word representations. arxiv 2018.arXiv preprint arXiv:1802.05365, 12, 1802.134[118] A. Piergiovanni, A. Angelova, and M. S. Ryoo. Evolving losses forunsupervised video representation learning. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages133–142, 2020.[119] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on pointsets for 3d classification and segmentation. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, pages 652–660,2017.[120] X. Qi, R. Liao, Z. Liu, R. Urtasun, and J. Jia. Geonet: Geometric neuralnetwork for joint depth and surface normal estimation. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages283–291, 2018.[121] S. Qian, D. Xue, H. Zhang, Q. Fang, and C. Xu. Dual adversarial graphneural networks for multi-label cross-modal retrieval. 2021.[122] K. Qiu, T. Qin, J. Pan, S. Liu, and S. Shen. Real-time temporal androtational calibration of heterogeneous sensors using motion correlationanalysis. IEEE Transactions on Robotics, 2020.[123] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler,A. Y. Ng, et al. Ros: an open-source robot operating system. In ICRAworkshop on open source software, volume 3, page 5. Kobe, Japan, 2009.[124] T. Raj, F. H. Hashim, A. B. Huddin, M. F. Ibrahim, and A. Hussain. Asurvey on lidar scanning mechanisms. Electronics, 9(5):741, 2020.[125] M. Ramezani, Y. Wang, M. Camurri, D. Wisth, M. Mattamala, andM. Fallon. The newer college dataset: Handheld lidar, inertial and visionwith ground truth. In 2020 IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS), 2020.[126] N. Rasiwasia, J. Costa Pereira, E. Coviello, G. Doyle, G. R. Lanckriet,R. Levy, and N. Vasconcelos. A new approach to cross-modal multimediaretrieval. In Proceedings of the 18th ACM international conference onMultimedia, pages 251–260, 2010.[127] F. A. Reda, D. Sun, A. Dundar, M. Shoeybi, G. Liu, K. J. Shih, A. Tao,J. Kautz, and B. Catanzaro. Unsupervised video interpolation using cycleconsistency. In Proceedings of the IEEE/CVF International Conference onComputer Vision, pages 892–900, 2019.135[128] J. Rehder, J. Nikolic, T. Schneider, T. Hinzmann, and R. Siegwart.Extending kalibr: Calibrating the extrinsics of multiple imus and ofindividual axes. In 2016 IEEE International Conference on Robotics andAutomation (ICRA), pages 4304–4311. IEEE, 2016.[129] J. Rehder, R. Siegwart, and P. Furgale. A general approach to spatiotemporalcalibration in multisensor systems. IEEE Transactions on Robotics, 32(2):383–398, 2016.[130] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-timeobject detection with region proposal networks. Advances in neuralinformation processing systems, 28, 2015.[131] T. Richter. Ssim as global quality metric: a differential geometry view. In2011 Third International Workshop on Quality of Multimedia Experience,pages 189–194. IEEE, 2011.[132] R. Rishav, R. Battrawy, R. Schuster, O. Wasenmüller, and D. Stricker.Deeplidarflow: A deep learning architecture for scene flow estimation usingmonocular camera and sparse lidar. In 2020 IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS), pages 10460–10467.IEEE, 2020.[133] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks forbiomedical image segmentation. In International Conference on Medicalimage computing and computer-assisted intervention, pages 234–241.Springer, 2015.[134] R. Salakhutdinov and G. Hinton. Deep boltzmann machines. In Artificialintelligence and statistics, pages 448–455. PMLR, 2009.[135] D. Scaramuzza, A. Harati, and R. Siegwart. Extrinsic self calibration of acamera and a 3d laser range finder from natural scenes. In 2007 IEEE/RSJInternational Conference on Intelligent Robots and Systems, pages4164–4169. IEEE, 2007.[136] N. Schneider, F. Piewak, C. Stiller, and U. Franke. Regnet: Multimodalsensor registration using deep neural networks. In 2017 IEEE intelligentvehicles symposium (IV), pages 1803–1810. IEEE, 2017.[137] D. Schubert, T. Goll, N. Demmel, V. Usenko, J. Stückler, and D. Cremers.The tum vi benchmark for evaluating visual-inertial odometry. In 2018IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), pages 1680–1687. IEEE, 2018.136[138] T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and R. Daniela. Lio-sam:Tightly-coupled lidar inertial odometry via smoothing and mapping. InIEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), pages 5135–5142, 2020.[139] H. Shao, A. Kumar, and P. Thomas Fletcher. The riemannian geometry ofdeep generative models. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops, pages 315–323, 2018.[140] J. Shi, Z. Zhu, J. Zhang, R. Liu, Z. Wang, S. Chen, and H. Liu. Calibrcnn:Calibrating camera and lidar by recurrent convolutional neural network andgeometric constraints. In 2020 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), pages 10197–10202. IEEE, 2020.[141] S. Shi, X. Wang, and H. Li. Pointrcnn: 3d object proposal generation anddetection from point cloud. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 770–779, 2019.[142] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo.Convolutional lstm network: A machine learning approach for precipitationnowcasting. Advances in neural information processing systems, 28, 2015.[143] J. P. Silva do Monte Lima, H. Uchiyama, and R.-i. Taniguchi. End-to-endlearning framework for imu-based 6-dof odometry. Sensors, 19(17):3777,2019.[144] H. Sommer, R. Khanna, I. Gilitschenski, Z. Taylor, R. Siegwart, and J. Nieto.A low-cost system for high-rate, high-accuracy temporal calibration forlidars and cameras. In 2017 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), pages 2219–2226. IEEE, 2017.[145] M. W. Spong. Remarks on robot dynamics: Canonical transformations andriemannian geometry. In Proceedings 1992 IEEE International Conferenceon Robotics and Automation, pages 554–559. IEEE, 1992.[146] N. Srivastava, R. Salakhutdinov, et al. Multimodal learning with deepboltzmann machines. In NIPS, volume 1, page 2. Citeseer, 2012.[147] M. Stölzle, T. Miki, L. Gerdes, M. Azkarate, and M. Hutter. Reconstructingoccluded elevation information in terrain maps with self-supervised learning.IEEE Robotics and Automation Letters, 7(2):1697–1704, 2022.137[148] K. H. Strobl and G. Hirzinger. Optimal hand-eye calibration. In 2006IEEE/RSJ international conference on intelligent robots and systems, pages4647–4653. IEEE, 2006.[149] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui,J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception forautonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVFconference on computer vision and pattern recognition, pages 2446–2454,2020.[150] L. Tamas and Z. Kato. Targetless calibration of a lidar-perspective camerapair. In Proceedings of the IEEE International Conference on ComputerVision Workshops, pages 668–675, 2013.[151] Z. Taylor and J. Nieto. Motion-based calibration of multimodal sensorextrinsics and timing offset estimation. IEEE Transactions on Robotics, 32(5):1215–1229, 2016.[152] B. A. Team. Apollo: Open source autonomous driving, 2017. URLhttps://github. com/ApolloAuto/apollo.[153] S. Thrun, M. Montemerlo, H. Dahlkamp, D. Stavens, A. Aron, J. Diebel,P. Fong, J. Gale, M. Halpenny, G. Hoffmann, et al. Stanley: The robot thatwon the darpa grand challenge. Journal of field Robotics, 23(9):661–692,2006.[154] C. Tian, L. Fei, W. Zheng, Y. Xu, W. Zuo, and C.-W. Lin. Deep learning onimage denoising: An overview. Neural Networks, 131:251–275, 2020.[155] H. Tian, Y. Chen, J. Dai, Z. Zhang, and X. Zhu. Unsupervised objectdetection with lidar clues. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 5962–5972, 2021.[156] R. Unnikrishnan and M. Hebert. Fast extrinsic calibration of a laserrangefinder to a camera. Robotics Institute, Pittsburgh, PA, Tech. Rep.CMU-RI-TR-05-09, 2005.[157] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neuralinformation processing systems, 30, 2017.[158] S. Verma, J. S. Berrio, S. Worrall, and E. Nebot. Automatic extrinsiccalibration between a camera and a 3d lidar using 3d point and plane138correspondences. In 2019 IEEE Intelligent Transportation SystemsConference (ITSC), pages 3906–3912. IEEE, 2019.[159] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting andcomposing robust features with denoising autoencoders. In Proceedings ofthe 25th international conference on Machine learning, pages 1096–1103,2008.[160] G. Wang, J. Qiu, Y. Guo, and H. Wang. Fusionnet: Coarse-to-fine extrinsiccalibration network of lidar and camera with hierarchical point-pixel fusion.In 2022 International Conference on Robotics and Automation (ICRA),pages 8964–8970. IEEE, 2022.[161] K. Wang, Q. Yin, W. Wang, S. Wu, and L. Wang. A comprehensive surveyon cross-modal retrieval. arXiv preprint arXiv:1607.06215, 2016.[162] R. Wang, F. P. Ferrie, and J. Macfarlane. Automatic registration of mobilelidar and spherical panoramas. In 2012 IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition Workshops, pages 33–40.IEEE, 2012.[163] J. Watson, O. Mac Aodha, V. Prisacariu, G. Brostow, and M. Firman. Thetemporal opportunist: Self-supervised multi-frame monocular depth. InProceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 1164–1174, 2021.[164] X. Weng and K. Kitani. Monocular 3d object detection with pseudo-lidarpoint cloud. In Proceedings of the IEEE/CVF International Conference onComputer Vision Workshops, pages 0–0, 2019.[165] B. Wu, A. Wan, X. Yue, and K. Keutzer. Squeezeseg: Convolutional neuralnets with recurrent crf for real-time road-object segmentation from 3d lidarpoint cloud. In 2018 IEEE International Conference on Robotics andAutomation (ICRA), pages 1887–1893. IEEE, 2018.[166] S. Wu, A. Hadachi, D. Vivet, and Y. Prabhakar. This is the way: Sensorsauto-calibration approach based on deep learning for self-driving cars. IEEESensors Journal, 21(24):27779–27788, 2021.[167] X. Wu, Z. Wu, Y. Zhang, L. Ju, and S. Wang. Multi-video temporalsynchronization by matching pose features of shared moving subjects. InProceedings of the IEEE International Conference on Computer VisionWorkshops, pages 0–0, 2019.139[168] L. Xie, C. Xiang, Z. Yu, G. Xu, Z. Yang, D. Cai, and X. He. Pi-rcnn: Anefficient multi-sensor 3d object detector with point-based attentive cont-convfusion module. In Proceedings of the AAAI Conference on ArtificialIntelligence, volume 34, pages 12460–12467, 2020.[169] D. Xu, D. Anguelov, and A. Jain. Pointfusion: Deep sensor fusion for 3dbounding box estimation. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 244–253, 2018.[170] H. Xu, P. Zhou, R. Tan, M. Li, and G. Shen. Limu-bert: Unleashing thepotential of unlabeled data for imu sensing applications. In Proceedings ofthe 19th ACM Conference on Embedded Networked Sensor Systems, pages220–233, 2021.[171] R. Xu, C. Xiong, W. Chen, and J. Corso. Jointly modeling deep video andcompositional text to bridge vision and language in a unified framework. InProceedings of the AAAI Conference on Artificial Intelligence, volume 29,2015.[172] F. Yan and K. Mikolajczyk. Deep correlation for matching images and text.In Proceedings of the IEEE conference on computer vision and patternrecognition, pages 3441–3450, 2015.[173] N. Yang, L. v. Stumberg, R. Wang, and D. Cremers. D3vo: Deep depth, deeppose and deep uncertainty for monocular visual odometry. In IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), June2020.[174] Z. Yang, P. Wang, Y. Wang, W. Xu, and R. Nevatia. Lego: Learning edgewith geometry all at once by watching videos. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPR), June2018.[175] Z. Yin and J. Shi. Geonet: Unsupervised learning of dense depth, opticalflow and camera pose. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 1983–1992, 2018.[176] K. Yuan and Z. J. Wang. A simple self-supervised imu denoising method forinertial aided navigation. IEEE Robotics and Automation Letters, 8(2):944–950, 2023.[177] K. Yuan, Z. Guo, and Z. J. Wang. Rggnet: Tolerance aware lidar-cameraonline calibration with geometric deep learning and generative model. IEEERobotics and Automation Letters, 5(4):6956–6963, 2020.140[178] K. Yuan, L. Ding, M. Abdelfattah, and Z. J. Wang. Licas3: A simplelidar–camera self-supervised synchronization method. IEEE Transactions onRobotics, 38(5):3203–3218, 2022.[179] J. Zhang and S. Singh. Visual-lidar odometry and mapping: Low-drift,robust, and fast. In 2015 IEEE International Conference on Robotics andAutomation (ICRA), pages 2174–2181. IEEE, 2015.[180] L. Zhang, R. Merrifield, A. Deguet, and G.-Z. Yang. Powering the world’srobots—10 years of ros. Science Robotics, 2(11):eaar1868, 2017.[181] M. Zhang, M. Zhang, Y. Chen, and M. Li. Imu data processing for inertialaided navigation: A recurrent neural network based approach. In 2021 IEEEInternational Conference on Robotics and Automation (ICRA), pages3992–3998. IEEE, 2021.[182] M. Zhang, X. Zuo, Y. Chen, Y. Liu, and M. Li. Pose estimation for groundrobots: On manifold representation, integration, reparameterization, andoptimization. IEEE Transactions on Robotics, 37(4):1081–1099, 2021.[183] Q. Zhang and R. Pless. Extrinsic calibration of a camera and laser rangefinder (improves camera calibration). In 2004 IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS)(IEEE Cat. No.04CH37566), volume 3, pages 2301–2306. IEEE, 2004.[184] G. Zhao, J. Hu, S. You, and C.-C. J. Kuo. Calibdnn: multimodal sensorcalibration for perception using deep neural networks. In Signal Processing,Sensor/Information Fusion, and Target Recognition XXX, volume 11756,pages 324–335. SPIE, 2021.[185] H. Zhao, O. Gallo, I. Frosio, and J. Kautz. Loss functions for imagerestoration with neural networks. IEEE Transactions on computationalimaging, 3(1):47–57, 2016.[186] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot:Image bert pre-training with online tokenizer. arXiv preprintarXiv:2111.07832, 2021.[187] L. Zhou, Z. Li, and M. Kaess. Automatic extrinsic calibration of a cameraand a 3d lidar using line and plane correspondences. In 2018 IEEE/RSJInternational Conference on Intelligent Robots and Systems (IROS), pages5562–5569. IEEE, 2018.141[188] L. Zhou, G. Sun, Y. Li, W. Li, and Z. Su. Point cloud denoising review: fromclassical to deep learning-based approaches. Graphical Models, 121:101140,2022.[189] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning ofdepth and ego-motion from video. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 1851–1858, 2017.[190] Y. Zhou and O. Tuzel. Voxelnet: End-to-end learning for point cloud based3d object detection. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 4490–4499, 2018.[191] D. Zuñiga-Noël, J.-R. Ruiz-Sarmiento, R. Gomez-Ojeda, andJ. Gonzalez-Jimenez. Automatic multi-sensor extrinsic calibration formobile robots. IEEE Robotics and Automation Letters, 4(3):2862–2869,2019.142
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Wang, Z. Jane
Self-supervised learning at the sensor layer in robotics
Text