Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Neubauer, Kai
Master of Applied Science - MASc
University of British Columbia
This thesis focused on developing a framework for evaluating the 3D reconstruction created through simultaneous localization and mapping (SLAM). Common benchmarking techniques rely on having a fully realized ground truth, either as a trajectory or 3D structural data. The proposed system evaluates relations between known sets of landmarks and compares those to relations between the corresponding features within the reconstruction. This requires far less data for evaluation and is much more applicable outside of controlled environments. First, CAD models of objects that are known to be within the environment are processed. The models are sampled into high resolution point clouds that are used in further processing. Features are extracted from the point clouds and compared to each other. For the experiments in this thesis, the features of interest were the centroids of each object with the comparison being the Euclidean distance within each other. The point cloud representation of these objects are then separated from the reconstruction of the environment using a combination of a passthrough filter and Euclidean cluster extraction. The same procedure for feature extraction and comparison used for the ground truth models is applied to these objects. The metric used to evaluate the success of the SLAM algorithm is the error between the ground truth comparison of these objects and the relations between their reconstructed counterparts. Experiments were conducted using a ground vehicle robot in both simulation and a laboratory environment. The experiments were used to evaluate the performance of different SLAM algorithms, and to compare the evaluation against evaluation metrics used in literature. The proposed system is designed to be extendable for use with any point cloud feature. It aims to help evaluate the suitability of algorithms within scenarios not properly covered within available data sets.
Benchmarking SLAMReconstructions in Absence of aComplete Ground TruthbyKai NeubauerB.A.Sc. University of British Columbia, 2019A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Mechanical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)February 2022© Kai Neubauer, 2022The following individuals certify that they have read, and recommend to the Collegeof Graduate Studies for acceptance, a thesis entitled:Benchmarking SLAM Reconstructions in Absence of a Complete Ground Truthsubmitted by Kai Neubauer in partial fulfilment of the requirements of the degree ofMaster of Applied ScienceHomayoun Najjaran, School of EngineeringSupervisorYang Cao, School of EngineeringSupervisory Committee MemberDean Richert, School of EngineeringSupervisory Committee MemberSumi Siddiqua, School of EngineeringUniversity ExamineriiAbstractThis thesis focused on developing a framework for evaluating the 3D reconstructioncreated through simultaneous localization and mapping (SLAM). Common benchmarkingtechniques rely on having a fully realized ground truth, either as a trajectory or 3D struc-tural data. The proposed system evaluates relations between known sets of landmarksand compares those to relations between the corresponding features within the recon-struction. This requires far less data for evaluation and is much more applicable outsideof controlled environments.First, CAD models of objects that are known to be within the environment are pro-cessed. The models are sampled into high resolution point clouds that are used in furtherprocessing. Features are extracted from the point clouds and compared to each other. Forthe experiments in this thesis, the features of interest were the centroids of each objectwith the comparison being the Euclidean distance within each other.The point cloud representation of these objects are then separated from the recon-struction of the environment using a combination of a passthrough filter and Euclideancluster extraction. The same procedure for feature extraction and comparison used forthe ground truth models is applied to these objects. The metric used to evaluate thesuccess of the SLAM algorithm is the error between the ground truth comparison of theseiiiobjects and the relations between their reconstructed counterparts. Experiments wereconducted using a ground vehicle robot in both simulation and a laboratory environment.The experiments were used to evaluate the performance of different SLAM algorithms,and to compare the evaluation against evaluation metrics used in literature.The proposed system is designed to be extendable for use with any point cloud feature.It aims to help evaluate the suitability of algorithms within scenarios not properly coveredwithin available data sets.ivLay SummaryThe general purpose of this thesis is to design a system capable of evaluating 3D repre-sentations of environments by looking at the relationships between known objects withinthem. The objects are extracted from the reconstructed environment and processed todetermine 3D features. These 3D features are then compared against the equivalent fea-tures of modeled versions of the objects. Experiments show that this method is a suitablealternative to evaluate mapping algorithms for environments where a full representation isunavailable; however, within simulation environments where a full representation is fullyavailable, existing approaches are better benchmarking tools.vPrefaceThe work performed in this thesis was conducted under the supervision of Dr. Najjaranin the Advanced Control and Intelligent Systems (ACIS) Laboratory at the Universityof British Columbia. This was conducted in partnership with Rosen Group through aMITACS Accelerate Grant.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Context and Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1viiTABLE OF CONTENTS1.3 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Chapter 2: Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . 52.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 SLAM Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3.1 2D SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3.2 Visual SLAM (vSLAM) . . . . . . . . . . . . . . . . . . . . . . . . 112.3.3 3D LiDAR SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3.4 Loop Closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.4 Active SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.4.1 Exploration vs. Exploitation . . . . . . . . . . . . . . . . . . . . . . 162.4.2 Path Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.5 Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.5.1 SLAM Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.5.2 Benchmarking metrics . . . . . . . . . . . . . . . . . . . . . . . . . 232.6 Limitations of SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Chapter 3: Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.1 Software and Hardware Stack . . . . . . . . . . . . . . . . . . . . . . . . . 263.2 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31viiiTABLE OF CONTENTS3.2.1 Localization Quality . . . . . . . . . . . . . . . . . . . . . . . . . . 313.2.2 Mapping Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.3 Proposed Framework Implementation . . . . . . . . . . . . . . . . . . . . . 413.4 Verification Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.4.1 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.4.2 Real-World Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Chapter 4: Validation Results and Discussion . . . . . . . . . . . . . . . . 474.1 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.1.1 Localization Quality (APE Metric) . . . . . . . . . . . . . . . . . . 484.1.2 Mapping Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494.2 Real-World Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58Chapter 5: Conclusions and Future Work . . . . . . . . . . . . . . . . . . 655.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71ixList of TablesTable 2.1 Summary of Datasets Usable for SLAM. . . . . . . . . . . . . . . . 20Table 3.1 Locations of the centres of each support pillar. . . . . . . . . . . . . 44Table 3.2 Locations of the laboratory landmarks. . . . . . . . . . . . . . . . . 46Table 4.1 APE statistics for the tank simulation. . . . . . . . . . . . . . . . . 48Table 4.2 Cloud to mesh (C2M) statistics for the tank simulation. . . . . . . . 54Table 4.3 Estimated centroids of the tank landmarks. . . . . . . . . . . . . . . 54Table 4.4 Displacement errors for the tank landmarks. . . . . . . . . . . . . . 55Table 4.5 Error statistics for the tank validation using the proposed metric. . 55Table 4.6 Estimated centroids of the laboratory landmarks. . . . . . . . . . . 59Table 4.7 Displacement error of the laboratory landmarks. . . . . . . . . . . . 59Table 4.8 Results for the real-world experiments using the proposed metric. . 59xList of FiguresFigure 2.1 An example of how mapping quality can be interpreted given dif-ferent metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Figure 3.1 Jackal UGV equipped with d435i cameras and VLP-32MR LiDAR. 27Figure 3.2 A simulated Jackal equipped with a VLP-32MR and single d435icamera. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Figure 3.3 Software abstraction using ROS. . . . . . . . . . . . . . . . . . . . 29Figure 3.4 Rough alignment of a point cloud to a mesh, before (left) and after(right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Figure 3.5 Drawing of the boxes used to derive the URDF representation. . . 37Figure 3.6 Boxes used as landmarks for SLAM (left), boxes modeled in URDF(center), modeled boxes sampled with features extracted (right). . 39Figure 3.7 UML diagram for the proposed framework. . . . . . . . . . . . . . 42Figure 3.8 Model of the tank used within the simulated experiments. . . . . . 45Figure 4.1 Estimated trajectories aligned to the ground truth for the tanksimulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50xiLIST OF FIGURESFigure 4.2 Components of the estimated trajectories compared to the groundtruth for the tank simulation. . . . . . . . . . . . . . . . . . . . . . 51Figure 4.3 Comparison of trajectory accuracy for the tank simulation. . . . . 52Figure 4.4 Trajectory APE statistics for the tank simulation. . . . . . . . . . 53Figure 4.5 Final maps of the tank produced by different SLAM algorithms.Top: HDL Graph SLAM (left), LeGO-LOAM (right); Bottom:RTAB-Map (2D LiDAR) (left), RTAB-Map (3D LiDAR) (right)). . 56Figure 4.6 Cloud to mesh (C2M) distances for RTAB-Map (2D LiDAR). . . . 57Figure 4.7 Cloud to mesh (C2M) distances for RTAB-Map (3D LiDAR). . . . 58Figure 4.8 Cloud to mesh (C2M) distances for HDL Graph SLAM. . . . . . . 60Figure 4.9 Cloud to mesh (C2M) distances for LeGO-LOAM. . . . . . . . . . 61Figure 4.10 Cloud to mesh (C2M) distances comparison for the tank simulation. 62Figure 4.11 Unsigned cloud to mesh (C2M) distances statistics for the tanksimulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Figure 4.12 An example of a correct loop closure detection. Current image(left) matched to an existing key frame (right). . . . . . . . . . . . 63Figure 4.13 An example of a false loop closure detection. Current image (left)matched to an existing key frame (right). . . . . . . . . . . . . . . 63Figure 4.14 Final maps of the tank produced by SLAM algorithms (Top (left-to-right): BLAM, HDL Graph SLAM, LeGO-LOAM; Bottom (left-to-right): RTAB-Map (2D LiDAR), RTAB-Map (3D LiDAR)). . . 64xiiList of AcronymsAMCL Adaptive Monte Carlo localization.APE Absolute pose error.API Application programming interface.BA Bundle adjustment.BLAM Berkeley localization and mapping.BRIEF Binary robust independent elementary fea-tures.C2M Cloud to mesh distance.CAD Computer aided design.CNN Convolutional neural network.DAE Digital asset exchange.DPC-MN Deep point cloud mapping network.EKF Extended Kalman filter.FAST Features from accelerated segment test.GPS Global positioning system.xiiiList of AcronymsGPU Graphics processing unit.GTSAM Georgia Tech Smoothing and Mapping.HDL High definition LiDAR.ICP Iterative closest point.IMU Inertial measurement unit.IR Infrared.LOAM LiDAR odometry and mapping.LeGO-LOAM Lightweight and ground-optimized LiDARodometry and mapping.LiDAR Light detection and ranging.MAP Maximum a posteriori.MBBC Match bounding-box centres.MSE Mean squared error.NDT Normal distributions transform.OBJ Object.ORB Oriented FAST and rotated BRIEF.PCD Point cloud data.PCL Point Cloud Library.PCN Point convolution network.xivList of AcronymsPnP Perspective-n-point.RADAR Radio detection and ranging.RANSAC Random sample consensus.RBPF Rao-Blackwellized particle filter.RGB-D Red, green, blue, and depth.RMSD Root mean squared distance.ROS Robot Operating System.RTAB-Map Real-Time Appearance-Based Mapping.SD Standard deviation.SE Squared error.SLAM Simultaneous localization and mapping.SURF Speeded-up robust features.UGV Unmanned ground vehicle.UML Unified modeling language.URDF Unified robot description format.vSLAM Visual SLAM.xvAcknowledgementsFirst, I would like to thank my research supervisor Dr. Homayoun Najjaran of theSchool of Engineering at the University of British Columbia. He was readily availablewith advice and a sense of humour when I needed it most. I experienced many hardshipsthroughout this degree and this thesis would not exist without his motivation. I wish himthe best with his future endeavors at the University of Victoria.I would also like to thank Rosen Group for providing me with the equipment necessaryto conduct this research and Jesse Roch for his help in conducting my experiments.Simulations can only take you so far and having access to the resources Rosen Groupgave me allowed me to learn so much more.Finally, I want to appreciate my family and friends for their support and continuousencouragement while I worked endlessly on this thesis. This would not have been possiblewithout them.xviDedicationI dedicate this achievement to my parents and all the hamsters (Hubble, Pip, Coconut,and Acorn) who graced myself with their presence during the writing of this thesis. Youwere the living stressballs that got me through this ordeal.xviiChapter 1Introduction1.1 MotivationAccurate mapping is the first stage in many inspection pipelines. This work can behazardous if done manually. Automation makes the workplace safer and allows employeesto work on less menial tasks. This work was completed in partnership with Rosen Group,who conduct inspections of floating roof tanks as a commercial business. The first step intheir inspection is a complete mapping of the inside of the tank using manual labour. Thisresearch aims to streamline their inspection pipeline and remove the human risk factor.Although there is much research on simultaneous localization and mapping (SLAM),there is relatively little literature on evaluating SLAM, particularly with environment-based approaches. It is frustratingly common to find no qualitative assessment of a newdevelopment within this research space.1.2 Context and BackgroundSimultaneous localization and mapping (SLAM) is an essential technique for autonomyof mobile robots first started when Smith and Cheeseman [1] proposed a method to11.3. OBJECTIVESestimate pose (i.e. position and orientation) among objects. The concepts were expandedupon by Leonard and Durrant-Whyte [2], who argued that the localization and mappingproblems must be resolved at the same time; hence, SLAM involves the act of creatinga map with landmarks, while also determining a robot’s pose within the map relative tothe landmarks. During recent years, taking advantage of multiple sensors such as camera,radio detection and ranging (RADAR), and light detection and ranging (LiDAR), hasresulted in new kinds of maps [3, 4]. The landmarks used in the mapping process are oftenvisual; however, there are some examples where acoustic noise is used as landmarks [5].A wide range of algorithms are utilized in SLAM techniques to ensure the robustness ofan indicated location. These algorithms include original research about SLAM [4], dataacquisition and association [5] and map storage [6].1.3 ObjectivesThe outcome of this thesis is a environment-based 3D SLAM benchmarking procedurethat functions without the need for a high-resolution ground truth scan. This systemcan be used independent of the environment. It allows for more realistic benchmarkingsituations in the real world, as having an existing high-resolution map eliminates the needfor a SLAM algorithm. The benchmark was applied to multiple algorithms in differentscenarios, both real and simulated, to provide recommendations for various situations.21.4. CONTRIBUTIONS1.4 ContributionsThe contributions of this thesis are as follows:• A benchmarking criterion that can be used independent of the environment• Benchmarks for different SLAM algorithms on an unmanned ground vehicle (UGV)in both simulation and the real world• Recommendations for SLAM algorithms within the testing environments‘1.5 Thesis OutlineThis thesis is structured as follows.Chapter 2 provides the literature review of techniques and concepts used within thisthesis. To start with, a brief introduction of of the history of SLAM is given. A rundownof common sensors is given with pros and cons for each. Then, examples on how differ-ent SLAM algorithms may be classified is described with attention given to algorithmsused within this thesis. More intelligent exploration, called active SLAM, is outlined toillustrate the broader landscape of mobile navigation.Chapter 3 details the methodology behind the benchmarking approach and experi-mental setup. Details about the hardware and software used, as well as the reasons forchoosing them are given. The evaluation metrics used are described in detail includ-ing descriptions of the algorithms applied, followed by the experimental setups for bothreal-world and simulated testing.31.5. THESIS OUTLINEChapter 4 demonstrates the results of the thesis. Comparisons are made to existingapproaches. Limitations are outlined and discussed.Chapter 5 concludes with the significance of the thesis and its contributions to thefield of mobile mapping. Possible improvements and integrations for the system are alsogiven.4Chapter 2Literature Review2.1 BackgroundThe primary concerns with SLAM are i) to manage the uncertainties associated withlocalizing the robot due to imperfect measurements and the error that may occur inmapping the environment due to occlusion, missing points, faults and discontinuities.Simultaneously estimating the location of the robot and reducing the mapping errors isthe objective SLAM. Beside these, “active SLAM” also requires that the robot maximizethe efficiency of exploring the entire environment by planning its trajectory in real time,parallel to solving the SLAM problem. The use of “active” is because in contrast “passiveSLAM” requires a predefined trajectory [7].More sophisticated SLAM solutions provide semantic segmentation data for objectsdetected during exploration. Bowman et al. [8] integrated inertial, geometric, and seman-tic data to decompose the problem into two interconnected optimization problems: theestimation of discrete data association and landmark class probabilities, and a continu-ous optimization over the metric states. The inertial and geometric measurements wereused to track the sensor trajectory locally, while the semantic measurements were used toconstruct a map of objects to perform loop closure. This can be further improved upon52.1. BACKGROUNDusing a convolutional neural network (CNN) for depth prediction [9]. The same networkarchitecture can be used for semantic segmentation of the image. The system can run inreal-time by delegating depth refinement to the GPU.A typical SLAM architecture can be broken into two key components, the front-endand the back-end [10]. The front-end takes sensor data, performs feature extraction anddata association, and passes it to the back-end. The data association can be classified aseither short-term (e.g. feature tracking between frames) or long-term (e.g. loop closure).The back-end receives the processed data and uses it to estimate the agent’s localizationand the environment’s geometrical reconstruction.The first step in any SLAM solution is robot perception. Using onboard sensors, therobot must be able to recognize obstacles in its surroundings in order to navigate aroundthem. Computer vision techniques process data from the sensors and output informationabout the robot’s environment. Techniques exist for both passive sensing (e.g. stereovision) and active sensing (e.g. LiDAR).One approach for observation involves the use of camera and 3D object recognitionalgorithms. These algorithms have numerous advantages such as utilizing depth informa-tion and their independence to ambient lighting [11, 12]. This category of approach iscommonly referred to as visual SLAM (vSLAM). The success of vSLAM is primarily de-pendent on the robust feature detection and description algorithms available [7]. For 2Dimages, techniques such as scale-invariant feature transform (SIFT) [13] and speeded-uprobust features (SURF) [14] can be used to extract features from the image. Use of thesealgorithms is commonly referred to as view-based methods. The 3D shape is projectedonto a 2D plane and the objects are recognized based on texture, grayscale, and shadow.62.1. BACKGROUNDMore recently, 3D shape-based approaches have been introduced [11, 12, 15]. Thesemethods consider the geometry and shape features for determining the object. The infor-mation used is obtained from 3D sensors such as RGB-D cameras, stereo vision cameras,and LiDAR. Traditionally, these techniques have had low accuracy and required largercomputational loads compared to view-based approaches. State-of-the-art approacheshave involved converting 3D point clouds into volumetric images and using a 3D CNNfor classification [12]. These methods have many limitations including dramatic increasesin computational load with an increase of volumetric image resolution, difficulties in per-forming transformation and rotation, and irreversibility, meaning point-wise operations(e.g. segmentation and labeling) are impossible after the conversion. Recent researchby Chen et al. [15] have proposed a method called deep point cloud mapping networks(DPC-MN) that can efficiently recognize objects directly from the point cloud data andestimate their poses. It exploits self-occlusion of objects as a criterion for unsupervisedmachine learning. The network is trained on CAD models that have been converted topoint clouds.In real-world environments, 3D sensors enable exploration based on point cloud ofobjects around them. The resolution of representation and computational efficiency is themain shortcoming of the point cloud. Point convolution networks (PCN) can be utilizedfor normal point clouds to address this weakness [11, 12]. This method benefits fromKd-tree-based local permutation and spatial commutative pooling strategies to establishdisordered and unstructured point clouds.A large amount of data is processed during SLAM. Many methods use a probabilisticmodel such as an extended Kalman filter (EKF) to estimate the states of the robot. An72.1. BACKGROUNDEKF allows for easy addition of additional states but has poor scalability. It has O(N2)for both its time and space complexities [16]. A way to reduce the complexity is touse a Rao-Blackwellized particle filter (RBPF) instead of an EKF. This is the basis forthe FastSLAM algorithm which separates the SLAM problem into a robot localizationproblem and landmark estimation problems that rely on the robot pose estimate [17]. Amodified version of the algorithm, FastSLAM 2.0 [18] relies on the most recent sensormeasurement in addition to the motion estimate, making the algorithm less wasteful.Both versions of FastSLAM require O(NM) memory and O(MlogN) time, where M isthe number of particles in the particle filter and N is the number of landmarks. Runningon the same hardware, the time taken to process a data set was reduced from 7,807seconds using an EKF to 403 seconds with regular FastSLAM with 50 particles. The timewas reduced even further to 140 seconds using FastSLAM 2.0 with a single particle.In an RBPF, the higher-order states have no effect in the observation. A downside toan RBPF is that velocity is considered to be constant, which may not be a true assumptionfor a robot working in cluttered environments. A solution to this is to modify the RBPFto be used for constant acceleration to allow the robot to start, control its speed, and stopwithout interrupting SLAM [19].Filtering techniques only estimate the most recent pose. This makes them very effi-cient but results in compounding errors. Using batch methods that optimize over entiretrajectories have notable performance increases [8]. Optimization variables (i.e. robotposes and landmarks) are represented as a set of nodes in a graph. Edges between tworobot poses exist when an odometry measurement is available between them. Likewise,an edge between a robot pose and landmark exists when the landmark can be observed82.2. SENSORSfrom the robot pose.Each of the position estimates must be maintained and frequently updated throughoutthe exploration. Tree data structures have become common to deal with the unknown sizeof the environment and allow for relatively lower access times. An open-source framework,relying on octrees, called OctoMap [6] has been widely used to allow for easy scalabilitywithin mapping applications. A fixed-size 3D occupancy grid can be used to improveperformance but requires the dimensions of the to be known during initialization.2.2 SensorsSLAM is a continuing area of research due to the rapidly advancing hardware andsensor technologies. Proposed algorithms may perform well in theory but require reval-uation with betterment of hardware. For example, Kalogeiton et. al [7] proposed analgorithm that performed well in simulation; however, the disparity maps resulting fromtheir cameras was quite poor when it was implemented with inferior hardware for physicaltesting. New sensors called “event cameras” only record data when the scene of inter-est changes [20]. They benefit from reduced latency and lower power usage but requireentirely new vision algorithms that can consider their asynchronous nature. Combiningevent cameras with omni-directional cameras and LiDAR will potentially lead to superiorSLAM performances; however, this combination has not undergone sufficient testing tosupport this claim [21].92.3. SLAM CLASSIFICATION2.3 SLAM ClassificationSLAM can result in either 2D or 3D maps. 2D SLAM has had extensive research sincethe 1980’s while 3D SLAM is much more recent and complex problem. 3D SLAM ap-proaches can generally be classified into two groups, vSLAM and LiDAR SLAM, based onthe sensors used for the reconstruction. They can also be categorized as online, where thedata is filtered and processed in real time during exploration, or full SLAM algorithms,where the map and entire trajectory is estimated. Full SLAM is much more compu-tationally expensive and is generally performed after exploration (i.e. offline). Offlinealgorithms allow for more accurate mapping, but their maps can only be used in futurenavigation of the environment. This section briefly outlines various 2D SLAM approacheswhile going into greater detail of 3D vSLAM and LiDAR techniques. It ends by discussingdifferent loop closure approaches. More attention is given to online SLAM.2.3.1 2D SLAMTwo major milestones for 2D SLAM were EKF-SLAM [2, 3] and FastSLAM [17,18]. EKF-SLAM uses an extended Kalman filter (EKF) to perform its estimation. Thesystem is broken into two sequential steps, a prediction (time update) and a correction(measurement update). Each step involves two state vectors to represent the robot andlandmarks, the position vector and the measurement vector. The prediction uses thechange observed in each landmark relative to the estimated robot and landmarks’ stateto estimate a new position. Noted issues with EKF-SLAM include its reliance on linearizedmodels and storage issues. Storage requirements for EKF-SLAM grow quadratically with102.3. SLAM CLASSIFICATIONevery additional landmark, making it illsuited to large mapping problems.FastSLAM [17] uses a particle filter approach to solve the SLAM problem. It uses aRao-Blackwellized particle filter (RBPF) to reduce the high dimensional state-space of theSLAM problem. The key benefit of FastSLAM over EKF-SLAM is its reduced complexity.The probability distribution is on the trajectory rather than a single pose, representingthe landmarks as a set of independent Gaussians that can be processed linearly ratherthan quadratically. Each particle in the particle filter contains one EKF for each of thelandmarks. An improved version of FastSLAM called FastSLAM 2.0 [18] which addedthe most recent sensor measurement to the robot pose estimation phase. This made thealgorithm less wasteful compared to the original FastSLAM.GMapping [22] is another particle-filter based approach that has become the defaultSLAM algorithm within the Robot Operating System (ROS). It builds 2D occupancygrids from a series of 2D laser scans and is commonly used alongside adaptive MonteCarlo localization (AMCL) [23] for autonomous navigation.2.3.2 Visual SLAM (vSLAM)Real-Time Appearance-Based Mapping (RTAB-Map) [24–26] is a popular open-sourceSLAM library. It was originally distributed as an appearance-based loop closure detectionapproach and memory management tool [24] but its continued development added vSLAMsupport [26] and LiDAR SLAM [25]. The ability to support both visual and LiDAR basedSLAM within a single framework makes it more desirable compared to other algorithmsthat only support one approach. It operates online by outputting with a maximum delayafter receiving a sensor update; however, as the map size increases it becomes increasingly112.3. SLAM CLASSIFICATIONdifficult to update the map at regular intervals. The computation will be executed butmay not finish processing within the requested time step. It also has the ability touse exterioceptive sensors (i.e. cameras and LiDAR) alongside proprioceptive sensors(e.g. wheel encoders and inertial measurement units) to improve odometry estimation infeature rich environments.ORB-SLAM [27] was a popular single camera method that relied on ORB [28] featuresfor feature extraction. Its extension, ORB-SLAM2 [29], is a popular stereo mappingSLAM algorithm. Another extended version of the algorithm, called ORB-SLAM3 [30],capable of visual-inertial SLAM, was recently released in beta. ElasticFusion [31] is areal-time dense vSLAM algorithm that uses a RGB-D camera to create a surfel-basedmap. The system requires fast and tight coupling between the mapping and tracking onthe GPU, making it impossible to use with middleware like ROS.2.3.3 3D LiDAR SLAMBerkeley Localization and Mapping (BLAM) [32] is a 3D LiDAR graph-based SLAM.It was published as a GitHub repository with almost no documentation. Loop closureappears to be detected using local scan matching and it has a tough time closing largeloops [25]. After loop closure, the map is optimized using the Georgia Tech Smoothingand Mapping (GTSAM) [33] library.Lidar Odometry and Mapping (LOAM) [34] extracts features by calculating the rough-ness of the point within its local region. Points with high roughness are then selected asedge features. Lastly, point feature to edge/plane scan matching is used to find cor-respondences between scans. LOAM optimized its SLAM by fusing the result of two122.3. SLAM CLASSIFICATIONalgorithms running in parallel. The first estimates the velocity at a high frequency butlow fidelity, while the other performs fine matching and registration of the point cloudsat a frequency of an order of magnitude lower. Lightweight and Ground-Optimized LidarOdometry and Mapping (LeGO-LOAM) [35] is an optimized version of LOAM for groundvehicles with lesser onboard resources. Small UGVs like the Jackal from Clearpath lacka suspension system, meaning the data can be distorted by uneven terrain. Having aLiDAR located close to the ground means that the ground appears quite often withinscans. If the environment is outdoors, grass or leaves on the ground can distort whatwould otherwise be a flat plane. Since LOAM relies on roughness for its scan-matching,this is particularly problematic and can lead to increased drift over time. LeGO-LOAMaccounts for these problems by performing point cloud segmentation to discard unreliablefeatures after ground separation and adding loop closure to its optimizations.HDL Graph SLAM [36] uses a graph optimization-based approach using the workfrom [37] and [38] with the ground plane and GPS positions as constraints for indoor andoutdoor scenarios respectively.2.3.4 Loop ClosureLoop closure is the agent’s ability to refine a map through recognizing an area that italready visited. There are several SLAM algorithms without loop closure capabilities, suchas Hector SLAM [39] and LOAM [34], which can lead to compounded errors over time.LeGO-LOAM [35] added loop closure using the iterative closest point (ICP) algorithm [40,41]. Mistaking a location for another previously visited location is known as false loopclosure and can result in catastrophic mapping errors.132.3. SLAM CLASSIFICATIONRTAB-Map [24–26] uses a bag-of-words approach for its loop closure. Extracted visualfeatures are quantized into an incremental visual word vocabulary that can be queried ata later time to better localize the robot. Any features within the OpenCV library canbe used for the loop closure (e.g. [13], SURF [14], BRIEF [42], FAST [43], ORB [28])The loop closure hypotheses are estimated through a Bayes filter. If a hypothesis meets afixed threshold, the loop is considered closed and the map is transformed. The transfor-mation is done in a similar way using Perspective-n-Point (PnP) RANSAC implementedin OpenCV [44] or, if a laser scan or point cloud is available, using ICP [40, 41].Maximum a Posteriori (MAP) estimation or Bundle Adjustment (BA) are other meth-ods that can be used to optimize graph based problems like SLAM. BA is the process ofjointly estimating the pose of 3D points and the corresponding pose of the sensor thatobtained those points, so that the reprojection error is minimized. Point and sensor posesin the long-term memory are used to refine more recently obtained sensor data. This issimilar to how SLAM works, although no map is produced (i.e. only localization of thesensor is performed). There are a large number of unknowns involved in these computa-tions so certain aspects need to be exploited for it to run in real-time. The sparse natureof the problem is the primary exploit involved. Not all features are observable from everysensor reading, meaning there is only a subset of features that need to be considered whenperforming the estimation. It is used within the odometry nodes of RTAB-Map [24–26]using the g2o library [38]. Likewise, HDL Graph SLAM [36] also uses the g2o library foroptimization.142.3. SLAM CLASSIFICATIONPoint Cloud RegistrationFor 3D SLAM, the output map is a single point cloud constructed from multiplesmaller point clouds. These point clouds need to be aligned and combined to form asingle output in a process called registration. Techniques can be categorized as eitherrough or fine registration depending on if they use an initial guess. Rough registrationmethods are either feature-based, where they measure correspondences between detected3D features [45]; randomized [46]; or probabilistic [47].The simplest point cloud registration method is the Match Bounding-Box Centres(MBBC) method [48], where the centres of the bounding boxes for each cloud are aligned.This method is only suitable for clouds that are largely identical. For example, theregistration can be used as a first step to compare terrain before and after a landslide.Iterative closest point (ICP) [40, 41] and normal distributions transform (NDT) [49]are the most common registration methods. More advanced registration methods such as[50] work to leverage deep learning capabilities to allow end-to-end registration, where arough registration method is not needed.ICP [40, 41] iteratively minimizes the point-to-point distances between clouds. Eachiteration works in minimising the following equation:E(R, t) =Nm∑i=1Nd∑j=1wi,j||mi − (Rdj + t)||2 (2.1)where Nm and Nd are the points in the model and measured datasets respectively.Weights, wi,j, are either 1 if mi is the point closest to point dj or 0 otherwise. R and t arethe rotation and translation components of the transformation applied to the measured152.4. ACTIVE SLAMpoints.NDT [49] uses a combination of normal distributions for the cloud instead of theindividual points. The distributions describe the probability of finding part of the surfaceat any given point in space. [51] found NDT converges from a larger range of initial poseestimates than ICP but is not as predictable. Registration could be successful from alarge initial error but fail with a significantly smaller one.2.4 Active SLAMThe primary goal of active SLAM is to reduce the entropy of the map while fully ex-ploring the area. This ensures appropriate decision making during the mission and allowsfor the minimal amount of area to be traversed with minimal uncertainty in the map [52].The entropy is reduced by adjusting the estimated locations after a previously mappedlocation is detected again through loop closure. Re-planning the robot’s trajectory aftera loop closure can also reduce the chance of collisions [53, 54]. If a loop closure is incor-rectly detected, the robot may become trapped in an already explored area with no wayof travelling to unexplored locations [52].2.4.1 Exploration vs. ExploitationA key constraint of active SLAM is the exploration vs. exploration principle. Thelearned pose of the robot depends on a belief state that is often high-dimensional andnon-linear. This makes common techniques for stochastic planning and reinforcementlearning inapplicable [55]. A truly autonomous system needs to know when the mapping162.5. BENCHMARKINGis finished. A tradeoff has to be made in terms of exploration (i.e. minimizing uncertaintyin the policy space) and exploitation (i.e. making the best use of current data).2.4.2 Path PlanningIn order to be used within the ROS navigation stack, a 2D costmap is required. Somealgorithms (e.g. RTAB-MAP [24–26]) can build the costmap internally, while othersrequire additional processing with a tool like Octomap Server [6]. Planning is a continuousprocess, needing to be updated as new information comes in [55].2.5 BenchmarkingRelatively few publications on benchmarking SLAM exist. In 1996, Lee [56] adopted asystematic approach for his thesis work with five main criteria. Schwertfeger [57] adoptedthese same criteria in his thesis on evaluating mapping performance. These properties fora valid mapping metric are:1. The metric must be clearly defined2. The metric must be multi-valued3. The metric must reflect the purpose of the map4. The metric must balance coverage and detail5. The metric must be applicable during the construction of the map172.5. BENCHMARKINGBenchmarking the outcome of a SLAM algorithm can largely be divided into twomain categories, trajectory-based and environment-based. Both require a highly accurateground truth in order to be used. Trajectory-based approaches require time-stamped poseinformation as the ground truth, while environment-based approaches require a groundtruth related to the structure of the map. Having access to either of these as referencedata defeats the purpose of the SLAM. Some frameworks exist for evaluating 2D mapquality but still require manual work by humans for each map [58]. Handa et al. [59]created a benchmarking procedure that aligned the resultant map to a 3D mesh of theenvironment and used the distance between the points and the mesh as their metric.The dense SLAM algorithm ElasticFusion [31] used this same procedure to evaluate thequality of their reconstructions.A third benchmarking approach could be the complexity and resources required tocomplete the SLAM process. These type of metrics can be used within the SlamBenchbenchmarking framework [60–62], provided your input data is compatible with their API.SlamBench does not currently support SLAM through ROS.Using fiducial landmarks has been used in SLAM benchmarking within the RoboCupRescue competition [57, 63–65]. The competition used barrels within the testing envi-ronment as their landmarks; however, highly accurate reference scans of the environmentwere required for this approach. Two-thirds of the barrel must be within the scan inorder to be a successful detection. Schwertfeger expanded the method to 3D [64] by de-tecting the barrels within 2D slices of 3D maps. A method detecting the barrels directlywithin the 3D maps was also introduced as a proof of concept; however, a lack of softwaresupport made it impossible to validate the accuracy of the extractions. This method is182.5. BENCHMARKINGthe closest in implementation to the suggested method within this thesis but differs in afew key areas, namely the need of the reference scans of the environment, and the use ofglobal instead of relative positioning as the metric.It should be noted that using fiducials as a benchmarking approach is different thanusing fiducials within the SLAM itself. Using fiducials in the SLAM process (e.g. AprilT-ags within TagSLAM [66] and QR codes in [67]) exploit the knowledge of the fiducials inorder to construct the maps. The most common fiducials for SLAM are 2D tags that arevisually distinguishable. Therefore, it is impossible to use these fiducials within algorithmsthat do not use cameras.2.5.1 SLAM DatasetsIn the area of mobile robotics, public datasets provide standardized data to comparedifferent algorithms. This section details a few commonly used data sets in both indoorand outdoor scenarios.It is a known issue that SLAM approaches are often benchmarked on datasets thatonly include LiDAR or vision datasets but not both [25], making it difficult to objectivelycompare certain algorithms. Furthermore, datasets commonly used for benchmarking lackground truth structure data. Localization ground truth data is easier to provide outsideof simulation. These datasets can be broken into two categories, indoor and outdoor. Asummary of some available datasets are given in Table 2.1. Some of these datasets aremore aimed towards object recognition (e.g. ScanNet [68]) but as long as multiple views(i.e. sensor poses) are available and annotated, they can be used in map evaluation. Notethat laserscan denotes a 2D LiDAR sensor whereas LiDAR is a full 3D sensor.192.5. BENCHMARKINGDataset Sensor Data Scene Type SizeReal-WorldScanNet [68] RGB-D, Camera poses Indoor scene 2.5m view, 1500 scenesMalaga Urban [69] RGB-D, Laserscan Urban outdoor scene 36.8km trajectoryKAIST Urban [70] RGB-D, LiDAR, Laserscan Urban outdoor scene 19 scenes, 165.9km trajectory (total)Matterport3D [71] RGB-D Indoor scene 194k images, 90 scenesArgoverse [72] LiDAR, RGB-D, RGB Urban street scene 113 scenesKITTI [68] LiDAR, RGB Urban street scene 15k imagesnuScenes [73] LiDAR, RGB Urban street scene 1.4m images, 1000 scenesTUM RGB-D [74] RGB-D Indoor scene 15 scenesEuRoC MAV [75] RGB-D Indoor scene 11 scenesSyntheticInteriorNet [76] RGB-D Indoor scene views 20m imagesSceneNet [77] RGB-D Indoor scene views 5m imagesUnrealCV [78] RGB-D Indoor scene N/ATable 2.1: Summary of Datasets Usable for SLAM.Outdoor DatasetsThe KITTI dataset [68] is a popular outdoor dataset used in autonomous vehicleresearch containing high-resolution RGB and grayscale images and LiDAR data from amoving car. GPS localization data can be used with an included tool to compute thelocalization accuracy. The Malaga Urban [69] and KAIST Urban [70] datasets are alsocommon for outdoor scenes and autonomous vehicle research. Both use GPS localizationas ground truth but neither provide a comparison tool like the KITTI dataset does. TheKAIST dataset was captured using two 2D and two 3D LiDARs alongside a front-facingstereo camera to collect a wide range of data and also includes a full map reconstruc-tion in point cloud format. Argoverse [72] and nuScenes [73] are other commonly usedautonomous driving datasets that provide a large number of annotated scenes for objecttracking.202.5. BENCHMARKINGIndoor DatasetsIndoor datasets include the TUM RGB-D SLAM dataset and benchmark [74] whichincludes color and depth images from a Microsoft Kinect sensor. The EuRoC MAV [75]dataset contains stereo images and synchronized IMU measurements collected from amicro-aerial vehicle. Notably, it also contains accurate structure ground truth in additionto motion. This structure data was collected via an external tracking system. TheScanNet [68] dataset is primarily a object detection and labelling dataset but since italso contains poses for the camera, it could theoretically be used to benchmark a vSLAMtrajectory.Simulated DataIn absence of real-world datasets, simulations can be used to construct ground truthdata. Gazebo and CoppeliaSim are examples of simulators that can be used. A downsideto using simulation is that visual elements are no longer photo realistic. Habitat [79, 80] isa photo realistic simulator used in AI research that was recently used to develop a bench-mark for vSLAM algorithms entitled MAOMaps [81]. A unique feature of MAOMaps ismultiple trajectories, allowing for map-merging algorithm evaluation. At the time of thisthesis, the available source code for MAOMaps failed to compile and could not be usedas a comparison metric. Synthetic datasets such as InteriorNet [76], SceneNet [77], andUnrealCV [78] can be used as near photo-realistic alternatives to simulations, but withreduced customizability.212.5. BENCHMARKINGGround Truth AvailabilityOf the datasets mentioned, only ScanNet [68], MatterPort3D [71], and The EuRoCMAV [75] include realized structural ground truth data.UnrealCV [78] uses Unreal Engine to create artificial environments. It is theoreticallypossible to extract a structural ground truth using this approach. InteriorNet [76] isanother simulated dataset that could offer ground truth structural data through theirofficial tool ViSim. SceneNet [77] includes trajectory and semantic labeling of objects asits ground truth. Structural information is not included.Autonomous vehicle datasets often include ground truth data for trajectory or objecttracking, but structural data is not common. Accurately mapping the topography ofenvironments is not a primary focus for these datasets. Argoverse [72] supplies a widevariety of ground truths for object tracking, including lane mapping, but does not includeground truth structural data about the surrounding buildings. Similarly, KITTI [68]includes a trajectory ground truth but no structural data. Malaga Urban [69] providesinertial data but does not include a ground truth. KAIST Urban [70] provides a baselinepose generated using ICP but does not recommend using it as a ground truth. It includesa point cloud reconstruction created using this baseline but this is meant as more of ademo than a ground truth.The TUM RGB-D [74] includes trajectory ground truth data extracted from a high-accuracy motion tracking system; however, structural data is not included.222.5. BENCHMARKINGFigure 2.1: An example of how mapping quality can be interpreted given different metrics.2.5.2 Benchmarking metricsBenchmarking metrics largely depend on the application of the mapping. Relativelysmall errors in one system could be catastrophic in others. For example, Figure 2.1represents a map where one cell has been incorrectly labeled. If correctly mapping theenvironment was the objective, this is a relatively small error; however, if determiningnavigatible areas was the objective, this error is catastrophic. Navigating from the toparea to the bottom would be considered impossible. Other requirements might be real-time operation, meaning computational performance would be a relevant benchmark.Another criterion could be navigation without crashing into obstacles, which may or maynot care about the resultant map.While middleware like ROS allows for standardized inputs and outputs, it comes at232.6. LIMITATIONS OF SLAMthe cost of putting an additional software barrier between the SLAM and its sensors.Certain algorithms like the popular ElasticFusion [31] cannot use ROS because of theirreliance on tight coupling with the GPU. Likewise, the SlamBench framework [60–62] hasyet to implement compatibility with ROS.2.6 Limitations of SLAMSLAM continues to be an ongoing area of research as there is not a single solution thatfits every scenario. Data association is a known issue within SLAM [82]. If the landmarkscannot be identified uniquely (e.g. repeated landmarks within a space) or are too sparse(e.g. vSLAM in a large white-walled room), correctly finding a SLAM hypothesis canbe incredibly difficult. The trajectory taken during mapping directly affects the SLAMresult. Longer trajectories result in increased measurement drift and can result in moredifficult loops to close, effecting inconsistencies in the reconstructed map.SLAM is an non-linear optimization problem influenced by the sensors and the environ-ments that serve as inputs for the process. Sensor degradation and complex environmentscan have a negative effect on the result. Also, SLAM is typically used within static envi-ronments. Adapting the SLAM problem to handle dynamic environments requires objecttracking integration [83].242.7. SUMMARY2.7 SummarySLAM has been an ongoing research field for a decades, resulting in a wide variety ofimplementations. Algorithms can be classified in multiple ways including based on thesensors used, the types of maps produced, or the way they handle redundant informa-tion. Extensions of SLAM such as active SLAM allow for more intelligent mapping ofenvironments. Benchmarking is a rather limited field that is not applicable in real-worldsituations using established methods. Frameworks exist for testing multiple algorithmsbut can be too restrictive to use in practice.25Chapter 3Methodology3.1 Software and Hardware StackThe Jackal unmanned ground vehicle (UGV) from Clearpath Robotics was chosen forits easy integration with ROS. Intel RealSense d435i cameras were chosen for their lowcost and higher resolution. RTAB-Map has the ability to use multiple cameras but band-width and storage limitations made it infeasible for longer testing. The VLP-32MR fromVelodyne was chosen to allow for higher resolution LiDAR scans that can be downsam-pled if processing became an issue. All exploration was controlled manually to removeany issues that could result from path planning algorithms failing. The Jackal with thecameras and LiDAR mounted are shown below in Figure 3.1.Using ROS standardizes sensor input and mapping output across each algorithm used,greatly simplifying the benchmarking procedure. The transforms between sensor framesare modeled once and then used for all validation in both simulation and the real world.Gazebo was chosen as the simulation platform due to its high compatibility with ROSand the Jackal. A Gazebo render of the Jackal with a camera and LiDAR attached areshown below in Figure 3.2.A configuration matching the specifications of the VLP-32MR was created for an263.1. SOFTWARE AND HARDWARE STACKFigure 3.1: Jackal UGV equipped with d435i cameras and VLP-32MR LiDAR.existing Velodyne simulator plugin [84] for Gazebo. It should be noted that this pluginwas designed for earlier versions of the Velodyne Puck line of LiDARs that have equalspacing between each of the rings on the LiDAR; however, higher end models, includingthe VLP-32MR, use non-linear spacing. Because of this, there are slight deviations inwhere the returned points are located within the simulation. Modifying the simulator touse the non-linear spacing can be considered in future work. Certain algorithms, suchas the original LOAM [34], rely on this linear spacing and would not function correctlywith non-linear spacing. Intel created a Gazebo plugin for their RealSense cameras [85];however, the plugin is not maintained and no longer works. It was also never designedto work with the more recent d435i cameras. Forks of the plugin [86, 87] were modifiedto add support for the d435i cameras used in this project. Gazebo lacks an infrared (IR)camera option that could be added to the d435i. The IR streams within the simulation273.1. SOFTWARE AND HARDWARE STACKFigure 3.2: A simulated Jackal equipped with a VLP-32MR and single d435i camera.are actually just grayscale images with Gaussian noise added to make it look similar toimages with the IR projection emitted from the camera. The depth was simulated using anative depth plugin within Gazebo, rather than using stereo imaging from the IR streamsas would be done in the real world.The successor to Gazebo, now called Ignition, allows for more advanced physics andrendering engines. It is designed to be a highly modular system of plugins with an“only pay for what you use” approach. Gazebo, the rendering engine, and the physicsengine are now separate plugins underneath the Ignition umbrella. Unfortunately, poorcompatibility with the robot platform and weak documentation made it illsuited for thisapplication. Simulated sensor plugins (i.e. Velodyne and RealSense cameras) would haveto be rewritten from scratch in order to be compatible.Similarly, ROS is also undergoing an upgrade to ROS 2. Compatibility between hard-ware and software dictated the versions of ROS and Gazebo that had to be used. Table3.3 shows a visual of how ROS is used to abstract the software components. Red indi-283.1. SOFTWARE AND HARDWARE STACKFigure 3.3: Software abstraction using ROS.cates a component that was not implemented. Purple boxes signify a component that wasmodified for use in this project, while white boxes represent unmodified components. TheGazebo portion of the software was designed to be interchangeable with the real-worldUGV. ROS 2 can communicate with ROS through the use of a bridge, but this addsunneeded complexity. Future work could include revisiting this principle using the moreadvanced features of Ignition and ROS 2 when they become more refined.Four commonly used SLAM algorithms were chosen and implemented on the Jackal.These algorithms consist of LeGO-LOAM [35], BLAM [32], HDL Graph SLAM [36],and RTAB-Map [24–26]. RTAB-Map is notably the only one that makes use of theRGB-D data provided by the RealSense cameras. These algorithms were chosen basedon their popularity in the open-source community and their compatibility with Velodyne293.1. SOFTWARE AND HARDWARE STACKLiDARs. Purely visual-based algorithms were excluded for this reason. Two RTAB-Mapconfigurations have been configured within this project. The first builds the map usinga 2D laser scan extracted from a lower ring of the LiDAR and the RGB-D data in thefinal map from the RealSense camera. The second builds the map from the full 3Dpoint cloud while only using the RGB-D data for loop closure. These are referred to as2D LiDAR and 3D LiDAR configurations respectively throughout the remainder of thethesis. The first benefits from having colour data provided by the RGB stream, whilethe second has a greater point cloud density due to the high resolution of the LiDAR.RTAB-Map also outputs a 2D projection costmap that can be used for path planning inthe navigation stack. The other algorithms require a costmap to be manually createdusing a package like Octomap Server [6]. Optimisation of these algorithms could be theentire subject of another thesis (e.g. RTAB-Map has 342 configurable parameters notincluding sensor configurations) and as such is not focused on within these experiments.General recommendations using sample configurations are given but other algorithms mayperform better with additional tuning.BLAM is only included due to its popularity within the ROS community and its easeof implementation. There is little to no documentation and the project has not beenupdated in several years. As such, it has the potential to break when other packages areupdated and is not suited for projects that require consistency. LeGO-LOAM is a fork ofLOAM [34] that has been optimized for use on ground vehicles. HDL Graph SLAM is a3D LiDAR based technique that uses normal distribution transforms (NDT) [49] for scanmatching and loop closure.303.2. EVALUATION METRICS3.2 Evaluation MetricsThere were multiple metrics involved in the experiments. A localization metric andmapping quality metric found in literature are compared to the mapping quality metricdeveloped within this thesis. The evaluation metric proposed in this research was the rel-ative distance error between features of known objects. A benefit to this approach is anaccurate ground truth of the environment or robot pose is not needed for the benchmark-ing. These landmarks can be artificially placed within the environment to find a suitablealgorithm during a test run and removed for the actual run. Alternatively, objects withinthe environment with known dimensions could be detected within the finalized map toprevent the need for additional scans. The landmarks used in this research were easy toextract and measure relative to each other.3.2.1 Localization QualityThe localization quality was measured by comparing the estimated pose to the groundtruth pose published within simulation. The absolute pose error (APE) was used toquantify the quality of the SLAM process. This metric was omitted for the real-worldtesting since a ground truth pose was not available.The APE metric is defined as follows:APE(t) =∥∥∥P⃗e,t − P⃗a,t∥∥∥2(3.1)where P⃗e,t represents the pose estimated by an algorithm at a time t and P⃗a,t representsthe actual (i.e. ground truth) pose at time t in vector form. Since APE uses the absolute313.2. EVALUATION METRICSdistance between the poses, the trajectories need to be properly aligned with the groundtruth trajectory. The method proposed by Umeyama [88] was used to minimize theleast-squares error between the two trajectories.3.2.2 Mapping QualityThe mapping quality was estimated in two ways. The first was using the methodintroduced by Handa et al. [59], where the final map was aligned to the mesh of the mapin a two stage process before computing the cloud to mesh (C2M) distance of each point.The alignment involved roughly aligning the cloud to the mesh by selecting correspondingpoint pairs. Figure 3.4 shows the results of aligning estimated points (i.e. those denotedwith an A) to a set of corresponding reference points (i.e. those denoted with an R). Thesecond stage is registration involving the iterative closest point (ICP) algorithm [40, 41].ICP works to minimize the root mean squared distance (RMSD) between two point clouds.As the reference is a mesh, CloudCompare samples the mesh in the background for thefine registration. The rough alignment uses the actual distance from the triangle on themesh that was selected.The second metric was the landmark extraction and processing approach which isa main contribution of this thesis. This approach is similar to the 2D fiducial methodimplemented within [63] and [57] and later theoretically extended into 3D in [64]; however,they used ground truth maps merged from multiple high resolution scans from a Faroscanner. The 3D map also had some details missing due to occlusions in the scans. Theproposed method only relies on models of the objects and therefore does not suffer fromthe occlusion issue. The models of the objects can be considered as a submap within the323.2. EVALUATION METRICSFigure 3.4: Rough alignment of a point cloud to a mesh, before (left) and after (right).larger environment.So far, two features have been implemented in the software: a manually placed Carte-sian point (i.e. BasicFeature) and a moment of inertia feature (i.e. MomentOfInertiaFea-ture). The moments of inertia are extracted by calculating the covariance matrix of thepoint cloud and extracting the eigenvectors and eigenvalues to find the principal axes.The principal axis with the highest and lowest moments of inertia are considered to bethe X-axis and Z-axis respectively.The moments of inertia are extracted from a given point cloud (i.e. the object ex-tracted from the finalized map) but it should be noted that the center of mass is thepoint used for comparison. The axes of rotation are stored in the feature but not used.Orientation of the landmarks was also ignored. Additional features can be implementedprovided they adhere to the point types provided by the PCL library. The comparisonis done by finding the Cartesian displacement. Other PCL point types allow for more333.2. EVALUATION METRICSdata like colour and intensity information; however, current comparison methods in thesoftware do not consider this information. The total number of comparisons derived usingthis method is:nC2 =22(n− 2)! (3.2)where n is the total number of features extracted from the known landmarks.The maps are preprocessed to be usable with this metric. The point clouds of theobjects are manually segmented and extracted as a group from the finalized map. Thesegmented cloud is run through a passthrough filter to remove the ground (i.e. thresh-olding points beyond a given Z location) before being downsampled using a voxel gridfilter with a leaf size of 1cm. This approximates the points within each voxel (i.e. 3D boxwith dimensions of 1cm) as the voxel’s centroid. Then the landmarks are extracted usinga process known as Euclidean cluster extraction [89].The passthrough filter used in this stage is very rough. Points that belong to theobjects are likely removed during this stage but since the objects used in this researchwere all symmetrical, this would not effect the end result. The centroid would move eitherup or down, but the relative distance between would each stay the same. This is providedthat the filtering was conducted on a level map. The maps were coarsely aligned to theground truth models using method shown in Figure 3.4 to make the map more inline withthe XY-plane. The ground truth representation does not need a passthrough filter as theground can be completely removed from the objects without it.Euclidean cluster extraction works by dividing the space into a 3D grid with fixed343.2. EVALUATION METRICSwidth boxes (i.e. an octree structure). If the following equation is satisfied:min ∥pi − pj∥2 ≥ dth (3.3)where dth is the minimum distance threshold, and sets of points pi and pj both existin the same space, then pi and pj belong to two separate clusters, Oi and Oj. Algorithm1 outlines the steps to extract the largest cluster from a point cloud. This process iscompleted until all the landmarks are extracted from the cloud. Any remaining pointsare considered outliers and discarded.Algorithm 1 Euclidean Cluster Extraction Algorithm1: procedure EuclideanCluster(X)2: Input X: Input point cloud3: Output Q: Largest Euclidean cluster in X4: P ← kdTreeRepresentation(X)5: for all dopi ∈ P6: Q← Empty Queue7: Q← Q+ pi8: for all pi ∈ Q do9: P ki ← Neighbours of pi within radius dth10: for all pki ∈ P ki do11: if pki has not been processed then12: Q← Q+ pik13: end if14: end for15: return Q16: end for17: end for18: end procedureThe full cluster extraction process is outlined in Algorithm 2. Implementing 3D objectdetection and segmentation could remove the need for this manual extraction. Another353.2. EVALUATION METRICSapproach could be scan matching the reference scan to the completed map to automati-cally find the object cluster.Algorithm 2 Landmark Extraction Algorithm1: procedure Extract(X)2: Input X: Input point cloud3: if X contains Landmarks then4: L← segmentLandmarks(X)5: L← filterOutGround(L)6: end if7: while L contains Landmarks do8: euclideanCluster(L)9: end while10: end procedureThe objects can be easily simulated in Gazebo using Unified Robot Description Format(URDF) files. URDF files can be converted into a single DAE object using ROS andconverted to OBJ objects using Blender that can then be sampled into a point cloud usingthe CloudCompare tool and saved as a PCD file. Lastly this PCD file can be processedand have features extracted using the tools created with the Point Cloud Library (PCL).Figure 3.6 shows computer boxes that were placed as landmarks in the laboratory, modeledin Gazebo, sampled using CloudCompare at a density of 1000 points/m2 of surface area,and then had their centroids located. PCL is fully integrated with ROS and therefore, thisalgorithm could be adapted to run during the mapping to measure the accuracy changesover time. Alternatively, the point clouds could be recorded to a rosbag and extractedafter the fact.To model the boxes in URDF files, each box had its corners labeled and a series ofmeasurements were taken between the corners of the boxes. A single box was chosen as363.2. EVALUATION METRICSC1A1 B1D1C2A2B2 D2C3A3 B3D3Figure 3.5: Drawing of the boxes used to derive the URDF representation.reference and used to construct the other two relative to it. Figure 3.5 shows the namingscheme of each of the corners. The box labeled in red was chosen as the reference box,with a global origin located at the corner labeled C1.Using the measured dimensions, the boxes were constructed using the following pro-cess:Let θ1 be the angle from C1 to A2:θ1 = arccos( |C1A2|2 + |C1D1|2 − |D1A2|22|C1A2||C1D1|)(3.4)373.2. EVALUATION METRICSLet θ2 be the angle from D1 to A2:θ2 = arccos( |C1D1|2 + |D1A2|2 − |C1A2|22|C1D1||D1A2|)(3.5)Let θ3 be the angle from C1 to B2:θ3 = arccos( |C1B2|2 + |C1D1|2 − |D1B2|22|C1B2||C1D1|)(3.6)Let θ4 be the angle from D1 to B2:θ4 = arccos( |C1D1|2 + |D1B2|2 − |C1B2|22|C1D1||D1B2|)(3.7)Calculate the vector components using both angles and take the average of the twomeasurements to find the C1A2. If more measurements were used to determine the ori-entation of the landmarks, they could be added to this equation to give a more accurateresult. This will be the position of the origin for the dependent landmark:C1A2x =|C1A2| cos θ1 + (|C1D1|+ |D1A2| cos (π − θ2))2(3.8)C1A2y =|C1A2| sin θ1 + |D1A2| sin (π − θ2)2(3.9)Do a similar process to find to find the components of C1B2:C1B2x =|C1B2| cos θ3 + (|C1D1|+ |D1B2| cos (π − θ4))2(3.10)383.2. EVALUATION METRICSFigure 3.6: Boxes used as landmarks for SLAM (left), boxes modeled in URDF (center),modeled boxes sampled with features extracted (right).C1B2y =|C1B2| sin θ3 + |C1B2| sin θ42(3.11)Calculate the angle from A2 to B2 to figure out how much to rotate the dependentlandmark by.α = arccos(C1B2x − C1A2xA2B2)(3.12)The dependent box is then constructed using the measured length, width, and height.The boxes are all roughly the same size but vary slightly from corner to corner. Individualbox dimensions were used to more accurately reflect the real world.The benchmarking is done offline due to multiple reasons. First, point cloud featureextraction requires a significant amount of computational effort. Online 3D mappingalso requires expensive computations, as existing points in the map must be transformedduring each update. As the map grows, it quickly becomes impossible to perform 3DSLAM at high resolutions. Also, each extracted cluster must have the correct id number393.2. EVALUATION METRICSto properly match against the ground truth meshes. The extraction identifies each clusterin ascending order starting with largest quantity of extracted points. The extraction isadjusted manually to ensure this is always the case. The manual effort and computationalrequirements of this metric make real-time mapping and evaluation not practical.A notable downside to this approach is the artificial altering of the test environment.Adding more potential landmarks to the environment will directly influence the mapping,although this is hard to quantify. Gazebo follows a publisher-subscriber model that doesnot allow the exact same trajectory to be repeated in a different environment. Ideally,the landmarks used would be existing objects in the environment instead of artificiallyadded.This approach was designed with the systematic approach Lee [56] and Schwert-feger [57] both used in their respective theses. Here is the justification for satisfyingthe following criteria:1. The metric must be clearly defined: The metric is the distance between thecentroids of the landmarks.2. The metric must be multi-valued: Distances are continuous values that can bedirectly compared.3. The metric must reflect the purpose of the map: The purpose of the map isaccurate placement of objects within the environment.4. The metric must balance coverage and detail: A satisfactory number of land-marks are chosen to properly cover the area.403.3. PROPOSED FRAMEWORK IMPLEMENTATION5. The metric must be applicable during the construction of the map: Aslong as the objects are in the map, the metric can be calcuated.Criterion 2 is in place to objectively compare multiple maps. If the clusters cannotbe extracted, the map cannot be directly compared against another. The metric ceasesto be multi-valued. That is to say, the map is simply either “good” or “bad” rather thana measure of how good or bad. Maps that do not meet this requirement would likely benoticeably poor based on simple inspection. Criterion 4 is the hardest to satisfy. Appro-priate coverage and detail will vary depending on the task and is difficult to qualitativelydefine.3.3 Proposed Framework ImplementationThe framework introduced within this thesis was built on top of PCL and is designedto be fully extendable. A simplified unified modeling language (UML) diagram is given inFigure 3.7. Items above the line for an entity are attributes (i.e. instance variables specificto that object), while those below are methods (i.e. functions that perform a specific taskgiven the type of object). A FeatureMatch object is constructed from two or moreFeatureSet objects, one ground truth and at least one estimated set. Each FeatureSetis of the same length and contains at least two Feature objects to be compared relativelyto each other. The comparisons are then compared to corresponding Feature comparisonsin another FeatureSet. This process is detailed in Algorithm 3, where the comparisonmetric is the Euclidean distance between the two features. All objects are constructedusing templates (i.e. PointT) to be evaluated at compile time. This ensures the framework413.4. VERIFICATION METHODSFigure 3.7: UML diagram for the proposed framework.is easily extendable.Each Feature is either a manually placed Cartesian point (i.e. a BasicFeature)or a more specialized one (i.e. MomentOfInertiaFeature). Each feature inherits theattributes and methods from the base Feature class. For each unique pair of Featureobjects in a FeatureSet, a comparison between them is calculated. For the purpose ofvalidation, this was the Euclidean distance between each Feature.3.4 Verification MethodsThis section outlines the trials conducted to validate the benchmarking procedure.Scenarios were created in both a real-world laboratory setting using a Jackal UGV andin simulation using a Gazebo simulated Jackal UGV.423.4. VERIFICATION METHODSAlgorithm 3 Feature Comparison Algorithm1: procedure Compute(X)2: Input X: FeatureMatch object3: X.groundTruth.displacements← computeDistances(X.groundTruth)4: for all FeatureSetE ∈ estimated do5: E.displacements← computeDistances(E)6: X.errorDisplacements← E.displacements−X.groundTruth.displacements7: end for8: end procedure9: procedure ComputeDistances(X)10: Input X FeatureSet object11: Output D Displacement vector12: for all Feature pair {F1, F2} ∈ X do13: D ← euclideanDisplacement(F1, F2)14: end for15: end procedure3.4.1 SimulationGazebo is the default simulation platform for ROS and Clearpath has integrated sup-port to simulate the Jackal. A successor to Gazebo, named Ignition is currently availablebut incomplete documentation and poor compatibility with the Jackal, made it illsuitedfor this project. The VLP-32MR was modeled using a custom profile for the Velodynesimulator plugin [84] while the RealSense camera used a custom Gazebo plugin basedon [86] and [87]. There are a couple notable differences between the simulated behaviourand the real-world sensors. First, the Velodyne simulator is based on the VLP-16 sensorand assumes that the lines (i.e. the horizontal ring of lasers emitted) of the LiDAR are allevenly spaced in the vertical direction; however, the VLP-32MR has non-linear spacing.Second, the simulated RealSense utilizes the Kinect depth plugin from Gazebo ratherthan calculating it from the stereo infrared images since Gazebo lacks an infrared sensor433.4. VERIFICATION METHODSPillarLocation (m)X Y ZTop left -2.17751 1.63793 2.06353Top right 2.3666 1.63642 2.06741Bottom left 2.36653 -1.55649 2.0537Bottom right -2.17789 -1.55716 2.06723Table 3.1: Locations of the centres of each support pillar.plugin. The exact pose was also published using Gazebo for better visualization and touse in localization benchmarks. All data was recorded into rosbags and played back withthe different algorithms running on top of them.EnvironmentThe environment used in this experiment was a small floating roof tank based on alarger model provided by Rosen Group. The model of the tank is shown in Figure 3.8.The ground truth landmarks were the four support pillars placed throughout the tank.The extracted centroids of each support pillar are shown in Table 3.1. The environmenthas has an inside diameter of 7m and contains the four support pillars, a heating coil,pipe, and a drainage hole in the centre. The Jackal could fit underneath the pipe butnot under the heating coil. The support pillars are also not completely symmetric withrespect to the center of the tank and there is not enough room for the Jackal to maneuveraround the pillars on the right. The floor of the tank consists of several overlapping plates,resulting in a mostly flat but slightly uneven terrain.443.4. VERIFICATION METHODSFigure 3.8: Model of the tank used within the simulated experiments.3.4.2 Real-World TestingTo verify the results in a real-world scenario, the boxes were placed in the AdvancedControl and Intelligent Systems (ACIS) laboratory at the Univerisity of British Columbia(UBC)’s Okanagan campus. The dimensions of the boxes were used to make a URDFrepresentation that could be converted to a point cloud and processed as a ground truthrepresentation. The extracted centroids of each box are shown in Table 3.2. The environ-ment consists of multiple desks, chairs and cabinets that look very similar from the UGV’spoint of view. To help minimize potential localization errors, the chairs were placed in asingle corner of the laboratory.453.4. VERIFICATION METHODSBoxLocation (m)X Y Z1 -3.58599 4.39562 0.3158472 -5.20697 4.07582 0.3136023 -4.29314 3.08009 0.314073Table 3.2: Locations of the laboratory landmarks.46Chapter 4Validation Results and DiscussionIn this chapter, the results of each set of validation trials are discussed. A more detailedanalysis can be conducted in simulation due to more readily available localization andstructural ground truth data. The existing metrics from literature were used to validatethe results of the proposed method.The real-world trials show that the proposed method can be expanded to situationsoutside of simulation. Prerecorded data from Gazebo was played back while the SLAMalgorithms were each run separately. The data used for the SLAM algorithms includedRGB-D, LiDAR, laserscan (i.e. a single ring extracted from the LiDAR readings), IMU,and filtered odometry data (i.e. fused from the wheel odometry and IMU data using anextended Kalman filter). RTAB-Map used the filtered odometry as one of its constraints,while HDL Graph SLAM and LeGO-LOAM both used the unfiltered IMU data.4.1 Simulation ResultsThe BLAM algorithm crashed and failed to save a completed map trajectory despitemultiple attempts. As a result, it has been excluded from comparison for this experiment.In total, there were four different configurations ran within the simulations: RTAB-Map474.1. SIMULATION RESULTSAlgorithmMatchingTimestampsError MetricsSD RMSE Min Median Mean MaxHDL Graph SLAM 2093 0.509 0.989 0.139 0.859 0.847 10.116LeGO-LOAM 501 0.857 2.690 0.556 2.454 2.550 5.241RTAB-Map (3D LiDAR) 2090 0.113 0.250 0.00512 0.221 0.223 0.659RTAB-Map (2D LiDAR) 2092 0.059 0.100 0.00138 0.0809 0.0363 0.398Table 4.1: APE statistics for the tank simulation.(2D LiDAR), RTAB-Map (3D LiDAR), HDL Graph SLAM, and LeGO-LOAM. Eachwere evaluated using the existing APE and C2M metrics, and the proposed frameworkusing the distance between the centroids of the Euclidean clusters.4.1.1 Localization Quality (APE Metric)The localization quality was evaluated in part using the Evo Python package [90].Each trajectory was aligned to the ground truth poses published within Gazebo using theUmeyama method [88]. The scale of each trajectory was left unaltered. The poses neededto be aligned according to their corresponding timestamp in order for the APE metric tobe used. Figure 4.1 shows the estimated trajectories projected in the XY-plane overlaidon top of the ground truth trajectory, while Figure 4.2 shows the Cartesian componentsof each trajectory over time. Lastly, Figures 4.3 and 4.4 illustrate the APE distributionfor an easier visual comparison. The APE statistics are also summarized in Table 4.1along with the number of corresponding timestamps for each algorithm.Clearly, the 2D LiDAR configuration of RTAB-Map produced the best trajectory re-sults and LeGO-LOAM performed the worst. It is possible that LeGO-LOAM’s poor484.1. SIMULATION RESULTSperformance could be the result of the rate at which it updates its position, since ithad the fewest corresponding timestamps. The literature for LeGO-LOAM [35] also fo-cused primarily on applications in large outdoor environments (e.g. autonomous vehicledatasets). Evaluating trajectories in small enclosed spaces may be a limitation of thealgorithm itself.Even though the tank floor looks flat, it actually consists of many small overlappingplates. Figure 4.2 shows that LeGO-LOAM was the only algorithm that seemed to haveissue with the uneven terrain. Forcing the algorithm to update more than the defaultrate of once every 0.3 seconds may lead to some improvement.Statistically, HDL Graph SLAM had the second worst performance and some remark-ably high APE early in its trajectory. It appears this was largely due to a large correctionin its pose estimate at the beginning of the trajectory. It otherwise performed reasonablywell.Based on the results of the APE metric, RTAB-Map would be the best algorithm toobtain an accurate robot trajectory using either the 2D or 3D LiDAR configuration. The2D configuration did perform slightly better, and would result in a 3D map with RGBdata.4.1.2 Mapping QualityThe mapping quality was evaluated in two ways, the first was using the C2M distancescomputed using the CloudCompare software, and the second using the proposed metric.The proposed metric could not be evaluated in some cases if the landmark clusters couldnot be cleanly extracted. This was an issue with LeGO-LOAM where the point cloud494.1. SIMULATION RESULTSFigure 4.1: Estimated trajectories aligned to the ground truth for the tank simulation.was too sparse to discern any landmarks within the tank. A minimum of 100 pointswithin a cluster were required for it to be extracted. Figure 4.5 shows the resultantmaps from each of the algorithms with meshes of the support posts superimposed. It isslightly difficult to see but LeGO-LOAM produced multiple copies of the tank at differentrotations. It localized the UGV within these duplicate structures throughout the mappingprocess, which was also reflected in the trajectory comparisons (i.e. the Z-axis trajectoryin Figure 4.2).504.1. SIMULATION RESULTSFigure 4.2: Components of the estimated trajectories compared to the ground truth forthe tank simulation.Cloud to Mesh (C2M) MetricFigures 4.6, 4.7, 4.8, and 4.9 each show a histogram of the resulting C2M values for thetank maps for RTAB-Map (2D LiDAR), RTAB-Map (3D LiDAR), HDL Graph SLAM,and LeGO-LOAM respectively. The data is summarized as a box plot in Figure 4.10 withsummary statistics in both Figure 4.11 and in Table 4.2.Once again, LeGO-LOAM performed the worst overall. Figures 4.6, 4.7, 4.8 all showhigh concentrations of points around zero for the other algorithms, whereas Figure 4.9514.1. SIMULATION RESULTS0.02.55.07.510.0HDL Graph SLAM LeGO−LOAM RTAB−Map (2D LiDAR) RTAB−Map (3D LiDAR)AlgorithmAPE (m)TankTrajectory AccuracyFigure 4.3: Comparison of trajectory accuracy for the tank simulation.shows a large percentage of points are at least 0.5m away from the mesh. It also had farfewer points available for comparison relative to the other algorithms (e.g. 2498 pointscompared to 96754 for HDL Graph SLAM).Proposed MetricFor the proposed metric to be applicable, the cluster of objects needs to be extractablefrom the map. The tank simulation resulted in usable maps from both RTAB-Map config-urations and HDL Graph SLAM. The map produced by LeGO-LOAM was too sparse forthe clusters to be properly extracted. The sparsity of the point cloud reveals a criteria forthe proposed method to be applied: a map needs to be dense enough to support the useof the metric. The extracted centroids from each map for each pillar are shown below inTable 4.3. Each of these produced six relative displacements that were evaluated against524.2. REAL-WORLD RESULTSMaxMeanMedianMinRMSESD0.0 2.5 5.0 7.5 10.0Value (m)MetricHDL Graph SLAMLeGO−LOAMRTAB−Map (2D LiDAR)RTAB−Map (3D LiDAR)TankEstimated Trajectory StatisticsFigure 4.4: Trajectory APE statistics for the tank simulation.the ground truth displacements calculated using the values in Table 3.1. The resultingerror displacements are shown in Table 4.4. Statistics about the difference between thesemeasurements are summarized in Table 4.5.All three algorithms resulted in successes given the proposed procedure but RTAB-Map(3D LiDAR) was clearly the best option, only falling behind in standard deviation andmax distance metrics to HDL Graph SLAM. These results were in line with what wasestablished using APE and C2M.4.2 Real-World ResultsThe real-world trials only served to validate the proposed metric outside of simulateddata. The laboratory could provide neither ground truth trajectory nor structural data.534.2. REAL-WORLD RESULTSAlgorithm PointsError MetricsSD Min Median Mean MaxHDL Graph SLAM 95754 0.0894 0 0.0534 0.0741 6.545LeGO-LOAM 2498 0.472 0.000069 0.453 0.563 2.214RTAB-Map (2D LiDAR) 4328330 0.0686 0 0.0185 0.0363 1.240RTAB-Map (3D LiDAR) 1777778 0.0904 0 0.0775 0.0992 1.354Table 4.2: Cloud to mesh (C2M) statistics for the tank simulation.Algorithm PillarLocation (m)X Y ZRTAB-Map (3D LiDAR)Upper left -2.13679 1.61255 1.36214Upper right 2.39693 1.69817 1.7476Lower left 2.38597 -1.56798 1.39744Lower right -2.12706 -1.55478 1.2796RTAB-Map (2D LiDAR)Upper left -2.12872 1.63525 1.87823Upper right 2.34215 1.63814 1.82464Lower left 2.35051 -1.51955 1.86095Lower right -2.18514 -1.52907 1.72955HDL Graph SLAMUpper left -2.18964 1.60244 1.09596Upper right 2.31944 1.66073 1.21779Lower left 2.40161 -1.38887 1.36161Lower right -2.16925 -1.4739 1.10777Table 4.3: Estimated centroids of the tank landmarks.544.2. REAL-WORLD RESULTSAlgorithm Pillar 1 Pillar 2Location (m)X Y Z DistanceRTAB-Map (3D LiDAR)Upper left Upper right 0.01039 -0.08713 -0.38158 0.391539124Upper left Lower left 0.02128 -0.01389 -0.04513 0.051792735Upper left Lower right -0.01011 -0.02776 0.08624 0.091160119Upper right Lower left 0.01089 0.07324 0.33645 0.344501513Upper right Lower right -0.0205 0.05937 0.46782 0.472017584Lower left Lower right -0.03139 -0.01387 0.13137 0.135778444RTAB-Map (2D LiDAR)Upper left Upper right 0.06285 0.08273 0.43905 0.45117544Upper left Lower left 0.04353 -0.02573 0.05258 0.072948956Upper left Lower right 0.06615 -0.00301 0.06614 0.093591571Upper right Lower left -0.01932 -0.10846 -0.38647 0.401865518Upper right Lower right 0.0033 -0.08574 -0.37291 0.382654029Lower left Lower right 0.02262 0.02272 0.01356 0.034810004HDL Graph SLAMUpper left Upper right -0.03821 -0.0554 -0.17542 0.18788651Upper left Lower left -0.11202 -0.16349 -0.28293 0.34543718Upper left Lower right -0.07681 -0.08798 -0.16049 0.198487523Upper right Lower left -0.07381 -0.10809 -0.10751 0.169380531Upper right Lower right -0.0386 -0.03258 0.01493 0.052671826Lower left Lower right 0.03521 0.07551 0.12244 0.148098136Table 4.4: Displacement errors for the tank landmarks.AlgorithmError Metrics (m)Max Min Mean Median SD SE MSE RMSERTAB-Map (3D LiDAR) 0.392 0.0518 0.178 0.0912 0.152 0.164 0.0274 0.165RTAB-Map (2D LiDAR) 0.451 0.0729 0.206 0.0936 0.174 0.218 0.0363 0.190HDL Graph SLAM 0.345 0.188 0.244 0.198 0.0719 0.194 0.0323 0.180Table 4.5: Error statistics for the tank validation using the proposed metric.554.2. REAL-WORLD RESULTSFigure 4.5: Final maps of the tank produced by different SLAM algorithms. Top: HDLGraph SLAM (left), LeGO-LOAM (right); Bottom: RTAB-Map (2D LiDAR) (left),RTAB-Map (3D LiDAR) (right)).Only the extracted centroids of the modeled boxes from Table 3.2 were used to evalu-ate the result. BLAM did not crash when run with real-world data; therefore, all fiveconfigurations could be used in attempts to validate the metric.The mapping resulted in two usable maps with the proposed method, RTAB-Map (3DLiDAR) and BLAM. LeGO-LOAM was once again too sparse to extract the necessaryclusters. The ACIS laboratory is occupied by many desks, chairs, and workstation com-564.2. REAL-WORLD RESULTS01,000,0002,000,000−1.0 −0.5 0.0 0.5 1.0Cloud to Mesh Distance (m)CountRTAB−Map (2D LiDAR)Cloud to Mesh Distances for Tank SimulationFigure 4.6: Cloud to mesh (C2M) distances for RTAB-Map (2D LiDAR).puters. The 2D configuration for RTAB-Map had a tough time discerning one side of thelaboratory from the other and suffered several false loop closures, resulting in an unusablemap. A similar situation happened with HDL Graph SLAM and its loop closure. Bothalgorithms did not result in landmarks that could be cleanly extracted and therefore hadto be excluded from the final comparison.An example of a correctly detected loop closure within RTAB-Map is shown in Fig-ure 4.12. Lines between corresponding features detected between each image have beenshown. The cluster of objects in the corner of the room provided lots of distinguishingfeatures making this an easy location to detect. Figure 4.13 shows a false loop closurewhere the UGV was localized under an incorrect desk. The underside of the desks in thelaboratory have very few distinguishable features and it can be difficult to automaticallydetect the difference between them. Over time, this caused the points in the map to drift,574.3. SUMMARY0100,000200,000300,000−0.5 0.0 0.5 1.0Cloud to Mesh Distance (m)CountRTAB−Map (3D LiDAR)Cloud to Mesh Distances for Tank SimulationFigure 4.7: Cloud to mesh (C2M) distances for RTAB-Map (3D LiDAR).resulting in an unusable map.The map from LeGO-LOAM contained too few points to extract any clusters from.Figure 4.14 shows the mapping results for each algorithm and shows that LeGO-LOAMwas able to at least map the shape of the laboratory. Table 4.8 details the error metricsof the proposed method. By visual inspection, BLAM appears to have mapped more ofthe laboratory; however, RTAB-Map was more accurate in locating the landmarks.4.3 SummaryIn each set of experiments, the best results came from RTAB-Map. Either configura-tion with most likely provide a suitable map and trajectory as long as there are enoughvisual features to prevent false loop closure incidents. BLAM demonstrated its mapping584.3. SUMMARYAlgorithm BoxLocation (m)X Y ZRTAB-Map (3D LiDAR)1 -3.66445 4.44474 0.4572562 -5.13813 4.09405 0.4620383 -4.21104 3.1355 0.451754BLAM1 -3.68236 4.58847 0.4598052 -5.13639 4.08901 0.4981943 -4.07372 3.20644 0.544006Table 4.6: Estimated centroids of the laboratory landmarks.Algorithm Box 1 Box 2Location (m)X Y Z DistanceRTAB-Map (3D LiDAR)1 2 0.1473 -0.03089 0.007027 0.1506680481 3 -0.16056 -0.00629 0.003728 0.16072642 3 0.01326 0.03718 -0.010755 0.040912712BLAM1 2 0.16695 -0.17966 0.040634 0.2485983511 3 -0.31579 0.0665 -0.085975 0.3339719672 3 0.14884 0.11316 0.045341 0.192391106Table 4.7: Displacement error of the laboratory landmarks.AlgorithmError Metrics (m)Max Min Mean Median SD SE MSE RMSERTAB-Map (3D LiDAR) 0.161 0.0409 0.117 0.151 0.0543 0.0502 0.0167 0.129BLAM 0.334 0.192 0.258 0.249 0.0582 0.210 0.0701 0.265Table 4.8: Results for the real-world experiments using the proposed metric.594.3. SUMMARY05,00010,00015,00020,00025,0000 2 4 6Cloud to Mesh Distance (m)CountHDL Graph SLAMCloud to Mesh Distances for Tank SimulationFigure 4.8: Cloud to mesh (C2M) distances for HDL Graph SLAM.ability in real-world testing, but failed to run in simulation. LeGO-LOAM gave neither anaccurate map nor trajectory. It is possible that results could be improved by increasingthe update frequency and by configuring it to output a more dense map. The proposedmethod complimented the other methods used within simulation and proved that it canbe applied to real-world data. The following criteria must be satisfied in order for theproposed method to be a valid benchmarking tool:1. The map must be sufficiently well made to extract the landmarks.2. The map must be dense enough to have enough points to conduct feature extraction.Criteria 1 was the main source of failure during validation. A simple visual inspectionwas enough in these cases to determine whether the landmarks could be extracted. If afull reference scan or mesh of the environment is available, the C2M metric is superior due604.3. SUMMARY050100150−0.5 0.0 0.5 1.0 1.5 2.0Cloud to Mesh Distance (m)CountLeGO−LOAMCloud to Mesh Distances for Tank SimulationFigure 4.9: Cloud to mesh (C2M) distances for LeGO-LOAM.to the wealth of information it can offer and its simplicity; however, this is not practicaloutside of simulations or pre-made datasets. The extendable nature of the frameworkalso introduces the opportunity to provide more meaningful insights into the mappingprocess.614.3. SUMMARY0246HDL Graph SLAM LeGO−LOAM RTAB−Map (2D LiDAR) RTAB−Map (3D LiDAR)AlgorithmDistances (m)Tank SimulationCloud to Mesh Distances ComparisonFigure 4.10: Cloud to mesh (C2M) distances comparison for the tank simulation.MaxMeanMedianMinSD0 2 4 6Value (m)MetricHDL Graph SLAMLeGO−LOAMRTAB−Map (2D LiDAR)RTAB−Map (3D LiDAR)Tank SimulationCloud to Mesh Distance StatisticsFigure 4.11: Unsigned cloud to mesh (C2M) distances statistics for the tank simulation.624.3. SUMMARYFigure 4.12: An example of a correct loop closure detection. Current image (left) matchedto an existing key frame (right).Figure 4.13: An example of a false loop closure detection. Current image (left) matchedto an existing key frame (right).634.3. SUMMARYFigure 4.14: Final maps of the tank produced by SLAM algorithms (Top (left-to-right):BLAM, HDL Graph SLAM, LeGO-LOAM; Bottom (left-to-right): RTAB-Map (2D Li-DAR), RTAB-Map (3D LiDAR)).64Chapter 5Conclusions and Future Work5.1 ConclusionsThe main contribution of this thesis is a framework for evaluating point cloud recon-structions of environments without the need for a fully realized ground truth. It is one ofthe few structural-based algorithms that can be applicable outside of premade datasets.The thesis begins with a literature review of SLAM in Chapter 2, including waysto classify algorithms, intelligent SLAM operations (i.e. active SLAM and semantic la-belling), and benchmarking approaches. Traditional benchmarking of SLAM fits intothree categories. Two of these, localization and environment-based, evaluate the qualityof the SLAM process in different ways, while the third evaluates hardware performance.Environment-based metrics are considerably harder to evaluate and have been often ne-glected in research due to the absence of a fully realized ground truth. A brief outline ofpoint cloud registration and a few registrations of algorithms is also given.Chapter 3 gives an outline of the hardware and software used in this research andreasons for their choosing. It also gives details on the validation experiments that wereconducted and the metrics used to evaluate them. Details about the implementation ofthe proposed framework are outlined and discussed, including the expandability of the655.1. CONCLUSIONSframework.Chapter 4 details the results of the experiments and discusses the outcomes. RTAB-Mapwas deemed to be the most suitable algorithm for both trajectory estimation and map-ping quality based on all three benchmarking approaches used. LeGO-LOAM was theonly SLAM algorithm that failed to perform well, although this may have to do withparameter optimization and the environment being a poor match for the algorithm. Theproposed framework failed to be applicable in situations where the landmarks could notbe extracted successfully. This was either because there were not enough points withinthe map to cluster the landmarks or the map was of too poor quality to allow for a cleanextraction.The framework proposed relies on knowledge of objects within the environment andcan be used when a full ground truth is not available. It extracts features from the objectswithin the environment and compares them relative to each other. These comparisons canthen be compared to corresponding measurements from reference objects. The featuresused for validation were the Cartesian locations of the centroids for each object with themetric being the euclidean distance between them; however, the framework can easily beextended to use additional features and comparison methods. Use of the metric reinforcesresults from the existing C2M metric, which is the most commonly used structural-basedapproach in literature.665.2. FUTURE WORK5.2 Future WorkThis thesis focuses on the development of a framework for evaluating SLAM recon-structions using an incomplete ground truth submap. A major drawback to this approachis the amount of manual work that the process currently requires. An improvement wouldbe implementing an end-to-end registration method to align the 3D model to the pointcloud map allowing for automated extraction of the relevant points from the point cloud.A false positive could result in a wildly inaccurate benchmark so the results should alwaysbe verified before relying on them.The system currently runs offline. Making the algorithm online would allow the user tosee how the mapping quality evolves over time. Additionally, it offer closed-loop feedbackto a mapping process, resulting in higher quality maps.Another approach would be to implement object detection algorithms to automaticallyextract these landmarks. Work has been done on 3D object detection algorithms thatonly rely on point cloud data [11]. The landmarks used in the validation experimentswere of a primitive type. More distinctive objects would allow for more easier objectrecognition and extraction. As with the registration method, effort would need be madeto reduce false positives, especially if the algorithm was to be made online.Using additional features is another area of improvement. For example, corner de-tection algorithms like [45] could be implemented on the corners of the boxes to addadditional features within the metric. Likewise, additional comparison methods couldbe implemented rather than distances. Features also do not need to only compared tofeatures from other objects. If multiple features can be extracted from a single landmark,675.2. FUTURE WORKthey can be compared to one another. Extracted objects can also be combined for somemetrics (e.g. a single centroid of two landmarks is a valid feature).Additional work needs to be done on multi-room SLAM. This work has dealt solelywith single room applications. It is possible that individual rooms may be mapped accu-rately but the relations between those rooms are not accurate. The metric is biased tosingle rooms, as it deals with local features. Further research needs to be done on theeffect of evaluating multiple clusters of objects to evaluate larger maps.The middleware and simulator used for experiments, ROS and Gazebo, are approach-ing their end of life. Unfortunately, their replacements, ROS 2 and Ignition, were notcompatible with the hardware. Ignition in particular allows for easier substitution ofphysics and rendering engines, which would greatly improve the realism of the simula-tions. Additionally, only the final version of ROS (i.e. ROS Noetic) supports Python3 which was not supported by Clearpath, meaning experiments were limited to Python2. Many machine and deep learning implementations dropped or never had support forPython 2 in the first place. Implementing 3D object detection algorithms that rely onthese frameworks, would require modification of the Jackal software or more complicatedsoftware approaches such as virtual environments. Additional work should consider thecurrent landscape of robotics software platforms.Adding additional objects to the scene adds additional structures which directly affectsthe mapping process; however, it is hard to quantify how much of an impact adding theseobjects has due to the uniqueness of the simulations. It is impossible to duplicate theexact motion of the robot due to the publisher-subscriber model used by Gazebo. Gazebopublishes an internal clock that is used as a timestamp within the header of most sensor685.2. FUTURE WORKmessages. A notable exception is the state of the robot model and link states (i.e. wheelrotations), which are continually published without a timestamp. Updating states withina second Gazebo simulation using states recorded from a previous simulation results innoticeably jerky motion. This is partially because not all states can be recorded due tobandwidth limitations. Furthermore, updating a state requires having a node subscribeto the recorded states and then publish them to a service which queues the states forGazebo. This will cause a delay and inconsistent timing between the states. Creating anew simulation environment that does not rely on the Gazebo API to publish the statescould be a potential fix.Since many SLAM algorithms rely on measured motion data as a constraint, repeatinga simulation requires the motion sensor data to be repeated as well. Reusing data requiresrecording the first simulation, and playing it back during the second simulation; however,the differing timestamps prevent the data from being usable. Subscribing to the sensortopics, updating the timestamp to the second simulation, and republishing the motiondata results in a non-constant delay and noisy sensor readings, which negatively affectmapping.Additional applications for the benchmarking framework exist outside of SLAM. Forexample, comparing the location of features within mining equipment for predictive main-tenance. It can also be used to calibrate the hardware stack for object detection. If anoffset in features is consistent during benchmarking, it can be accounted for in futuremeasurements. This could be used for individual sensors or within a sensor fusion hierar-chy.Lastly, integration of these metrics within the SlamBench framework [60–62] would695.2. FUTURE WORKgreatly improve the usability of the application. However, as SlamBench is not ROS-compatible, either an ROS integration needs to be developed, or each SLAM algorithmneeds to have a SlamBench API integration. SlamBench is also designed to be usedonline and uses recorded sensor data to evaluate different metrics. Using this approachon a method that purely evaluates the end map would be incredibly wasteful.70Bibliography[1] R. C. Smith and P. Cheeseman, “On the representation and estimation of spatialuncertainty,” The International Journal of Robotics Research, vol. 5, no. 4, p. 56–68,1986. → pages 1[2] J. Leonard and H. Durrant-Whyte, “Simultaneous map building and localizationfor an autonomous mobile robot,” in Proceedings IROS ’91:IEEE/RSJ InternationalWorkshop on Intelligent Robots and Systems ’91, vol. 3, p. 1442–1447, IEEE, 1991.→ pages 2, 10[3] T. Bailey and H. Durrant-Whyte, “Simultaneous localization and mapping (slam):Part ii,” IEEE Robotics and Automation Magazine, vol. 13, no. 3, p. 108–117, 2006.→ pages 2, 10[4] H. Durrant-Whyte and T. Bailey, “Simultaneous localization and mapping: Part i,”IEEE Robotics and Automation Magazine, vol. 13, no. 2, p. 99–108, 2006. → pages2[5] C. Evers and P. A. Naylor, “Acoustic slam,” IEEE/ACM Transactions on AudioSpeech and Language Processing, vol. 26, no. 9, p. 1484–1498, 2018. → pages 271Bibliography[6] A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, “Octomap:An efficient probabilistic 3d mapping framework based on octrees,” AutonomousRobots, vol. 34, no. 3, p. 189–206, 2013. → pages 2, 9, 17, 30[7] V. S. Kalogeiton, K. Ioannidis, G. C. Sirakoulis, and E. B. Kosmatopoulos, “Real-time active slam and obstacle avoidance for an autonomous robot based on stereovision,” Cybernetics and Systems, vol. 50, no. 3, p. 239–260, 2019. → pages 5, 6, 9[8] S. L. Bowman, N. Atanasov, K. Daniilidis, and G. J. Pappas, “Probabilistic data asso-ciation for semantic slam,” Proceedings - IEEE International Conference on Roboticsand Automation, p. 1722–1729, 2017. → pages 5, 8[9] K. Tateno, F. Tombari, I. Laina, and N. Navab, “CNN-SLAM: Real-Time DenseMonocular SLAM with Learned Depth Prediction,” in 2017 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), vol. 2017-Janua, pp. 6565–6574,IEEE, jul 2017. → pages 6[10] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, andJ. J. Leonard, “Past, present, and future of simultaneous localization and mapping:Toward the robust-perception age,” IEEE Transactions on Robotics, vol. 32, no. 6,p. 1309–1332, 2016. → pages 6[11] X. Chen, Y. Chen, K. Gupta, J. Zhou, and H. Najjaran, “Slicenet: A proficient modelfor real-time 3d shape-based recognition,” Neurocomputing, vol. 316, p. 144–155,2018. → pages 6, 7, 6772Bibliography[12] X. Chen, Y. Chen, and H. Najjaran, “3d object classification with point convolu-tion network,” IEEE International Conference on Intelligent Robots and Systems,vol. 2017-Septe, no. 250, p. 783–788, 2017. → pages 6, 7[13] D. G. Lowe, “Object recognition from local scale-invariant features,” Proceedings ofthe IEEE International Conference on Computer Vision, vol. 2, p. 1150–1157, 1999.→ pages 6, 14[14] H. Bay, T. Tuytelaars, and L. V. Gool, “Lncs 3951 - surf: Speeded up robust fea-tures,” Computer Vision–ECCV 2006, p. 404–417, 2006. → pages 6, 14[15] X. Chen, Y. Chen, and H. Najjaran, End to end robust point-cloud alignment usingunsupervised deep learning, vol. 12015 LNCS. Springer International Publishing,2020. → pages 7[16] S. Thrun, “Probabilistic robotics,” Communications of the ACM, vol. 45, no. 3,p. 52–57, 2002. → pages 8[17] M. Montemerlo, S. Thrun, D. Koller, and B. Wegbreit, “Fastslam: A factored so-lution to the simultaneous localization and mapping problem michael,” in NationalConference on Artificial Intelligence, p. 593–598, American Association for ArtificialIntelligence, Jul 2002. → pages 8, 10, 11[18] M. Montemerlo, S. Thrun, D. Roller, and B. Wegbreit, “Fastslam 2.0: An improvedparticle filtering algorithm for simultaneous localization and mapping that prov-ably converges,” IJCAI International Joint Conference on Artificial Intelligence,p. 1151–1156, 2003. → pages 8, 10, 1173Bibliography[19] M. Farrokhsiar and H. Najjaran, “A velocity-based rao-blackwellized particle filterapproach to monocular vslam,” Journal of Intelligent Learning Systems and Appli-cations, vol. 03, no. 03, p. 113–121, 2011. → pages 8[20] C. Tzelepis, Z. Ma, V. Mezaris, B. Ionescu, I. Kompatsiaris, G. Boato, N. Sebe, andS. Yan, “Event-based media processing and analysis: A survey of the literature,”Image and Vision Computing, vol. 53, p. 3–19, 2016. → pages 9[21] M. Zaffar, S. Ehsan, R. Stolkin, and K. M. D. Maier, “Sensors, slam and long-term autonomy: A review,” 2018 NASA/ESA Conference on Adaptive Hardwareand Systems, AHS 2018, p. 285–290, 2018. → pages 9[22] G. Grisetti, C. Stachniss, and W. Burgard, “Improved techniques for grid map-ping with rao-blackwellized particle filters,” IEEE Transactions on Robotics, vol. 23,p. 34–46, Feb 2007. → pages 11[23] D. Fox, W. Burgard, F. Dellaert, and S. Thrun, “Monte carlo localization: efficientposition estimation for mobile robots,” Proceedings of the National Conference onArtificial Intelligence, no. Handschin 1970, p. 343–349, 1999. → pages 11[24] M. Labbe and F. Michaud, “Appearance-based loop closure detection for online large-scale and long-term operation,” IEEE Transactions on Robotics, vol. 29, p. 734–745,Jun 2013. → pages 11, 14, 17, 29[25] M. Labbe and F. Michaud, “Rtab-map as an open-source lidar and visual slam libraryfor large-scale and long-term online operation,” Journal of Field Robotics, vol. 36,no. 2, p. 416–446, 2018. → pages 11, 12, 1974Bibliography[26] M. Labbe´ and F. Michaud, “Long-term online multi-session graph-based splam withmemory management,” Autonomous Robots, vol. 42, no. 6, p. 1133–1150, 2018. →pages 11, 14, 17, 29[27] R. Mur-Artal, J. M. Montiel, and J. D. Tardos, “Orb-slam: A versatile and ac-curate monocular slam system,” IEEE Transactions on Robotics, vol. 31, no. 5,p. 1147–1163, 2015. → pages 12[28] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “Orb: An efficient alternativeto sift or surf,” in 2011 International Conference on Computer Vision, vol. 2008,p. 2564–2571, IEEE, Nov 2011. → pages 12, 14[29] R. Mur-Artal and J. D. Tardos, “Orb-slam2: An open-source slam system for monoc-ular, stereo, and rgb-d cameras,” IEEE Transactions on Robotics, vol. 33, no. 5,p. 1255–1262, 2017. → pages 12[30] C. Campos, R. Elvira, J. J. G. Rodriguez, J. M. M. Montiel, and J. D. Tardos,“ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, andMultimap SLAM,” IEEE Transactions on Robotics, pp. 1–17, 2021. → pages 12[31] T. Whelan, S. Leutenegger, R. F. Salas-Moreno, B. Glocker, and A. J. Davison,“Elasticfusion: Dense slam without a pose graph,” Robotics: Science and Systems,vol. 11, 2015. → pages 12, 18, 24[32] E. Nelson, “Blam.” https://github.com/erik-nelson/blam/, 2016. → pages 12,2975Bibliography[33] F. Dellaert, “Factor graphs and gtsam,” Technical Report, no. GT-RIM-CPR-2012-002, p. 1–27, 2012. → pages 12[34] J. Zhang and S. Singh, “Low-drift and real-time lidar odometry and mapping,” Au-tonomous Robots, vol. 41, no. 2, p. 401–416, 2017. → pages 12, 13, 27, 30[35] T. Shan and B. Englot, “Lego-loam: Lightweight and ground-optimized lidar odom-etry and mapping on variable terrain,” IEEE International Conference on IntelligentRobots and Systems, no. September 2019, p. 4758–4765, 2018. → pages 13, 29, 49[36] K. Koide, J. Miura, and E. Menegatti, “A portable three-dimensional lidar-basedsystem for long-term and wide-area people behavior measurement,” InternationalJournal of Advanced Robotic Systems, vol. 16, p. 172988141984153, Mar 2019. →pages 13, 14, 29[37] G. Grisetti, R. Kummerle, C. Stachniss, and W. Burgard, “A tutorial on graph-basedslam,” IEEE Intelligent Transportation Systems Magazine, vol. 2, no. 4, p. 31–43,2010. → pages 13[38] R. Ku¨mmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “G2o: A gen-eral framework for graph optimization,” Proceedings - IEEE International Conferenceon Robotics and Automation, p. 3607–3613, 2011. → pages 13, 14[39] S. Kohlbrecher, O. von Stryk, J. Meyer, and U. Klingauf, “A flexible and scalableSLAM system with full 3D motion estimation,” in 2011 IEEE International Sym-posium on Safety, Security, and Rescue Robotics, pp. 155–160, IEEE, nov 2011. →pages 1376Bibliography[40] P. Besl and N. D. McKay, “A method for registration of 3-d shapes,” IEEE Trans-actions on Pattern Analysis and Machine Intelligence, vol. 14, p. 239–256, Feb 1992.→ pages 13, 14, 15, 32[41] F. Pomerleau, F. Colas, R. Siegwart, and S. Magnenat, “Comparing icp variants onreal-world data sets: Open-source library and experimental protocol,” AutonomousRobots, vol. 34, no. 3, p. 133–148, 2013. → pages 13, 14, 15, 32[42] M. Calonder, V. Lepetit, C. Strecha, and P. Fua, “Brief: Binary robust independentelementary features,” Lecture Notes in Computer Science (including subseries LectureNotes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 6314 LNCS,no. PART 4, p. 778–792, 2010. → pages 14[43] E. Rosten and T. Drummond, “Machine learning for high-speed corner detection,”in Computer Vision – ECCV 2006 (A. Leonardis, H. Bischof, and A. Pinz, eds.),(Berlin, Heidelberg), pp. 430–443, Springer Berlin Heidelberg, 2006. → pages 14[44] G. Bradski, “The opencv library,” Dr. Dobb’s Journal of Software Tools, 2000. →pages 14[45] I. Sipiran and B. Bustos, “Harris 3D: A robust extension of the Harris operator forinterest point detection on 3D meshes,” in Visual Computer, vol. 27, pp. 963–976,2011. → pages 15, 67[46] M. A. Fischler and R. C. Bolles, “Random sample consensus: A Paradigm for ModelFitting with Applications to Image Analysis and Automated Cartography,” Commu-nications of the ACM, vol. 24, no. 6, pp. 381–395, 1981. → pages 1577Bibliography[47] B. Jian and B. C. Vemuri, “Robust point set registration using Gaussian mixturemodels,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33,no. 8, pp. 1633–1645, 2011. → pages 15[48] N. Ahmad Fuad, A. R. Yusoff, Z. Ismail, and Z. Majid, “Comparing the performanceof point cloud registration methods for landslide monitoring using mobile laser scan-ning data,” in International Archives of the Photogrammetry, Remote Sensing andSpatial Information Sciences - ISPRS Archives, vol. 42, pp. 11–21, 2018. → pages15[49] P. Biber, “The normal distributions transform: A new approach to laser scan match-ing,” IEEE International Conference on Intelligent Robots and Systems, vol. 3,no. November 2003, p. 2743–2748, 2003. → pages 15, 16, 30[50] W. Lu, G. Wan, Y. Zhou, X. Fu, P. Yuan, and S. Song, “DeepVCP: An end-to-end deep neural network for point cloud registration,” in Proceedings of the IEEEInternational Conference on Computer Vision, vol. 2019-Octob, pp. 12–21, Instituteof Electrical and Electronics Engineers Inc., may 2019. → pages 15[51] M. Magnusson, A. Nu¨chter, C. Lo¨rken, A. J. Lilienthal, and J. Hertzberg, “Evaluationof 3D registration reliability and speed-A comparison of ICP and NDT,” in Proceed-ings - IEEE International Conference on Robotics and Automation, pp. 3907–3912,2009. → pages 16[52] M. L. Rodriguez-Arevalo, J. Neira, and J. A. Castellanos, “On the importance of78Bibliographyuncertainty representation in active slam,” IEEE Transactions on Robotics, vol. 34,no. 3, p. 829–834, 2018. → pages 16[53] X. Deng, Z. Zhang, A. Sintov, J. Huang, and T. Bretl, “Feature-constrained active vi-sual slam for mobile robot navigation,” Proceedings - IEEE International Conferenceon Robotics and Automation, p. 7233–7238, 2018. → pages 16[54] R. Valencia and J. Andrade-Cetto, Mapping, Planning and Exploration with PoseSLAM, vol. 119. 2018. → pages 16[55] R. Martinez-Cantin, N. de Freitas, E. Brochu, J. Castellanos, and A. Doucet, “ABayesian exploration-exploitation approach for optimal online sensing and planningwith a visually guided mobile robot,” Autonomous Robots, vol. 27, pp. 93–103, aug2009. → pages 16, 17[56] D. C. Lee, The Map-Building and Exploration Strategies of A Simple Sonar-EquippedRobot. 1996. → pages 17, 40[57] S. Schwertfeger, “Robotic mapping in the real world: Performance evaluation andsystem integration.,” Jacobs University, 2012. → pages 17, 18, 32, 40[58] A. Filatov, A. Filatov, K. Krinkin, B. Chen, and D. Molodan, “2d slam quality evalu-ation methods,” in 2017 21st Conference of Open Innovations Association (FRUCT),p. 120–126, IEEE, Nov 2017. → pages 18[59] A. Handa, T. Whelan, J. McDonald, and A. J. Davison, “A benchmark for rgb-d visual odometry, 3d reconstruction and slam,” Proceedings - IEEE InternationalConference on Robotics and Automation, p. 1524–1531, 2014. → pages 18, 3279Bibliography[60] L. Nardi, B. Bodin, M. Z. Zia, J. Mawer, A. Nisbet, P. H. J. Kelly, A. J. Davison,M. Lujan, M. F. P. O’Boyle, G. Riley, N. Topham, and S. Furber, “IntroducingSLAMBench, a performance and accuracy benchmarking methodology for SLAM,” in2015 IEEE International Conference on Robotics and Automation (ICRA), vol. 2015-June, pp. 5783–5790, IEEE, may 2015. → pages 18, 24, 69[61] B. Bodin, H. Wagstaff, S. Saecdi, L. Nardi, E. Vespa, J. Mawer, A. Nisbet, M. Lu-jan, S. Furber, A. J. Davison, P. H. J. Kelly, and M. F. P. O’Boyle, “SLAMBench2:Multi-Objective Head-to-Head Benchmarking for Visual SLAM,” in 2018 IEEE In-ternational Conference on Robotics and Automation (ICRA), pp. 3637–3644, IEEE,may 2018. → pages[62] M. Bujanca, P. Gafton, S. Saeedi, A. Nisbet, B. Bodin, O. Michael F.P., A. J.Davison, K. Paul H.J., G. Riley, B. Lennox, M. Lujan, and S. Furber, “SLAMBench3.0: Systematic Automated Reproducible Evaluation of SLAM Systems for RobotVision Challenges and Scene Understanding,” in 2019 International Conference onRobotics and Automation (ICRA), vol. 2019-May, pp. 6351–6358, IEEE, may 2019.→ pages 18, 24, 69[63] S. Schwertfeger, A. Jacoff, C. Scrapper, J. Pellenz, and A. Kleiner, “Evaluationof maps using fixed shapes: The fiducial map metric,” Performance Metrics forIntelligent Systems (PerMIS) Workshop, p. 339–346, 2010. → pages 18, 32[64] S. Schwertfeger and A. Birk, “Using fiducials in 3d map evaluation,” SSRR 2015 -2015 IEEE International Symposium on Safety, Security, and Rescue Robotics, 2016.→ pages 18, 3280Bibliography[65] R. Sheh, S. Schwertfeger, and A. Visser, “16 Years of RoboCup Rescue,” KI - Kun-stliche Intelligenz, vol. 30, no. 3-4, pp. 267–277, 2016. → pages 18[66] B. Pfrommer and K. Daniilidis, “TagSLAM: Robust SLAM with Fiducial Markers,”oct 2019. → pages 19[67] H. Zhang, C. Zhang, W. Yang, and C.-Y. Chen, “Localization and navigation usingQR code for mobile robot in indoor environment,” in 2015 IEEE International Con-ference on Robotics and Biomimetics (ROBIO), pp. 2501–2506, IEEE, dec 2015. →pages 19[68] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kittidataset,” International Journal of Robotics Research, vol. 32, no. 11, p. 1231–1237,2013. → pages 19, 20, 21, 22[69] J.-L. Blanco-Claraco, F.-A´. Moreno-Duen˜as, and J. Gonza´lez-Jime´nez, “The Ma´lagaurban dataset: High-rate stereo and LiDAR in a realistic urban scenario,” The In-ternational Journal of Robotics Research, vol. 33, pp. 207–214, feb 2014. → pages20, 22[70] J. Jeong, Y. Cho, Y. S. Shin, H. Roh, and A. Kim, “Complex urban dataset withmulti-level sensors from highly diverse urban environments,” International Journalof Robotics Research, vol. 38, no. 6, p. 642–657, 2019. → pages 20, 22[71] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song,A. Zeng, and Y. Zhang, “Matterport3D: Learning from RGB-D data in indoor envi-ronments,” International Conference on 3D Vision (3DV), 2017. → pages 20, 2281Bibliography[72] M. F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang,P. Carr, S. Lucey, D. Ramanan, and J. Hays, “Argoverse: 3D Tracking and Fore-casting with Rich Maps,” Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition, vol. 2019-June, pp. 8740–8749, nov 2019.→ pages 20, 22[73] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan,G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driv-ing,” arXiv preprint arXiv:1903.11027, 2019. → pages 20[74] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark forthe evaluation of rgb-d slam systems,” IEEE International Conference on IntelligentRobots and Systems, p. 573–580, 2012. → pages 20, 21, 22[75] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik,and R. Siegwart, “The euroc micro aerial vehicle datasets,” International Journal ofRobotics Research, vol. 35, no. 10, p. 1157–1163, 2016. → pages 20, 21, 22[76] W. Li, S. Saeedi, J. McCormac, R. Clark, D. Tzoumanikas, Q. Ye, Y. Huang,R. Tang, and S. Leutenegger, “Interiornet: Mega-scale multi-sensor photo-realisticindoor scenes dataset,” in British Machine Vision Conference (BMVC), 2018. →pages 20, 21, 22[77] J. McCormac, A. Handa, S. Leutenegger, and A. J.Davison, “Scenenet rgb-d: Can5m synthetic images beat generic imagenet pre-training on indoor segmentation?,”2017. → pages 20, 21, 2282Bibliography[78] Y. Z. S. Q. Z. X. T. S. K. Y. W. A. Y. Weichao Qiu, Fangwei Zhong, “Unrealcv:Virtual worlds for computer vision,” ACM Multimedia Open Source Software Com-petition, 2017. → pages 20, 21, 22[79] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub,J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat: A Platform forEmbodied AI Research,” in 2019 IEEE/CVF International Conference on ComputerVision (ICCV), vol. 2019-Octob, pp. 9338–9346, IEEE, oct 2019. → pages 21[80] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre,M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur,F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Ba-tra, “Habitat 2.0: Training Home Assistants to Rearrange their Habitat,” pp. 1–32,jun 2021. → pages 21[81] A. Bokovoy, K. Muravyev, and K. Yakovlev, “Maomaps: A photo-realistic benchmarkfor vslam and map merging quality assessment,” p. 2–7, 2021. → pages 21[82] J. Aulinas, Y. Petillot, J. Salvi, and X. Llado´, “The SLAM problem: A survey,”Frontiers in Artificial Intelligence and Applications, vol. 184, no. 1, pp. 363–371,2008. → pages 24[83] M. S. Bahraini, A. B. Rad, and M. Bozorg, “SLAM in Dynamic Environments : ADeep Learning,” Sensors, vol. 19, pp. 1–20, 2019. → pages 24[84] D. Inc., “Velodyne simulator.” https://bitbucket.org/DataspeedInc/velodyne_simulator/, 2021. → pages 27, 4383Bibliography[85] Intel, “Gazebo realsense plugin.” https://github.com/intel/gazebo-realsense,2018. → pages 27[86] P. Robotics, “Intel realsense gazebo ros plugin.” https://github.com/pal-robotics/realsense_gazebo_plugin/, 2021. → pages 27, 43[87] P. Robotics, “Ros wrapper for intel realsense devices.” https://github.com/pal-robotics-forks/realsense, 2021. → pages 27, 43[88] S. Umeyama, “Least-squares estimation of transformation parameters between twopoint patterns,” IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. 13, pp. 376–380, 1991. → pages 32, 48[89] R. B. Rusu, Semantic 3D Object Maps for Everyday Manipulation in Human LivingEnvironments. PhD thesis, Computer Science department, Technische UniversitaetMuenchen, Germany, October 2009. → pages 34[90] M. Grupp, “evo: Python package for the evaluation of odometry and slam..” https://github.com/MichaelGrupp/evo, 2017. → pages 4884
Thesis/Dissertation
eng
Mechanical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Najjaran, Homayoun
Benchmarking SLAM reconstructions in absence of a complete ground truth
Text