Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Abdelfattah, Mazen
Master of Applied Science - MASc
University of British Columbia
Modern Autonomous Vehicles (AVs) rely on sensory data often acquired by cameras and LiDARs to perceive the world around them. To operate the vehicle safely and effectively, Artificial Intelligence (AI) is used to process this data to detect objects of interest around the vehicle. For 3D object detection, recent advances in deep learning have resulted in the development of state-of-the-art multi-modal models which are built using Deep Neural Nets (DNNs) to process camera images, and LiDAR point clouds. While DNN-based models are very powerful and accurate they may be vulnerable to adversarial attacks which introduce a small change to a modelâs input and can result in great errors in its output. These attacks have been heavily investigated for models that operate on camera image input only, and recently for point cloud processing models, however they have rarely been investigated in models that utilize both modalities as is often the case in modern AVs. To address this gap we propose a realistic adversarial attack on such multi-modal 3D detection models. We place a 3D adversarial object on a vehicle with the aim of hiding this objectâs host vehicle from detection by powerful multi-modal 3D detectors. This objectâs shape and texture are trained so that it can be used to prevent a specific model from detecting any host vehicle in any scene. 3D detection models are often based on either a cascaded architecture where each input modality is processed consecutively, or a fusion architecture where multi-input features are extracted and fused simultaneously. We use our attack to study the vulnerabilities of representative models of these architectures to realistic adversarial attacks and to understand the effects of multi-modal learning on the robustness of a model. Our experiments show that a single adversarial object is capable of hiding its host vehicle 55.6% and 63.19% of the times from the cascaded model and from the fusion model respectively. This vulnerability was found to be mainly due to RGB image features which were much less robust to adversarial scene changes compared to the point cloud features.
Adversarial Attacks on Multi-Modal 3D Detection ModelsbyMazen AbdelfattahA THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of Applied ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Electrical & Computer Engineering)The University of British Columbia(Vancouver)April 2021© Mazen Abdelfattah, 2021The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Adversarial Attacks on Multi-Modal 3D Detection Modelssubmitted by Mazen Abdelfattah in partial fulfillment of the requirements for thedegree of Master of Applied Science in Electrical & Computer Engineering.Examining Committee:Prof. Rabab Ward, Electrical & Computer EngineeringCo-supervisorProf. Jane Z. Wang, Electrical & Computer EngineeringCo-supervisorProf. Panos Nasiopoulos, Electrical & Computer EngineeringSupervisory Committee MemberiiAbstractModern Autonomous Vehicles (AVs) rely on sensory data often acquired by cam-eras and LiDARs to perceive the world around them. To operate the vehicle safelyand effectively, Artificial Intelligence (AI) is used to process this data to detectobjects of interest around the vehicle. For 3D object detection, recent advancesin deep learning have resulted in the development of state-of-the-art multi-modalmodels which are built using Deep Neural Nets (DNNs) to process camera images,and LiDAR point clouds. While DNN-based models are very powerful and accu-rate they may be vulnerable to adversarial attacks which introduce a small changeto a modelâs input and can result in great errors in its output. These attacks havebeen heavily investigated for models that operate on camera image input only, andrecently for point cloud processing models, however they have rarely been investi-gated in models that utilize both modalities as is often the case in modern AVs. Toaddress this gap we propose a realistic adversarial attack on such multi-modal 3Ddetection models. We place a 3D adversarial object on a vehicle with the aim ofhiding this objectâs host vehicle from detection by powerful multi-modal 3D de-tectors. This objectâs shape and texture are trained so that it can be used to preventa specific model from detecting any host vehicle in any scene. 3D detection mod-els are often based on either a cascaded architecture where each input modality isprocessed consecutively, or a fusion architecture where multi-input features are ex-tracted and fused simultaneously. We use our attack to study the vulnerabilities ofrepresentative models of these architectures to realistic adversarial attacks and tounderstand the effects of multi-modal learning on the robustness of a model. Ourexperiments show that a single adversarial object is capable of hiding its host vehi-cle 55.6% and 63.19% of the times from the cascaded model and from the fusioniiimodel respectively. This vulnerability was found to be mainly due to RGB imagefeatures which were much less robust to adversarial scene changes compared to thepoint cloud features.ivLay SummaryFor Autonomous Vehicles (AVs), identifying 3D objects around the vehicle, likepedestrians or other vehicles, is crucial for its navigation and safety. AVs usuallyutilize complimentary inputs from the cameras and LiDARs to accurately detectthese objects using powerful Artificial Intelligence (AI) models. Such models how-ever could be vulnerable to attacks when the modelâs input is maliciously alteredto fool it into making prediction mistakes. Imagine if simply putting a single pe-culiarly shaped and colored 3D object on a vehicle can hide it from state-of-the-artdetectors in AVs. This is exactly what our research proves to be possible. Wedeveloped an adversarial object that, when placed on a vehicle, can hide it frompowerful detectors in more than 50% of the times. We use this attack to study thevulnerabilities of different design choices of AI detectors to inform researchers andprofessionals in the field of AV.vPrefaceThis thesis is my original work with the advice of colleague Kaiwen Yuan, and theguidance of co-supervisors Professor Wang and Professor Ward. The work on thecascaded modelâs attack presented in parts of Chapter 3 and Chapter 4 has beensubmitted for a conference and remains under consideration.A paper summarizing the work of my thesis has been submitted to a journalwith a conference option as Abdelfattah, M., Yuan, K., Wang, Z. J., & Ward, R.(2021). Adversarial Attacks on Camera-LiDAR Models for 3D Car Detection.Kaiwen Yuan proposed the idea of a multi-modal attack on Autonomous Vehi-cles and advised me on some tools and methods to use. I thoroughly researched theidea, developed and implemented the attack, conducted experiments, and reportedand discussed results in this thesis. My supervisors provided constant feedbackand guidance on the implementations, experiments, and writing of this work.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 3D Multi-Modal Detection . . . . . . . . . . . . . . . . . . . . . 21.1.1 Deep Neural Nets . . . . . . . . . . . . . . . . . . . . . . 21.1.2 Cascaded Models . . . . . . . . . . . . . . . . . . . . . . 51.1.3 Fusion Models . . . . . . . . . . . . . . . . . . . . . . . 71.2 Adversarial Attacks . . . . . . . . . . . . . . . . . . . . . . . . . 81.3 Motivation and Research Problems . . . . . . . . . . . . . . . . . 101.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11vii1.5 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141.6 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . 152 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.1 Image-Based Adversarial Attacks . . . . . . . . . . . . . . . . . 162.2 Point Cloud-Based Adversarial Attacks . . . . . . . . . . . . . . 202.3 Multi-Modal Adversarial Attacks . . . . . . . . . . . . . . . . . . 213 Multi-Modal Adversarial Attack . . . . . . . . . . . . . . . . . . . . 253.1 Adversarial Object . . . . . . . . . . . . . . . . . . . . . . . . . 253.2 Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.2.1 LiDAR Point Cloud Rendering . . . . . . . . . . . . . . . 273.2.2 RGB Image Rendering . . . . . . . . . . . . . . . . . . . 293.3 Cascaded Attack . . . . . . . . . . . . . . . . . . . . . . . . . . 303.4 Fusion Attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.5 Objective Functions . . . . . . . . . . . . . . . . . . . . . . . . . 353.5.1 Cascaded Model . . . . . . . . . . . . . . . . . . . . . . 373.5.2 Fusion Model . . . . . . . . . . . . . . . . . . . . . . . . 384 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 394.1.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.1.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.1.3 Implementation Details . . . . . . . . . . . . . . . . . . . 414.2 Recall & Attack Success Rate . . . . . . . . . . . . . . . . . . . 464.3 Effect on Average Precision . . . . . . . . . . . . . . . . . . . . . 515 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61A Qualitative Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 67viiiList of TablesTable 2.1 Summary of related work. . . . . . . . . . . . . . . . . . . . . 23Table 4.1 Attack success rates for the two models with IoU = 0.7. . . . . 47Table 4.2 Attack success rates of two similar models but one is multi-modal and the other is LiDAR-only with IoU = 0.7. . . . . . . 50Table 4.3 Cascaded: F-PN car detection AP results. . . . . . . . . . . . . 53Table 4.4 Fusion: EPNet car detection AP results. . . . . . . . . . . . . . 53ixList of FiguresFigure 1.1 Front view sample obtained from the KITTI dataset [6]. . . . 2Figure 1.2 A simple illustration of a Deep Neural Net (DNN). . . . . . . . 4Figure 1.3 Architecture of the cascaded model: Frustum-PointNet [26]. . 6Figure 1.4 Shortcomings of utilizing only camera images for detection. . 6Figure 1.5 Architecture of the fusion model: EPNet [10]. . . . . . . . . . 7Figure 1.6 An illustrative example of an adversarial attack on an imageclassification DNN as demonstrated in [8] . . . . . . . . . . . 8Figure 1.7 Overview of the proposed adversarial attack. . . . . . . . . . 12Figure 2.1 An illustrative example of the adversarial attack on a face clas-sification DNN proposed in [30] . . . . . . . . . . . . . . . . 17Figure 2.2 An illustrative example of the adversarial attack on a road signclassification DNN proposed in [5] . . . . . . . . . . . . . . . 18Figure 2.3 An illustrative example of the adversarial attack on a persondetection DNN proposed in [37] . . . . . . . . . . . . . . . . 19Figure 3.1 Overview of the LiDAR point cloud rendering process. . . . . 28Figure 3.2 Overview of the RGB image rendering process. . . . . . . . . 30Figure 3.3 Results of rendering the adversarial mesh to 3D point cloudand 2D RGB image. . . . . . . . . . . . . . . . . . . . . . . 31Figure 3.4 Attack pipeline for Frustum-PointNet (F-PN) [26]. . . . . . . . 33Figure 3.5 Attack pipeline for EPNet [10]. . . . . . . . . . . . . . . . . 36Figure 4.1 The initial grey spherical mesh with 162 vertices and 320 faces. 41xFigure 4.2 The rendered initial spherical mesh which has a diameter of 80cm. The sphere is brighter than the one shown in Figure 4.1because of simulated sunlight and Phong shading. . . . . . . . 42Figure 4.3 Multi-view illustration of the adversarial object from our multi-modal attack on the cascaded model. . . . . . . . . . . . . . . 44Figure 4.4 Multi-view illustration of the adversarial object from our multi-modal attack on the fusion model. . . . . . . . . . . . . . . . 45Figure 4.5 The adversarial mesh produced from our âImage onlyâ attackrendered to image and point cloud. . . . . . . . . . . . . . . . 46Figure 4.6 The adversarial mesh produced from our âLiDAR onlyâ attackrendered to image and point cloud. . . . . . . . . . . . . . . . 47Figure 4.7 Victim vehicle recall vs. IoU thresholds plot. . . . . . . . . . . 48Figure 4.8 An example of the fusion modelâs detection without our adver-sarial attack. . . . . . . . . . . . . . . . . . . . . . . . . . . 52Figure 4.9 An example of the fusion modelâs detection after placing ouradversarial object on a vehicle. . . . . . . . . . . . . . . . . . 52Figure A.1 First example of the multi-modal attack on the fusion model. . 68Figure A.2 Second example of the multi-modal attack on the fusion model. 69Figure A.3 First example of the multi-modal attack on the cascaded model. 70Figure A.4 Second example of the multi-modal attack on the cascadedmodel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71xiGlossaryAI Artificial IntelligenceAV Autonomous VehicleAP Average PrecisionBCE Binary Cross EntropyBEV Birdâs Eye ViewCNN Convolutional Neural NetDNN Deep Neural NetF-PN Frustum-PointNetIoU Intersection-over-UnionSGD Stochastic Gradient DescentxiiAcknowledgmentsI would like to express my gratitude to my supervisors, Professor Rabab Ward andProfessor Jane Z. Wang. They took me in, offered me a space to explore new ideas,gave invaluable advice, and were always supportive. Their welcoming environmentallowed me to meet my colleague Kaiwen Yuan who introduced me to the field ofdeep learning for autonomous driving and fueled the interest that produced thiswork.I must also thank Professor Amr Mohamed, for being such a great mentorwhose advice both personal and professional guided me through many hard deci-sions. His passion for research and science has been truly inspiring and greatlyinfluenced my thinking.I was also lucky enough to intern at Qatar Computing Research Institute un-der the mentorship of Dr. Kareem Darwish with whom Iâve enjoyed many deepdiscussions on research and beyond. I aspire to one day live up to his example ofleadership, generosity, and deep curiosity.I would be remiss if I didnât thank my friends nearby and abroad. Their con-stant support especially in these trying times of the pandemic helped me moveforward and offered comfort and solace.My utmost gratitude and love goes to my parents who made me the person I amtoday. Without their love and encouragement I would not have been a researcherand this thesis would not have been possible.xiiiDedicationI. K@ éJÃ @Ã° IÃÂ¿Ã±K éJÃÂ« é<ÃAK. B@ Ã¹®J ¯Ã±K AÃðxivChapter 1IntroductionModern Autonomous Vehicles (AVs) rely on many sensors to be able to safely nav-igate on their own. These sensors usually include cameras, LiDARs and radars.However, the two most common sensors are RGB cameras which provide dense2D RGB images, and LiDAR sensors which provide sparse 3D point clouds of thescene around a vehicle. These complimentary sensory inputs are needed by per-ception systems to help a self-driving vehicle move, navigate, and maneuver safelyand autonomously. An RGB image provides rich semantic information but itâsonly two-dimensional and thus does not provide any information regarding the dis-tance between a vehicle and objects around it or the 3D geometry of these objects.Moreover, camera imaging can suffer from poor lighting conditions and occlusion.Thatâs where the LiDAR sensor is useful. A LiDAR sends thousands of laser raysand receives the subsequent reflection of rays off objects in the scene thus produc-ing a 3D point cloud of reflection points. This point cloud is usually sparse andunordered but provides a lot of depth information. An example of an RGB imageand the corresponding LiDAR point cloud obtained from the KITTI dataset [6] areshown in Figure 1.1. While a LiDAR provides a 360â¦ view of the scene, only thefront view has a corresponding image since KITTI only provides images from afront view camera.1Figure 1.1: Front view sample obtained from the KITTI dataset [6]. Topshows a 3D point cloud captured by a LiDAR. Bottom shows an RGBimage captured by a camera.1.1 3D Multi-Modal Detection3D object detection is the task of locating (x,y,z 3D location), classifying (car,pedestrian, cyclist, etc.) and predicting the oriented 3D bounding box dimensionsof objects in 3D [3]. The task of detecting objects in 3D is critical for any percep-tion system to operate safely and effectively. Since images provide a lot of orderedand dense semantic information but no depth information and since point cloudsprovide depth information but are unordered and sparse, many works proposedutilizing both modalities to overcome their respective shortcomings. Perceptionmodels based on Deep Neural Nets (DNNs) that use images and point clouds asinput (multi-modal) achieved state-of-the-art performance in 3D detection.1.1.1 Deep Neural NetsDNNs are mainly used for supervised learning where they act as layered functionapproximators that map the raw input to the labelled output. Their great perfor-mance in many tasks stems from their ability to map very complex relationships inhigh-dimensional space. To do that DNNs take the raw input through an input layer2to transform it using non-linear operations to higher more abstract representations.These representations go through hidden layers where more non-linear transforma-tions occur to get even higher levels of abstraction until at the end an output layerproduces a prediction [13].The process of training a DNN is the process of changing its internal weights(parameters that define the transformations) towards minimizing a certain objectivefunction that defines the amount of error in a DNNâs prediction. Stochastic GradientDescent (SGD) is one of the most common optimization tools that uses differenti-ation and backpropogation to optimize the weights by attempting to find a âgoodâminimum for the objective function [13]. It is called stochastic since it does notcompute gradients on the whole dataset, instead it uses a few input/output exam-ples to iteratively update the parameters. Since supervised learning is data-drivenit usually needs huge amounts of data and exposure to as many samples from theinput distribution as possible to perform well. The recent abundance of labelleddata is one of the major reasons behind the success of DNNs.The field of image classification and detection popularized a type of DNNscalled Convolutional Neural Nets (CNNs). These nets have proven to be very ef-fective and efficient in extracting semantic image features using multiple convolu-tional, activation, pooling and normalization layers [3, 13]. The CNNâs hierarchicalstructure allows it to extract image features at multiple levels of abstraction de-pending on the networkâs depth (number of layers).Deep learning has also allowed for great strides in the field of machine learningon point clouds. Due to the sparse and continuous nature of point clouds it is nottrivial to learn from them using deep learning methods. Many works proposeddiscretizing the input into 3D grids and applying 3D convolutions [20] or projectingthe 3D point cloud to 2D views and then using CNNs for learning [35]. Howeverthese approach usually suffer from information loss and 3D convolutions are oftencomputationally expensive [3, 9]. Qi et al. [24] proposed a DNN that directly takesthe point cloud as input without projection or discretization. They used sharedmulti-layer perceptrons (or 1 dimensional convolutions) to extract features for eachinput point and then a symmetric function (a max-pooling layer) to aggregate thisinformation into a global feature vector. This simple method proved very effectiveand many deep learning work on point clouds use PointNet [24] or its successor3Figure 1.2: A simple illustration of a Deep Neural Net (DNN).PointNet++ [25] as a backbone for feature extraction [10, 26, 31, 32, 42].A key to the success of DNNs is their assumption that input data can be thoughtof as compositional hierarchies where higher level (more abstract) features can becomposed of smaller features down to the raw input [13]. This assumption is oftentrue of natural data like images and point clouds. For example, a camera image canconsist of edges, a group of edges can define a part of an object, and a group ofparts can define an object. For point clouds, it was found that PointNet [24] learnsâcritical pointsâ which are the few specific points that define a skeleton of a 3Dshape and heavily influence the learned features. This further proves the potencyof DNNs in learning complex and meaningful information from different types ofreal-life data. The great feature extraction ability of DNNs popularized their use inmany Artificial Intelligence (AI) models and systems. In some of the figures in ourthesis we will represent a DNN by the symbol shown in Figure 1.2These advances in image-based and point cloud-based deep learning openedthe door for multi-modal deep learning where image input from a camera and4point cloud input from a LiDAR are jointly used to accomplish different tasks.Multi-modal 3D detection models utilize a large trove of annotated (labelled) im-ages and point clouds to learn 3D detection in a supervised manner. The mostcommon dataset and benchmark for 3D detection is the KITTI dataset [6]. Thismulti-modal dataset consists of thousands of images and their corresponding Li-DAR point clouds obtained from the vantage point of an ego AV. Multi-modal 3Ddetection models are usually based on one of two common architectures: cascaded,or fusion.1.1.2 Cascaded ModelsCascaded models [26, 32, 42] utilize each modality separately. First, images aresent through an off-the-shelf 2D detection DNN to generate region proposals wherethe DNN estimates a car may reside [9]. LiDAR points in each proposed regionare then extracted and sent through a point cloud-based DNN for 3D detection. Anexample of a cascaded modelâs architecture is shown in Figure 1.3.Cascaded models thus have two separate pipelines, one to process images andthen another for point clouds. This approach has many advantages: first, usingthe reduced 3D search space can significantly reduce the computational cost of de-tection and can improve runtime [3]. Moreover, cascaded models rely on mature2D detection models and so they can benefit greatly from the extensive work andfast advances in the 2D object detection field. Most importantly the work and ad-vances in deep learning interpretability which is necessary for safety-critical tasksincluding of course autonomous driving [15]. Because of these many advantages,cascaded models are usually closer to training, testing, and deployment in industry.Moreover, cascaded models have an implicit bias that in the post-proposal limited3D point cloud a single object must reside. Focusing on a single object can makethem effective and robust in regressing bounding box dimensions but also can leadto them missing objects in crowded scenes or smaller objects. Also, cascadedmodels are often limited by the performance and accuracy of their 2D detectionpipeline [3]. Moreover, using image-based region proposals doesnât utilize one ofthe main benefits of LiDAR which is to avoid cases where lighting and occlusionaffect detection and localization as shown in Figure 1.4.5Figure 1.3: Architecture of the cascaded model: Frustum-PointNet (F-PN)[26]. The model takes the image through a 2D detector for region pro-posal. Proposed regions are lifted from 2D to 3D to produce frustum-shaped search spaces. Points within these frustums are extracted andsent through 2 PointNet-based [24] DNNs for segmentation and thenbounding box estimation. These two nets are illustrated as the 3D de-tection pipeline.Figure 1.4: Shortcomings of utilizing only camera images for detection. Im-ages obtained from the KITTI dataset [6].6Figure 1.5: Architecture of the fusion model: EPNet [10]. The model takesthe image through standard 2D CNNs and the point cloud through DNN-based modules from [25] for feature extraction. LiDAR point featuresare enhanced by image features on multiple scales of abstraction downthe network using a fusion module proposed in [10]. Fused features arethen sent through a 3D detection DNN to predict 3D bounding boxes.1.1.3 Fusion ModelsOn the other hand, fusion models [10, 34] use DNNs to extract and combine im-age and point cloud features in parallel. Afterwards, a combined representation issent through a DNN for 3D detection. An example of a fusion modelâs architectureis shown in Figure 1.5. This architecture allows a model to utilize the intuitivereal-life correspondence between an image and a point cloud which enriches theirrespective features. However, due to the opposite nature of the input data (sparse,continuous point clouds, and dense, discrete images) the fusion task is non-trivialand has many unique approaches [10, 14, 34]. These methods are united in fusingfeatures from different modalities but differ in the fusion method. Fusion modelscan usually mitigate the shortcomings of every input modality. So using LiDARpoint clouds can help in cases of occlusion or poor lighting, and images can helpwith rich color information when points arenât enough to discern an object. How-ever, some fusion methods usually suffer from a âfeature blurringâ problem wherethe projection of image features to points or vice versa dilutes the features amongdifferent units (pixels or points) which can be a waste of useful information [3].7Figure 1.6: An illustrative example of an adversarial attack on an image clas-sification DNN as demonstrated in [8]. Specific learned perturbations areadded to the input to produce an error in the output. The ε is a weightthat controls the trade-off between an imperceptible attack and a moreeffective attack. The image is obtained from the KITTI dataset [6].1.2 Adversarial AttacksMost AI 3D detection models rely on DNNs for feature extraction, and then detec-tion. While very effective in learning features, DNNs have been shown to be vul-nerable to adversarial attacks [36]. These attacks introduce specific small changesto the input of a DNN that produce great errors in the predicted output. This meansthat while DNNs learn good features, these features are however brittle and canbreak easily with specific and imperceptible changes to the input. Adversarial at-tacks were initially observed and studied in the RGB image classification domain[8, 12, 22, 30, 36].The classical example of an adversarial attack on a DNN-based image classifierwas demonstrated in [8]. The input is a clean example of a panda that is classifiedcorrectly. When specifically-crafted imperceptible perturbations are added to thepanda picture producing an adversarial example, the DNN classified what is clearly(to a human observer) still a panda as a gibbon instead with very high confidence.The perturbed input is called an adversarial example and it led the DNN to make aglaring error. A demonstrative example is shown in Figure 1.6.While this attack is effective, it still has two major weaknesses: first, it requirespixel-wise perturbations and so it can be difficult for this attack to be replicatedand thus be threatening in real life. Such digital perturbations work under the weak8assumption that an attacker can access the input digital stream to a model. Sec-ond, this attack would work only on a single image since the perturbations werespecifically crafted for that single image. To address the first issue, restrictionswere introduced to ensure the physical feasibility of an attack and so realistic ad-versarial accessories (eyeglasses) [30] and patches were proposed [5, 37]. Theseattacks do not assume an attacker has access to the input digital stream, insteadtheir image-based attacks work when a real-life event is captured by a camera. Asfor the second issue of input-specific attacks, the concept of a universal adversarialattack was introduced in [22]. A universal attack uses a single adversarial perturba-tion that works across the input distribution (input-agnostic), i.e. when this singleperturbation is applied to any input to a DNN it can lead to an error in prediction.All the aforementioned work are from the area of white-box adversarial attackswhich assume the complete knowledge of the victim model and its parameters.These attacks often require gradients of all operations performed on the input to beable to learn which minimal perturbations would fool a model, and so it requiresthe model to be entirely differentiable. Out of the scope of our thesis are black-boxattacks which assume only the knowledge of a modelâs prediction to a given inputand tries to develop perturbations that would fool this model without knowing itsinternal parameters.In the field of autonomous driving, perturbing the RGB image input may notbe enough to mount an effective and realistic adversarial attack because most AVsare equipped with an additional sensor: a LiDAR. Successful adversarial attacksusing point cloud perturbations were investigated in [43, 46]. These attacks wouldperturb the input point cloud by adding, removing, or displacing individual points.Such specific digital perturbations are extremely unrealistic due to LiDAR proper-ties. Each point in a point cloud must have a corresponding LiDAR ray, a constraintthese attacks do not consider.Towards more realistic point cloud adversarial attacks in the context of AVs,the work in [2, 39] proposed using an adversarial 3D mesh (with a specific learnedshape) that is placed in the 3D scene and rendered to point cloud using a LiDARsimulation algorithm. They targeted point cloud-based 3D detection models anddemonstrated that such an insertion attack can be effective while being much morerealistic than previous work in point cloud-based attacks.91.3 Motivation and Research ProblemsThe previous section has shown that an adversarial attack is a valid threat to anymodel that relies on DNNs. However, most prior work on adversarial attacks focusonly on one modality (e.g. images or point clouds) and very little work have beendone on multi-modal attacks on models that take as input images and point clouds.This is the first gap we seek to address in our thesis. In the next chapter we reviewthe very few and recent work on multi-modal attacks and explain how our work isdifferent from them. Second, many of the previous attacks proved that an adver-sarial attack is theoretically possible, but did not have a clear real-life applicationor threat. Third, many of the aforementioned adversarial attacks can be nonthreat-ening in real-life for many reasons: i) They directly perturb pixels or points in amanner that is difficult to replicate in real life. ii) They make the weak assumptionthat an attacker has access to the digital input to the model and so physical realiz-ability is not considered. iii) They do not consider the various scene conditions likelighting, pose, or distance and so the attack only works under extremely controlledcircumstances. iv) The attack is tailored to fool the model for a single input sampleand so it is not universal or effective with other input samples.With the deployment of many AI systems in many industries, especially au-tonomous driving, it is crucial to focus on application-specific realistic adversarialattacks. We seek to not just prove that a multi-modal adversarial attack is possiblebut to also utilize these attacks towards a clear objective: informing model designchoices in multi-modal deep learning for 3D detection.This means we are left to answer the following questions:1. Can we design a multi-modal adversarial attack that perturbs multiple inputmodalities simultaneously?2. Can these perturbations be physically realizable and applicable under variousscene conditions? Does the attack represent a real-life threat?3. Can this adversarial attack be used to expose realistic vulnerabilities in multi-modal architectures and thus inform AI design choices in multi-modal deeplearning?101.4 ContributionsAs safety is critical for AVs, this thesis investigates universal and realistic adver-sarial attacks on multi-modal 3D detection models that take both images and pointclouds as input, specifically we consider the common cascaded and fusion archi-tectures. To the best of our knowledge this thesis is among the first works onmulti-modal adversarial attacks on camera-LiDAR models. We chose the appli-cation of car detection in AVs due to its intuitive multi-modal nature (inputs frommany sensors) and safety implications. Unlike many prior works on adversarialattacks we make sure our attack is realistic and physically realizable.As mentioned earlier, an adversarial attack introduces specific changes to theinput with the aim of fooling a model. Our adversarial addition to the 3D scenearound an AV is in the form of a small 3D adversarial object with specific shapeand texture (colors of the object). The adversarial object is placed on top of ahost vehicle to avoid occlusion from other elements in the scene. The aim is thatthis object can make its host vehicle evade detection by multi-modal 3D detectionmodels. Our attack does not make the weak assumption that an attacker has accessto the digital input feed to a perception system, instead the adversarial object isplaced in the 3D scene for the camera and LiDAR to capture it. We show an overalldemonstration of our attack in Figure 1.7. Such an attack can be detrimental to aperception system which in turn can pose grave threats in the safety-critical fieldof autonomous driving. The accuracy of 3D car detection is very consequential inautonomous decision-making and has a major impact on many stakeholders.11Figure 1.7: Overview of the proposed adversarial attack. The goal of this attack is to hide the adversarial objectâs hostvehicle from detection by a multi-modal 3D detector. Notice that the objectâs host vehicle is not detected likeother vehicles. We explain the rendering processes and the models we attack in Chapter 3.12The shape and texture of the adversarial object have trainable parameters thatare perturbed to accomplish the adversarial goal. Learning these perturbations canbe formulated as an optimization problem and approximated using SGD. We designour own differntiable GPU implementation of a ray tracing algorithm that willfunction as a LiDAR renderer. This implementation allowed for processing speedgains in rendering as well as backpropogation through the rendering process.We use our attack to study two main aspects of multi-modal learning in thecontext of AV: First, which architecture type (fusion or cascaded) is more vulnera-ble to realistic adversarial attacks. Second, which input modality (images or pointclouds) contributed most to this vulnerability. This study thus informs researchersand industry professionals in the field of autonomous driving on possible weakpoints in deep learning-based perception systems. It also provides insights to in-form AI design choices (e.g. which architecture to use, or which input modality toutilize) so that they avoid these weak points or mitigate their effects.To summarize, our attack is:â¢ Leading: Among the first works on multi-modal adversarial attacks in thecontext of autonomous driving.â¢ Universal: The adversarial objectâs shape and texture are trained on thou-sands of vehicles from the input distribution. Thus a single adversarial objectcan be effective in hiding any vehicle in any 3D scene from state-of-the-art3D detectors.â¢ Realistic: The graphical representation of the 3D adversarial object is ren-dered to both images and point clouds and so it is consistent across inputmodalities as if it was present in the original scene. To do that we usedmany differentiable rendering tools and algorithms. We designed our owndifferentiable GPU implementation of the LiDAR point cloud renderer.â¢ Effective: Our adversarial object was able to make more than 50% of its hostvehicles evade detection.â¢ Insightful: We conducted experiments to see how different multi-modallearning architectures were vulnerable to our attack. We also study howeach input modality affects robustness.131.5 ChallengesMost of the prior work on adversarial attacks is mainly in the field of image clas-sification where a small image of an object is given to a model to output a classprediction and a confidence score to that prediction. These attacks and their modelsare much simpler when compared to any detection models (2D or 3D) and attackson detection models [18]. A detection model is often much more sophisticated thana classifier as it has to generate box proposals, classify these boxes, and generatea combined score of the confidence in the box prediction and the classification.Moreover, a detector learns an objectâs semantic details under different poses andangles and in different scenes making it much more robust when compared to asimple classifier [18]. This challenge compounds further for 3D detection modelsand especially fusion models.Also, a white-box attack like ours needs all operations done in developing anadversarial attack on a 3D detection model to be differntiable including the modelâsinternal operations. DNNs are inherently differntiable, but the challenge is mainlywith complicated 3D detection pipelines which may include non-differntiable op-erations, and with the rendering process of a 3D object (as will be explained inChapter 3) which is often non-differntiable.We also need to decide on the 3D objectâs representation (mesh, volumetricgrids or voxels, etc.) and the parameters to optimize that will alter this objectâsshape and texture towards adversarial ends. 3D objects have many representa-tions in computer graphics and each has its challenges especially when it comesto rendering. Moreover, the rendering process includes many elements like light-ing, shading, and texture mapping. This requires some knowledge of 3D computergraphics.The rendering process itself is very computationally expensive and requiresgreat memory and processing resources. As an example, LiDAR rendering requireschecking if every ray from the LiDAR would intersect with every triangle of amesh. In our case we have approximately 64,000 rays and 320 faces of a mesh. Andso to render a single mesh to point cloud we require 20,480,000 complex matrixcomputations. Not only do we need to perform millions of operations but we alsomust store the gradients of these operations in memory for backpropogation. Not14to mention the RGB rendering process which is also computationally expensiveand also requires storing of gradients. Finally, the model itself usually has millionsof parameters and operations that also require processing and memory resources.This requirement of extensive resources proved to be an obstacle in our work.Finally, our insertion attack does not cover the victim object at all. Comparedto prior work which often placed perturbations on the object of interest (see Fig-ure 2.1, Figure 2.2, and Figure 2.3) and thus at least partially covering it, our 3Dobject rests on the vehicle and so the vehicleâs pixels and points are never tamperedwith. This can pose a challenge to the efficacy of our attack which seeks to hide itsvictim vehicle without significantly altering the semantic information of a vehicle.1.6 Thesis OrganizationIn the next chapter we explore some of the literature around adversarial attacksespecially in the field of autonomous driving and detail how our work is differentfrom theirs. In Chapter 3 we elaborate on our adversarial object, the renderingtools and algorithms, the selected victim models from the cascaded and fusionarchitectures, and finally the optimization functions. All experimentation detailsare in Chapter 4 along with the results and discussion. In Chapter 5 we summarizeour findings and point to some potential future directions.15Chapter 2Related WorkThis thesis builds upon recent work on adversarial attacks, but it mainly opens upnew directions in the the very novel field of multi-modal adversarial attacks. Asmentioned earlier, most work in the field of adversarial attacks focus only on onemodality, we detail some of this work in sections 2.1 and 2.2, and conclude withthe limited related work on multi-modal attacks in Section 2.3. At the end of eachsection we explain how this prior work was helpful to us and how our work differsfrom theirs. We summarize the related work in Table 2.1.2.1 Image-Based Adversarial AttacksSzegedy et al. [36] first showed the existence of adversarial examples in image clas-sification Deep Neural Nets (DNNs) by optimizing pixel-wise perturbations. Then[8] expanded on that by providing a faster method of generating the perturbations.These attacks were mostly digital pixel-wise perturbations as shown in Figure 1.6,and so they were not strictly realistic and made the weak assumption that an at-tacker would have access to the digital feed to a DNN.Kurakin et al. [12] showed that pixel-wise adversarial attacks can be threat-ening in real-life by printing the adversarial examples and capturing them with acell phone camera. However, this attack is only effective when the image is cap-tured from a single-view (top). Moreover, these attacks would perturb the entireimage including not just the object of interest for classification but also the back-16Figure 2.1: An illustrative example of the adversarial attack on a face classi-fication DNN proposed in [30]. The adversarial perturbations are limitedto the frame of the eyeglasses which are illustrated by the red color.ground (a desk, wall, street, etc.) which is almost impossible to replicate in reallife. Towards more realistic attacks, Sharif et al. [30] proposed utilizing facial ac-cessories, specifically a set of adversarially-colored eyeglasses frames, to fool faceclassification models from correctly classifying certain faces. Here the adversarialperturbations are not added to the entire input image, instead they only perturb theframe of the glasses. An illustration of this attack is shown in Figure 2.1. They alsoproposed a âprintability lossâ to account for color distortions in printing during ad-versarial optimization. Some of these 2D attacks however have been shown to belargely ineffective in the physical world since they do not sufficiently account forthe numerous transformations that occur in the real world like distance to object,lighting, viewing angle, or camera noise [19].Athalye et al. [1] proposed a more robust attack on classifiers by optimizing anadversarial texture on a 3D object over multiple views and transformations. Alongsimilar lines, and in the context of autonomous driving, Eykholt et al. [5] proposedusing adversarial stop-sign posters and adversarial stickers placed on stop-signs tofool road sign classifiers, see Figure 2.2. They optimize these perturbations overmultiple viewing angles and distances. Sticker/patch attacks [4, 5, 37] as well asthe eyeglasses attack [30] can be described as insertion attacks since perturbationsare not over the entire input image, instead an object with adversarial colors isinserted in the scene and captured with a camera.Many of the aforementioned adversarial attacks were specifically crafted for asingle input example and thus the perturbations were largely ineffective on other17Figure 2.2: An illustrative example of the adversarial attack on a road signclassification DNN proposed in [5]. The adversarial perturbations arelimited to the sticker placed on the stop sign. They also proposed aseparate attack similar to Kurakin et al. [12] where they print an entirestop sign with imperceptible perturbations and place it over the originalsign.inputs. To remedy that, [22] proposed universal (image-agnostic) adversarial at-tacks where a single adversarial image of perturbations can be added to any imagein the input distribution and it would be effective in fooling a DNN. Generatingsuch a universal attack mainly required training the adversarial perturbations overthe entire training dataset instead of a single input sample. With the introduction ofuniversal perturbations a clear danger of adversarial attacks was presented wherean attacker can threaten a model in a much wider set of circumstances than everbefore.Most of the initial work on adversarial attacks was mainly concerned with theimage classification task including attacks claiming to represent a real-life threat.However, Lu et al. [18] have shown that some of these classification attacks arenâteffective with a detector that predicts bounding boxes as well as classifies objects.They argued that the proposed classification attacks use cropping as a proxy forbounding box detection which makes their attacks often ineffective in real lifewhere detectors are most likely to be used. No sign detection system on an Au-tonomous Vehicle (AV) crops a sign before classification, because knowing whereto crop is non-trivial, itâs the detectorâs job. Detectors often use background andscene context to propose multiple bounding boxes and then refine their estimations.Moreover, cropping can mitigate the effect of using synthetic transformations to18Figure 2.3: An illustrative example of the adversarial attack on a person de-tection DNN proposed in [37]. The adversarial perturbations are limitedto the square patch placed at the waist. The detected person is shownwith an accurately estimated bounding box and a correct classification,however the person with the adversarial patch is not detected.make classification attacks robust to viewing angles and distances [18, 19]. Thismeans that the proxy cropping procedure used in most classification attacks is of-ten naive. It also means that the task of fooling a detector is probably much moredifficult than fooling a classifier.To explore this new direction, [4, 37] proposed attacks on popular DNN-basedobject detectors. Eykholt et al. [4] extended their prior work on fooling stop-signclassifiers to fooling detectors with relative success albeit with some failure in out-doors testing. Thys et al. [37] presented an example of universal insertion attackson DNN-based person detectors. They used adversarial stickers to hide people wear-ing them at the waist level from a person detection DNN, see Figure 2.3. This in-sertion attack was physically realizable and universal, since a single sticker wasprinted and applied on different people in different scenes and it was effective inhiding them from a powerful detector.Recently, differentiable renderers were used to make realistic adversarial at-tacks that extend beyond simple pixel perturbations and interacts with the physical19world [16, 44, 49]. Zeng et al. [49] targeted image classification and visual ques-tion answering models by perturbing the 3D physical scene parameters (surfacenormals, illumination, material) and then rendering to 2D. Xiao et al. [44] per-turbed a 3D objectâs shape and rendered it using a simple differentiable rendererto fool classifiers and detectors. Liu et al. [16] proposed manipulating an objectâsshape as well as the lighting of the scene. However, scene manipulations [16, 49]are often unrealistic especially in open environments. So an adversarial attack thatonly works under specific lighting requires a very controlled environment and thuscan be less threatning in real life.Many concepts and tools introduced in image-based attacks were helpful tous like universal attacks, and differentiable rendering. However, contrary to mostprior work on image-based attacks, we aim not to learn a sticker or pixel-wise per-turbations but a texture on a 3D object that is effective even when viewed fromdifferent angles and distances. Moreover, most of this work was done on classifi-cation models which are much less complex compared to 2D detection models, letalone 3D detection models. Finally, our attack extends beyond perturbing a singlemodality (image). Our attack is multi-modal i.e. we manipulate the input imageand point cloud with a single adversarial object.2.2 Point Cloud-Based Adversarial AttacksAdversarial attacks on point cloud models were initially explored in [43, 46]. Theseworks designed adversarial attacks using point-wise perturbations. While Xianget al. [43] added new points, or slightly moved all points, Yang et al. [46] alsoconsidered removing some points. While these attacks were very effective in fool-ing point cloud classification models, they were largely unrealistic since they donot account for LiDAR ray casting. Any point in a point cloud must have a corre-sponding LiDAR ray, a fact these attacks did not take into account. So hand-craftedpoint-wise perturbations that were only effective under very controlled settings canbe very unrealistic. Moreover, these attacks were made on relatively dense pointclouds that represent shapes and small objects, which is different from the verysparse nature of point clouds generated by an AVâs LiDAR. Finally, these proposedattacks were point cloud-specific and not universal.20Cao et al. [2] proposed a realistic adversarial attack on point cloud 3D detectionmodels in the context of autonomous driving. They used an adversarial 3D meshthat is placed somewhere in the scene around an AV. When this mesh is renderedusing a LiDAR simulator it produces an adversarial point cloud capable of foolingthe model by either misclassifying the 3D object, or not detecting it at all. Thismeans that the 3D mesh is the subject of detection (or âundetectionâ) and not otherreal-life objects in the scene like a pedestrian or a vehicle. Their adversarial objectis trained on a single input frame from a small in-house dataset, their attack is thusnot universal. In other words, their proposed attack is unlikely to be threatening inother 3D scenes. Moreover, they evaluate their results on their in-house dataset andnot a publicly available benchmark which can make it difficult to generalize theirresults. To remedy these shortcomings, [39] proposed a similar attack howevertheirs was universal (trained on many scenes from a large dataset). Moreover, [39]placed their adversarial mesh on top of a car with the objective of minimizing thedetection score of the host car. This means that the adversarial mesh interacts withreal-life objects in the scene. Their attack was evaluated on different point cloudDNNs using the KITTI benchmark.Similar to the aforementioned realistic point cloud attacks we use a 3D meshas our malicious adversarial object not just for its representation of 3D shape butalso texture. Moreover, the objective functions and LiDAR rendering algorithmsused in these attacks partially guided our attack. However, all these prior attacksfocus only on point cloud detection, while an AV also has a camera sensor. It canbe argued that point cloud-only attacks can be futile since camera inputs are muchmore dense and semantically rich and thus can be more influential in the learningprocess.2.3 Multi-Modal Adversarial AttacksWe found mainly three prior works on multi-modal adversarial attacks in the con-text of autonomous driving. Yu et al. [47] investigated adversarial attacks on seg-mentation networks that input RGB and thermal images. However, their attack wasdone through pixel-wise perturbations which means itâs largely unrealistic espe-cially that they donât consider cross-modality consistency in perturbations. Wang21et al. [41] studied adversarial attacks on camera-LiDAR detection models by shift-ing individual points in the 3D space and using a 2D RGB sticker. As stated ear-lier, simple point perturbations can be unrealistic due to LiDAR sensor properties.Moreover, their fusion attack is mainly focused on 2D detection which is very dif-ferent from our 3D detection attack.Towards realistic multi-modal attacks on perception systems, Tu et al. [40](concurrently with our work) used a 3D adversarial object to attack a fusion-basedBirdâs Eye View (BEV) detector. BEV detection is different from 3D detection in2 main respects: First, BEV detection is done on a 2D plane, estimating only anobjectâs (x,y) location, orientation, width, and length (ignoring the height which3D detection requires). Second, BEV learning is done on projected point cloudswhich leads to some information loss. This can make their attack results harderto extend to fusion multi-modal models that reason on LiDAR points directly like[10, 34]. Moreover, their work does not consider cascaded models which are moreinterpretable and in many cases less computationally expensive than fusion models[3]. This makes cascaded models easier to train, test, and deploy in industry andthus more critical to examine. Our thesis thus presents a chance to compare andcontrast the vulnerabilities of each architecture type (cascaded and fusion). More-over, we report additional attack metrics that can aid in comparing our work tofuture attack and defense methods. Finally, the model they attack and its sensorfusion module are not commonly used or publicly available which makes their re-sults harder to generalize. The two models we consider are both open-source andbuilt upon commonly used backbones as will be explained in Chapter 3.22Related Work Modality Task Perturbation Universal ThreatSzegedy et al. [36] Image Classification pixels - -Goodfellow et al. [8] Image Classification pixels - -Moosavi-Dezfooliet al. [22]Image Classification pixels x -Kurakin et al. [12] Image Classification pixels - -Sharif et al. [30] Image Classification pixels/texture - *Athalye et al. [1] Image Classification pixels/texture - *Eykholt et al. [5] Image Classification pixels/texture x *Eykholt et al. [4] Image 2D Detection pixels/texture x **Thys et al. [37] Image 2D Detection pixels/texture x ***Xiao et al. [44] Image Classification +2D Detection3D shape - **Liu et al. [16] Image Classification 3D shape +lighting- *Table 2.1: Summary of related work. The threat level (more asterisks = higher risk) is based on the possibility of thisattack being a real-life danger. In determining the degree of threat we consider: i) the attackâs physical realizabilityand efficacy in real-world environments, ii) the complexity of the task and its applicability in real life, iii) if it isuniversal (input-agnostic) or not, and finally iv) the reported attack success rates.23Related Work Modality Task Perturbation Universal ThreatXiang et al. [43] Point Cloud Classification point-wise - -Yang et al. [46] Point Cloud Classification point-wise - -Cao et al. [2] Point Cloud 3D Detection 3D shape - *Tu et al. [39] Point Cloud 3D Detection 3D shape x **Yu et al. [47] Thermal +ImageSegmentation pixels - -Wang et al. [41] Point Cloud +Image2D Detection point-wise - -Tu et al. [40](Concurrent)Point Cloud +ImageBEV Detection 3D shape +texturex ***Ours Point Cloud +Image3D Detection 3D shape +texturex ****Table 2.1: Summary of related work. The threat level (more asterisks = higher risk) is based on the possibility of thisattack being a real-life danger. In determining the degree of threat we consider: i) the attackâs physical realizabilityand efficacy in real-world environments, ii) the complexity of the task and its applicability in real life, iii) if it isuniversal (input-agnostic) or not, and finally iv) the reported attack success rates.24Chapter 3Multi-Modal Adversarial AttackWe propose a universal and realistic adversarial attack on 3D detection models thattake images and point clouds as input. Our method is inspired by âstickerâ attacksin the image domain [37] and âmeshâ attacks from the point cloud domain [39],where a single adversarial object attached to a host object can hide its host objectfrom detection. We place a 3D object on a vehicle and render it to the 2D imageand 3D point cloud inputs with the aim of hiding its host vehicle from detectionby a multi-modal 3D detection model. This object has two mutable properties:texture and geometry. Perturbing these properties changes the adversarial objectwhich in turn perturbs the input image and point cloud. These shape and textureperturbations are adversarially learned, i.e. they are optimized towards minimiz-ing the detection probability of the objectâs host vehicle by a victim model. Weexplain how the 3D adversarial object is crafted in Section 3.1. We elaborate onthe rendering processes and why they need to be differentiable in Section 3.2. Wetalk briefly about the models we attack in Section 3.3 and Section 3.4. Finally,the objective functions we use to optimize the shape and texture of the adversarialobject are detailed in Section 3.5.3.1 Adversarial ObjectSince our 3D object needs mutable shape and texture we choose to represent itusing a 3D triangle mesh. Not only can a mesh efficiently contain shape and texture25information, it can also be differentiably and accurately rendered to RGB imagesand point cloud. Moreover, using a mesh allows us to make a physically realizableattack since it can be easily 3D printed and applied on a vehicle.To attack the point cloud reasoning abilities of a camera-LiDAR model, weneed to change the 3D information of the input scene. Our adversarial object shouldhave a specific shape such that its rendered LiDAR points can effectively perturbthe input point cloud in a way that fools the detection model. To learn this shape,and following prior work [2, 39], we start with an initial mesh S with V vertices,where each initial vertex is defined as v0i â R3, i â {1,2, ...,V}. We alter the shapeof the mesh by displacing each vertex with a corresponding displacement vectorvËi â R3. Thus we have V displacement vectors that are learnable parameters tobe optimized towards an adversarial shape for the whole mesh. The displacementvectors are used to produce an adversarial mesh Sadv with vertices vi â R3, asshown in Equation 3.1. This adversarial mesh is first placed at the origin with afixed orientation. We then use ground truth location and orientation of the vehicleto make a 4Ã 4 transformation matrix T that takes the mesh from the origin andputs it on top of a host car and gives it the same orientation as the carâs headingdirection. For the transformation in Equation 3.1 we use the 4D homogeneouscoordinates of the vertices and then project them back to 3D.vi = T · (v0i + vËi) (3.1)We use optimization tools and algorithms to calculate displacement vectors thatproduce an adversarial mesh capable of hiding its host vehicle. It can be seen thatthe shape perturbation process in Equation 3.1 is entirely differentiable (basicallyit consists of matrix multiplication and addition), we will explain in Section 3.2why differentiability is necessary.For the texture of the adversarial mesh, a learnable vector ci â R3 is assignedto each vertex to represent an RGB color. We end up with V color vectors whichare also used as learnable parameters. To produce the texture of the mesh, eachface is colored by interpolating the colors from its three vertices. The mesh is thenrendered and added to the 2D RGB camera space with a given projection matrix.Our criticism of prior work on adversarial attacks was mainly that it was unre-26alistic. To ensure our adversarial object is realistic we put constraints on its size soit can fit conspicuously on top of a vehicle. Compared to prior work [39, 40] ouradversarial object is smaller as will be shown in Chapter 4. Also towards a realisticattack we ensure the geometry of the mesh is smooth by adding a âsmoothing lossâto our optimization function as will be detailed in Section 3.2. Finally, since i) weuse a mesh with a small number of vertices in our experiments (as will be explainedin Chapter 4) and ii) we interpolate the color of each face from its vertices, we geta realistic smooth texture without abrupt changes in color. Such abrupt changes incolor are typically observed in pixel-wise perturbations [8] or dense texture repre-sentations [40] which produce very complex color patterns that could be difficultto print.3.2 RenderingTo design a realistic attack we need rendering to simulate the presence of the 3Dadversarial object in the original scene captured in the dataset. Since we havetwo input modalities we need separate rendering procedures that would simulatethe capturing of the 3D object by a camera and by a LiDAR. The reason behindsimulation is to enable the learning of a specific and realistic shape and texture ofthe adversarial object that would achieve our adversarial goal of hiding the objectâshost vehicle from detection. The optimization of the shape and texture of the objectrequires the gradients of all operations performed on the parameters vi and ci, andso we need the rendering process itself to be differentiable. We use these gradientsto perform the gradient descent method to minimize our objective functions.3.2.1 LiDAR Point Cloud RenderingTo realistically render a mesh to point cloud, we need to find which LiDAR rayswould hit the mesh if it was placed in the scene when the original point cloud wascaptured. An overview of the point cloud rendering process is shown in Figure 3.1.The mesh is nothing but its triangles (faces) so we need a ray-triangle intersec-tion algorithm that finds exactly where rays intersect with triangles. We use theMo¨llerâTrumbore algorithm [21] since itâs fast and does not require too much stor-age and most importantly, it is differentiable. Consider a triangle of a mesh with27Figure 3.1: Overview of the LiDAR point cloud rendering process. We usethe Mo¨llerâTrumbore ray-triangle intersection algorithm [21] for differ-entiable rendering to point cloud. To accurately simulate the LiDARused in capturing the dataset, we produce unit length rays using thesame angular frequencies of that LiDAR. The mesh is placed on top ofa host vehicle in 3D world coordinates. The red dots on the 3D meshare the calculated intersection points between the meshâs triangles andthe simulated LiDAR rays. Once the intersection points are calculatedwe add them to the original sceneâs point cloud.vertices V0V1V2 â R3, and a LiDAR ray R â R3 that starts from the origin O â R3with a direction D â R3 and a distance to the intersection point t â R. Then aray-triangle intersection point P â R3 can be expressed as follows:P= O+ tD= wV0 +uV1 + vV2 (3.2)Wherew,u,vâR are the barycentric coordinates of the intersection point insidethe triangle and thus:w+u+ v= 1 (3.3)must be true, which means that w= 1âuâ v, and 0â¤ u,v,wâ¤ 1. To solve for thethree unknowns t,u,v from Equation 3.2, the Mo¨llerâTrumbore algorithm [21] usesCramerâs rule from linear algebra. Let E1 =V1âV0, E2 =V2âV0 and T =V0âOthen we simply need to solve:28ï£®ï£¯ï£° tuvï£¹ï£ºï£»= 1D · (E1ÃE2)ï£®ï£¯ï£° T · (E1ÃE2)(DÃE2) Â·TT · (E1ÃD)ï£¹ï£ºï£» (3.4)We check the intersection points between each LiDAR ray and each mesh tri-angle and only take intersection points that fulfil 0 â¤ u,v,w â¤ 1 which indicatesthat a point is within the triangle plane. If there are multiple correct intersectionpoints for a single ray (due to intersection with multiple mesh faces) we choosethe nearest intersection point. We finally add all the resulting points to the LiDARpoint cloud scene. The matrix operations in Equation 3.4 are differentiable and sothe backpropogation of gradients can pass through the LiDAR rendering processto directly optimize the shape of the adversarial object. We design a GPU imple-mentation of this algorithm resulting in a significant speed gain in the renderingprocess. In Figure 3.3 we show the adversarial mesh placed on all the vehicles inthe 3D scene, using the transformation matrix T of each vehicle, and rendered topoint cloud.3.2.2 RGB Image RenderingFor RGB image rendering we use the fast, differentiable rendering tools developedin [27]. An overview of the process is shown in Figure 3.2. We utilize cameraintrinsic and extrinsic information provided with the dataset for realistic and accu-rate projection from 3D to 2D. Realistic rendering of 3D meshes requires shading,which is a process in rendering that uses depth information as well as lightingand material information to produce photo-realistic images. We use Phong shad-ing which is one of the most common mesh shading algorithms used in computergraphics. Finally, we simulate sunlight by pointing a light source from far abovethe scene at a 45â¦ angle. The rendering process produces the pixels of the ren-dered mesh as well as a mask that indicates which pixels are occupied by the 3Dobject. We use this mask to clear the pixels in the original image, and then add therendered objectâs pixels to the original scene as shown in Figure 3.2. An exampleof the overall results from rendering multiple meshes to image and point cloud isshown in Figure 3.3.29Figure 3.2: Overview of the RGB image rendering process. We use the toolsdeveloped in [27] to handle differentiable rendering to RGB images. Toaccurately simulate the camera used in capturing the dataset, we providethe renderer with the intrinsic and extrinsic parameters of that camera.We use Phong shading to get realistic colors and we also simulate sun-light by pointing a light source from above.3.3 Cascaded AttackWe choose Frustum-PointNet (F-PN) [26]1 as a representative of cascaded modelsfor many reasons: First, itâs one of the early cascaded models and many moreworks [32, 42] were inspired by its architecture. Second, it uses PointNet [24] as abackbone for point cloud reasoning. PointNet [24] is a very popular Deep NeuralNet (DNN) for learning with point cloud inputs. This means that our attack couldbe a threat to many other detection models that have a similar architecture to that ofF-PN [26] or use a PointNet [24] backbone. Finally, F-PN [26] showed competitiveresults on the KITTI benchmark.1publicly available at https://github.com/charlesq34/frustum-pointnets30Figure 3.3: Results of rendering the adversarial mesh to 3D point cloud and2D RGB image. To illustrate the mesh placement we represent the 3Dscene by its point cloud. The LiDAR ray-triangle intersection pointsare shown in red in the point cloud. The rendered mesh in the image isslightly bright because we simulate daylight.31As shown previously in Figure 1.3, a cascaded model like F-PN [26] deals witheach input modality separately. First, F-PN [26] takes the RGB image through a2D detection DNN to propose 2D bounding boxes where objects may reside. Thesebounding boxes are then projected to 3D space, thus producing frustum-shaped 3Dsearch spaces (narrow near the LiDAR and wide away from the LiDAR) that sur-round each object in 3D. Points within each frustum are extracted and sent througha PointNet-based [24] DNN for 3D instance segmentation deciding which pointsbelong to an object and which belong to the background. Finally the segmentedpoints are sent through another PointNet-based [24] DNN for 3D bounding boxestimation.For the image detection pipeline in F-PN [26], we use YOLOv3 [28]2. It isa standard fast 2D object detection DNN that outputs 2D region proposals, andconfidence scores. The F-PN [26] and YOLOv3 [28] models were pre-trained onthe KITTI dataset.The attack pipeline for the cascaded model is shown in Figure 3.4. We divideour attack to 2 stages, as F-PN [26] has two separate pipelines: RGB-based detectionand point cloud-based detection. We first use the point cloud pipeline to learn theshape of the mesh that would greatly reduce the accuracy of 3D detection. Oncewe have an adversarial shape we move to the next stage: we render it to 2D imagesand learn the adversarial texture that would fool the 2D detection pipeline. Asmentioned earlier, the texture is produced via interpolation of per-vertex colors.F-PN [26] directly outputs one 3D bounding box estimation from a given pointcloud frustum, since the assumption is that there must be a single object in a frus-tum. This implicit bias can pose a challenge for adversarial attacks that seek to sup-press detection by targeting only the point cloud pipeline. Also, the post-proposalpoint cloud is very sparse and small in size so there is not much maneuvering spacefor our realistic adversarial attack to take place. Despite these apparent challengesour attack was successful as discussed in Chapter 4.2publicly available at https://github.com/packyan/PyTorch-YOLOv3-kitti32Figure 3.4: Attack pipeline for Frustum-PointNet (F-PN) [26]. The dashed lines represent the flow of gradients (back-propogation) from prediction to parameters. Purple is for texture-perturbing gradients, and red is for shape-perturbing gradients. The shape of the adversarial object is learned first from the 3D detection stage. Then thetexture is learned from the 2D detection stage. As seen in the end of the pipeline, the adversarial objectâs hostvehicle is undetected. The adversarial object shown can be considered âunder trainingâ.333.4 Fusion AttackUnlike the cascaded model, fusion models take image and point cloud inputs si-multaneously, and then learns and fuses their features in parallel. From the fusionarchitecture we attack the multi-modal 3D detection model EPNet [10]3 shown inFigure 1.5. EPNet [10], which was published recently, uses a novel fusion modulethat enhances point features with semantic image features at multiple scales downthe network. This allows the model to directly utilize the rich semantic informa-tion of images to complement the 3D information from the point cloud. EPNet [10]operates directly on LiDAR points not their projection to Birdâs Eye View (BEV)to avoid losing fine geometric information which can lead to missing objects incrowded scenes. Its geometric stream takes the point cloud input and learns itsfeatures on multiple abstraction levels using modules developed in [25]. Its im-age stream uses 2D Convolutional Neural Nets (CNNs) to extract image features onmultiple scales. The extracted image features are fused to the point features usingthe fusion module proposed in [10].After fusion, the enhanced point features are sent through DNNs for point seg-mentation and 3D bounding box proposal generation. EPNet [10] generates many3D proposals for each foreground point of a car and then uses a refinement networkto filter proposals and refine bounding box dimensions. Since the fusion model rea-sons on both modalities simultaneously, we can train the shape and texture of ouradversarial object simultaneously and end-to-end as in Figure 3.5.We note that EPNet [10] is similar to the popular point cloud-only modelPointRCNN [31] in one key respect: PointRCNN [31] also generates many pro-posals per foreground point of a car and thus generating a significant number ofproposals for each car. Due to that, a previous point cloud-only adversarial attack[39] found it very difficult to suppress all these proposals in their single-modalityattack. It can be said that PointRCNNâs [31] utilization of many proposals for eachvehicle made it relatively robust to realistic adversarial attacks. This is probablytrue since it was shown in [39] that the closer a vehicle is to the LiDAR, the harderit was to hide it from detection. This is because a close vehicle registers manypoints and thus produces much more proposals. EPNetâs [10] fusion of image fea-3publicly available at https://github.com/happinesslz/EPNet34tures with point features gave it a better detection accuracy compared to that ofthe point cloud-only PointRCNN [31]. It can be interesting to see if and how thisutilization of image features affects the robustness of a fusion model thatâs similarto an apparently robust point cloud-only model.3.5 Objective FunctionsWe follow many prior work [30, 37, 39] in formulating our adversarial attack asa stand-alone optimization problem, but in our case the parameters are shape de-formation vectors and per-vertex color vectors. We optimize those parameters tominimize the modelâs detection of the adversarial objectâs host vehicle.Each model architecture has its own specific objective functions but the com-mon thing among them is that we aim to minimize the detection score (given bythe model to a detected host vehicle) using Binary Cross Entropy (BCE) which isa commonly used loss in classification and detection problems. Car detection is abinary problem where a detected car can have a label = 1 and anything else has alabel = 0. The BCE formula is shown in Equation 3.5.BCE =âlabel â log(score)â (1â label)log(1â score) (3.5)Since we want to minimize detection we can just assign every host vehicle alabel = 0 and so the BCE loss increases with every detection and decreases with amissed detection. So we can reformulate the BCE loss as shown in Equation 3.6.BCEadv =âlog(1â score) (3.6)To ensure the adversarial mesh is smooth and realistic we add a smoothingloss to all shape-optimizing objective functions. We follow prior work [17, 39]in minimizing a Laplacian loss shown in Equation 3.7, where δi is the distancebetween a vertex vi and the centroid of its neighbors N(i). This loss aids in avoidingcases where vertex displacements produce wildly unrealistic meshes.Llap =âiâÎ´iâ22 where δi = viâ1âN(i)â âjâN(i)v j (3.7)35Figure 3.5: Attack pipeline for EPNet [10]. The dashed lines represent the flow of gradients (backpropogation) fromprediction to parameters. Purple is for predominantly texture-perturbing gradients, and red is for predominantlyshape-perturbing gradients. As seen in the end of the pipeline, the adversarial objectâs host vehicle is undetected.The shape and texture of the object are trained simultaneously and end-to-end. The adversarial object shown canbe considered âunder trainingâ.3.5.1 Cascaded ModelSince F-PN [26] has two pipelines (image and then point cloud) and our attack is intwo stages, we develop two objective functions for each pipeline. The first objec-tive function alters the texture of the adversarial mesh to attack the 2D detectionpipeline. Our 2D detector YOLOv3 [28] provides an objectness score with everyregion proposal indicating the confidence of the model that this region contains anobject. We use BCE to minimize the objectness scores given to car detections. Soif a region proposal is classified as a car and given an objectness score os then ouroptimization function for the 2D pipeline of the cascaded model is:Lyolo =ââ1 · log(1âos) (3.8)The point cloud pipeline of F-PN [26] takes as input the LiDAR points p â R3in the frustum extruded from 2D detection. F-PN [26] then uses its PointNet-based[24] segmentation network to classify all input points as either background points0(p) or car points 1(p) and giving each point a segmentation score s(p) â (0,1)indicating the modelâs confidence in its classification. Since F-PN [26] does notexplicitly define a score for its detections, we tried many score definitions, forexample the sum of segmentation scores given to all car points. We found that themost effective objective was to minimize the maximum score given to a âcarâ pointin a point cloud.m= maxpâ1(p){s(p)} (3.9)We use BCE to minimize this score as shown in Equation 3.10. Following pre-vious work [39], we weigh the objective function by the rotated Intersection-over-Union (IoU) score of the detection. An IoU score indicates how well the predictedbounding box yË fits the ground truth ygt bounding box.L fâpn =ââ1 · log(1âm) · IoU(ygt , yË) (3.10)As shown in Equation 3.11, IoU is calculated by finding the overlap area be-tween the two boxes and dividing it by their union area, all from a BEV perspective.We calculate the IoU from this perspective because it focuses on location, width,37length, and rotation prediction and ignores the height which can be noisy becauseof the adversarial objectâs placement. A high IoU score means the predicted bound-ing box fits the car well and so using it as a weight in our loss function means wediscourage such a good fit and put a high cost on good detections which in turnencourages missing detections.IoU(ygt , yË) =Intersection(ygt , yË)Union(ygt , yË)(3.11)The overall objective function for the point cloud pipeline in the cascaded F-PN[26] in shown below in Equation 3.12, where λ is a weight coefficient for theLaplacian smoothing loss.L fâpnadv = L fâpn+Î»Llap (3.12)3.5.2 Fusion ModelAs mentioned earlier, for the fusion model we can optimize the shape and textureof our adversarial object end-to-end without requiring separate stages. This meansthat only one objective function is minimized to craft the adversarial object.EPNet [10] generates multiple bounding box proposals for each detected ve-hicle. Each proposal has an associated score s indicating EPNetâs [10] confidencein its detection. Following prior work [39], we aim to suppress all bounding boxproposals yË â Y , where Y is the set of proposals with scores s > 0.1. We use BCEto minimize the scores given to proposals as shown in Equation 3.13. As we did inthe objective function of the cascaded model, we also weigh the fusion objectivefunction by the IoU score.Lepnet = 1|Y| âs,yËâYâ1 · log(1â s) · IoU(ygt , yË). (3.13)The overall adversarial attack objective function for EPNet [10] is shown inEquation 3.14 below.Lepnetadv = Lepnet +λLlap (3.14)38Chapter 4ExperimentsWe conduct experiments aimed at evaluating the susceptibility of each architecturetype (cascaded and fusion) to our adversarial attack. We also conduct an ablationstudy to evaluate the effects of each input modality (images and point clouds) onthe robustness of a multi-modal 3D detection model. In this chapter, we describethe dataset we use to train the shape and texture of the 3D adversarial object, andalso test the effectiveness of our attack. We also explain the metrics we use toevaluate our attack, describe the implementation details of our experiments, andfinally report and discuss our results.4.1 Experimental Setup4.1.1 DatasetThe most popular dataset and benchmark in the field of autonomous driving is theKITTI dataset [6]. The 3D detection benchmark of this dataset consists of 7481scene images, and their corresponding LiDAR point clouds from the perspectiveof an ego Autonomous Vehicle (AV). It also includes 7518 test samples, howevertheir annotations are not publicly available so they are mainly used for the officialevaluation of models by the benchmark administrators. The dataset also providesthe intrinsic and extrinsic camera parameters. These parameters are necessary forprojection between images and point clouds and for rendering.39For the training set, we use 3,712 image and point cloud samples, and for thevalidation set, we use 3,769 samples. This split is similar to that of many priorwork [10, 26]. All reported results in the following sections are from the validationset. Since our attack is a universal attack we train the shape and texture of the 3Dmesh on all the vehicles in the training set.4.1.2 MetricsWe report multiple metrics to show the efficacy of our attack and to help us ex-tract meaningful insights from our experiments. We first report the attack successrate, which can be defined as the percentage of vehicles detected normally thatare undetected after placing the adversarial object on them. We count a vehicle asâdetectedâ if its Intersection-over-Union (IoU) score is greater than 0.7. Meaningthat there is significant overlap between the predicted bounding box and the groundtruth bounding box to warrant counting this car as âdetectedâ. It should be notedthat we calculate the rotated IoU from a Birdâs Eye View (BEV) perspective sincethis gives us a score independent from the bounding box height prediction whichcan vary greatly because of the adversarial objectâs placement on top of the vehicle.We also provide recall-vs-IoU plots. Recall is the number of host vehicles de-tected over the number of all host vehicles in the validation set. By sweepingdifferent IoU thresholds and measuring recall we can get a sense of how effectiveour attack is. For example, if applying our adversarial object to vehicles in thevalidation set produced a drop in recall even when the IoU threshold is very smallit means the model did not detect the host vehicle at all (not even inaccurately). Aswe increase the IoU threshold we make stricter requirements for detection and sowe can get an idea of how our adversarial object affects accurate detection.After placing the adversarial mesh on all the vehicles in the validation set wereport a modelâs Average Precision (AP) in detection from the BEV perspective.We should note that this is not 2D BEV detection, but 3D detection results simplyviewed from above. The AP is calculated by finding the area under the precisionvs. recall curve. We use a new KITTI evaluation protocol proposed in [33] thatuses 40 recall points in calculating the AP instead of the usual 11 to get a betterapproximation of the AP. Detection in the KITTI [6] benchmark is divided into 340Figure 4.1: The initial grey spherical mesh with 162 vertices and 320 faces.difficulty levels depending mainly on occlusion with âEasyâ being fully visible,âModerateâ is partly occluded, and âHardâ is difficult to see. A car counts asdetected if the IoU between its predicted bounding box and the ground truth bound-ing box is greater than 0.7. Reporting the AP metric provides an insight into therobustness of a detection model by comparing its AP on the validation set beforeand after an attack. It also helps in accounting for occlusion and its effect on APunder our adversarial attack. This metric can also aid in comparing our results tofuture adversarial attack and defense methods.4.1.3 Implementation DetailsWe have two attacks, one on the cascaded model, and the other on the fusion model.For both attacks we start with an initial grey spherical mesh shown in Figure 4.1.This mesh has a 40 cm radius and comprised of 162 vertices and 320 faces. Un-constrained optimization could produce meshes that can be successful at hidingcars but wildly unrealistic. To avoid this case we deform the meshes using Equa-tion 3.1 and maintain box constraints during optimization using projected gradientdescent. This ensures that adversarial deformation cannot make the mesh exceed apredefined height, width, and length. All optimization is done using the StochasticGradient Descent (SGD) optimizer ADAM [11].41Figure 4.2: The rendered initial spherical mesh which has a diameter of 80cm. The sphere is brighter than the one shown in Figure 4.1 because ofsimulated sunlight and Phong shading.To study the influence of input modalities we conduct an ablation study wherewe disable the perturbations of one modality and attack the other. We call these at-tacks either âImage onlyâ or âLiDAR onlyâ attacks. There is a slight difference inhow these two attacks are conducted between the cascaded and fusion case. We ex-plain this difference in the following subsections. As a control in our experimentswe apply the initial spherical mesh on host vehicles and report the resulting met-rics as a baseline. This can help in better analyzing the influence of our adversarialobject. The rendered initial mesh is shown in Figure 4.2.Cascaded Model ExperimentAs mentioned in Section 3.3, we start with the point cloud pipeline to learn theshape of the mesh. The point cloud pipeline takes a small number of points be-cause of the post-proposal frustum extraction. This means we canât have a largemesh since it can produce a high number of points after rendering and so by con-tributing so significantly to the point cloud input our attack will no longer be a42simple perturbation and the adversarial object will have undue influence on themodel. Thus we apply box constraints on the meshâs size to keep it below 80 cmin width, length, and height. This mesh is relatively small compared to prior work[39, 40]. We use the ground truth 2D annotations as region proposals to gener-ate the point extraction frustums used in training and evaluating the attack on thepoint cloud pipeline. We minimize the objective function in Equation 3.12 usingan ADAM [11] optimizer.Once the adversarial objectâs shape is trained we move on to training the tex-ture. We use an ADAM [11] optimizer to minimize the 2D detection objectivefunction in Equation 3.8. Since we use region proposals from the 2D detector ingenerating the frustums, a missed 2D detection means a frustum is not generatedand subsequently 3D detection fails.In the âImage onlyâ attack we render our adversarial object to images andextract post-attack frustums for 3D detection without rendering our object to pointcloud. As for the âLiDAR onlyâ and âSphereâ attacks we use ground truth 2Dregions (for frustum generation) without perturbation to the image input, but renderthe adversarial object to point cloud and add it to the input points. While this cross-modality inconsistency is unrealistic, it is necessary for the ablation study. Weuse these attacks to see which input modality caused higher vulnerability. For theâImage+LiDARâ attack, the shape produced from the âLiDAR onlyâ attack andthe texture produced from the âImage onlyâ attack are combined to produce thefinal realistic adversarial object. The resulting mesh from this joint attack is shownin Figure 4.3 from multiple angles.Fusion Model ExperimentFor the fusion experiment we relax some of the box constraints. Unlike the cas-caded model which takes a small point cloud, the fusion model takes the whole3D scene and so we allow a slightly larger adversarial mesh by keeping width andlength under 120 cm and height remains under 80 cm. Optimization here runsend-to-end and so shape and texture are simultaneously optimized by minimizingEquation 3.14. This joint optimization produces the final adversarial object whichis shown from multiple views in Figure 4.4.43Figure 4.3: Multi-view illustration of the adversarial object from our multi-modal attack on the cascaded model.44Figure 4.4: Multi-view illustration of the adversarial object from our multi-modal attack on the fusion model.45Figure 4.5: The adversarial mesh produced from our âImage onlyâ attackrendered to image and point cloud.For the âLiDAR onlyâ attack on the fusion model, we learn an adversarialshape for the mesh while keeping the texture a constant grey. In other words, theparameters for this optimization are just the vertex displacement vectors. For theâImage onlyâ attack, we learn an adversarial texture for the mesh while keepingthe shape a constant sphere. The resulting adversarial objects from these attacks arerendered and shown in Figure 4.5 and Figure 4.6. The rendering of the multi-modalattackâs adversarial mesh is shown in Figure 3.3.4.2 Recall & Attack Success RateOur realistic adversarial object was capable of hiding its host vehicle from detec-tion by both models more than 50% of the times as shown in Table 4.1. Thisvulnerability can be mainly blamed on the image input, where every attack thatinvolved perturbing the image input resulted in a steep drop in the recall across IoUthresholds as shown in Figure 4.7.It can be clearly seen in Table 4.1 that the fusion model was very robust toâLiDAR onlyâ attacks compared to the cascaded model. This can be due to the fact46Figure 4.6: The adversarial mesh produced from our âLiDAR onlyâ attackrendered to image and point cloud.Attack Type Cascaded: FP-N [26] Fusion: EPNet [10]No Attack (Sphere) 24.39% 5.45%LiDAR Only 34.24% 5.91%Image Only 42.8% 48.64%Image+LiDAR 55.60% 63.19%Table 4.1: Attack success rates for the two models with IoU = 0.7.that the fusion model relies on dense image features to help with detection whilethe cascaded model does not utilize any image features and relies instead on pointcloud features for bounding box estimation. The cascaded modelâs use of regionproposals to limit the 3D search space, while it has some computational benefits, itproduced 3 main weaknesses:1. It neglects useful and dense image features. Image features are useful in acascaded model only for the region proposal task, but it does not contributedirectly to 3D bounding box estimation. This is a waste of a significant47Figure 4.7: Victim vehicle recall vs. IoU thresholds plot. The fusion modelis very robust to LiDAR-only attacks especially when compared to thecascaded model. Both models are very vulnerable to image attacks andimage+LiDAR attacks. Overall the fusion model is more robust thanthe cascaded model.amount of semantic information that a fusion model utilizes much more ef-fectively and this utilization made it more robust to âLiDAR onlyâ attacks.2. All DNN-based approaches are data-driven and exposure to a lot of real-lifedata is important for robustness and generalization. The cascaded modelâsuse of frustums that just encapsulate a vehicle means it is not exposed toobjects around or on cars in the scene. Which means alien objects in thescene can be damaging to the accuracy of the model. Just placing a sphereon a vehicle dropped the recall of the cascaded model as shown in Figure 4.7.The cascaded modelâs lack of exposure to various elements in the 3D pointcloud scene and its consequent failure in 3D detection can be dangerous inthe context of autonomous driving.3. Since the cascaded model utilizes fewer points inside a frustum instead ofall the points from the whole scene, the points of an object on the vehiclecan represent a significant amount of the input point cloud. This meansthat the cascaded model can be sensitive to any object on the vehicle sinceany object can contribute significantly to the input. Sensitivity to objects48on vehicles is further proven by the high percentage of vehicles undetectedwith just a sphere on the car as shown in Table 4.1. This is also reflectedin Figure 4.7 where a sphere was able to significantly drop recall across IoUthresholds and especially at the higher thresholds. Recall dropped furtherwith the placement of the adversarial object. This is especially dangerousbecause many vehicles carry objects on them like kayaks or bikes and theseobjects being on the vehicle seem to confuse a cascaded model. Fixing thisproblem in the context of DNNs can be as simple as training a model witha dataset where objects are on vehicles. Which can be true, however, thefusion model which is trained on the exact same dataset was not sensitive toany object on the vehicle, and was very robust to âLiDAR onlyâ attacks.We mentioned in Chapter 3 that the cascaded modelâs implicit assumption thatan object must be in a frustum can make it robust. However our attackâs successrates as shown in Table 4.1 have proven otherwise. Looking at Figure 4.7, we cansee that recall in the cases of âNo Attackâ, âSphereâ, and âLiDAR onlyâ startsalmost at the same point at very low IoU thresholds. This means the cascadedmodel always seems to estimate a bounding box near the vehicle, however thisestimation is not always close to the correct estimation (high recall at low IoU).This as well as the steep drop in recall with the âImage onlyâ attack prove thatthe bottleneck of the cascaded modelâs performance is the region proposals fromthe image input. All region proposals get a detection (even an incorrect one asdemonstrated by the relatively similar recall at low IoUs after some attacks) but ifthe region proposals miss a vehicle, this vehicle is never detected which is a glaringweakness since vehicles are often missed in images due to occlusion, overexposure(high brightness), or underexposure (low brightness) as shown in Figure 1.4. Whatfusion models do in incorporating the depth information from the LiDAR in theregion proposal process often compensates for some of these issues.Both models were greatly vulnerable to image-based attacks. As shown inFigure 4.7, the âImage onlyâ attacks which only perturb the input image weresuccessful in hiding more than 40% of the host vehicles from detection by bothmodels. This vulnerability can be due to the inability of image features to learndiscernible characteristics of the input that are robust to such simple perturbations.49Model Model Type Attack Success RatePointRCNN [31] LiDAR-only 32.3% [39]âEPNet [10] camera-LiDAR 63.19%â uses an adversarial object that could be slightly larger than the one used to attack EPNet [10]Table 4.2: Attack success rates of two similar models but one is multi-modaland the other is LiDAR-only with IoU = 0.7.This brittleness is a problem generally in deep learning but it is exacerbated by thedense features of images. While the image features lend robustness to the fusionmodel against simple perturbations in the point cloud input, they themselves werethe weak point of vulnerability with regards to image perturbations.Moreover, looking at Table 4.1 it can be seen that the cascaded model wasslightly more robust to âImage onlyâ attacks compared to the fusion model. Thiscan be caused by the cascaded modelâs utilizing of mature 2D detectors which canbe relatively robust to a realistic and limited attack such as ours. Compared to manyprior work in realistic image-based insertion attacks where the object of interest ispartially or totally covered by a patch, poster, or a sticker, our adversarial meshdoes not cover the object at all, it rests above it. This placement above objects canreduce our adversarial attackâs ability to hide objects from mature 2D detectors inthe case of the cascaded modelâs attack.On both models, our multi-modal adversarial attack is more threatening thanthe âImage onlyâ attack but it follows the same trend of being slightly more robustin the cascaded case. The fusion modelâs vulnerability demonstrated in the âImageonlyâ attack was probably exacerbated with the larger adversarial mesh shown inFigure 3.3. This vulnerability can be blamed on the fusion of the corrupt imagefeatures to a larger set of points produced from the larger mesh which probablygave the corrupt adversarial features an even bigger influence.We mentioned in Section 3.4, that the the LiDAR-only PointRCNN model [31]is similar to the camera-LiDAR fusion model EPNet [10] in that they both use avehicleâs foreground points to generate many proposal per vehicle. Tu et al. [39] at-tacked PointRCNN [31] using an adversarial object on a vehicle and differentiableLiDAR rendering. While their adversarial object could be slightly larger than ours50(their box constraints were 170Ã170Ã100 cm instead of our 160Ã160Ã80 cm)we still use their reported attack success rate for PointRCNN [31] to compare withour results for EPNet [10] as shown in Table 4.2. A multi-modal adversarial at-tack on EPNet [10] was 30% more successful than a point cloud-only attack onPointRCNN [31]. While utilizing image features gave EPNet [10] a higher re-ported accuracy than PointRCNN [31], it also made it much more vulnerable toadversarial attacks. PointRCNNâs [31] robustness was attributed to the utilizationof foreground points for proposal generation [39] and thus closer vehicles whichregister a high number of points were the most difficult to hide. In Appendix Ap-pendix A we show examples where the adversarial objectâs host vehicle is hiddenfrom detection by the fusion model even when the host vehicle is very close to theego vehicle.To demonstrate the efficacy of our multi-modal attack on the fusion modelwe first show in Figure 4.8 the detection performance of the fusion model undernormal circumstances without our adversarial attack. We then place our adversarialobject on a single vehicle in the scene and show how the object effectively hides itshost vehicle from detection by the fusion model in Figure 4.9. Further qualitativeexamples of our multi-modal attack on both models can be found in Appendix A.4.3 Effect on Average PrecisionTo study the effect of our adversarial attack on a modelâs overall detection perfor-mance we apply the adversarial object to all vehicles and report each modelâs APafterwards. Table 4.3 shows the AP of the cascaded model without any attacks,after the ablation attacks we described earlier, and finally after our multi-modalattack. Table 4.4 shows the same information but for the fusion model. The cas-caded modelâs AP dropped from 91% under the âEasyâ scenario to 29.04% a near68% reduction in accuracy. The fusion modelâs AP also dropped by 77% under theâEasyâ scenario.Table 4.3 and Table 4.4 show a similar trend to the one seen in the recall data.LiDAR-only attacks were much more threatening to the cascaded model, while thefusion model was very robust to them. The cascaded model was generally sensi-tive to objects on the vehicle as seen with the AP drop when a sphere is used. Both51Figure 4.8: An example of the fusion modelâs detection without our adver-sarial attack.Figure 4.9: An example of the fusion modelâs detection after placing our ad-versarial object on a vehicle. The adversarial objectâs host vehicle isundetected.52Attack Type Easy Moderate HardNo Attack (Clean) 91.17 85.08 81.00No Attack (Sphere) 52.81 51.46 48.19LiDAR Only 37.36 37.79 39.10Image Only 50.71 36.17 31.07Image+LiDAR 29.04 23.52 19.66Table 4.3: Cascaded: F-PN car detection AP results.Attack Type Easy Moderate HardNo Attack (Clean) 95.94 88.83 88.50No Attack (Sphere) 94.74 87.47 84.58LiDAR Only 93.92 87.02 82.56Image Only 37.99 41.32 37.25Image+LiDAR 22.05 25.63 22.58Table 4.4: Fusion: EPNet car detection AP results.modelsâ AP dropped significantly when the image input is perturbed. However, theâImage onlyâ attack on the cascaded model was not as threatening especially underthe âEasyâ scenario when compared to the fusion model. This can be attributed tothe fact that under the âEasyâ scenario the car is clearly and entirely visible andnot occluded. This can make suppressing 2D detection with just the texture of anadversarial object placed on the vehicle difficult compared to the other occlusionlevels. This also strengthens our initial hypothesis that utilizing mature 2D de-tectors helped the cascaded model mitigate the threat of image-perturbing attacks.On the other hand the fusion model fails almost equally under all occlusion levelswhich can be due to its relying on LiDAR input which often mitigates the effectsof occlusion.In the fusion modelâs case, we notice a slight discrepancy between the attacksuccess rate shown and AP shown in Table 4.4, where successfully hiding 63% ofthe vehicles in the multi-modal attack as shown in Table 4.1 resulted in a slightlyhigher drop in AP. If we calculate the mean of the AP under different difficulties (53we notice a nearly 75% drop in AP between the clean model and the model underour multi-modal attack. This discrepancy can indicate that the multi-modal attacksintroduced some false positives in detection that reduced the AP in addition to justmissing host vehicles (false negatives). While our attack did not aim to generatefalse positives, they can happen due to either false positives that the model itselfgenerates even without our attack, or the projection of image features to LiDARpoints in a distance as reasoned in [40]. This usually happens when the LiDARpoints beyond an adversarial object are very close to it in 2D image space due tothe perspective projection used in most cameras. This projection effect compoundswith the abstractions inherent to DNNs which can allow corrupt adversarial featuresto have an outsized effect.Moreover, we notice in Table 4.4 that the multi-modal âImage+LiDARâ at-tack reduced the AP significantly compared to the âImage onlyâ attack. This isimportant because the âLiDAR onlyâ attack was not so effective suggesting thatadversarial shape perturbations may not be very detrimental to fusion models. Thenearly 15% drop in AP between the âImage onlyâ attack and the multi-modal âIm-age+LiDARâ attack is most likely caused by the âImage+LiDARâ mesh whichis certainly larger than the sphere mesh used in the âImage onlyâ attacks. Thesemeshes can be seen after rendering in Figure 3.3 and Figure 4.5. So the âIm-age+LiDARâ mesh takes a larger space of the input image and thus can corruptmore LiDAR point features after projection. This further proves a key weaknessin fusion models where the projection of easily corruptible image features to pointfeatures impacts the modelâs overall performance.54Chapter 5Conclusions5.1 DiscussionAutonomous driving is one of the most safety-critical applications of modern Ar-tificial Intelligence (AI). Autonomous Vehicles (AVs) aim to capitalize on the mas-sive gains of the last few years in the development of Deep Neural Nets (DNNs) toprocess the raw input from their sensors. From the many sensors within an AVâsperception system, the two most commonly used for 3D object detection are theimage-producing camera and the point cloud-producing LiDAR. The 3D detec-tion process depends heavily on AI models (specifically deep learning models) thatemploy DNNs to process the inputs from the cameras and LiDARs.We explained the basics of modern deep learning and its vulnerability to ad-versarial attacks. These attacks have been initially discovered and heavily exploredin the field of image classification. They uncovered a great weakness within DNNswhere an imperceptible pixel-wise perturbation of the image input would fool amodel into misclassification [36]. These attacks were then developed to be morerealistic and effective in real-life [5, 30, 37]. Adversarial attacks were then ex-plored for point cloud classification where simple point-wise perturbations wouldfool point cloud-based DNNs [43]. These attacks were also further developed tobe more realistic by inserting a 3D object in a scene and simulating a LiDAR togenerate realistic adversarial point clouds [2, 39]. We found that there was verylimited prior work on multi-modal adversarial attacks that could pose a threat to55multi-modal deep learning on both images and point clouds. One of the maincontributions of this thesis is demonstrating a realistic, universal (input-agnostic)multi-modal adversarial attack on models that take images and point clouds as in-put.We focused on the safety-critical field of autonomous driving to try and dis-cover if adversarial attacks can be threatening to multi-modal 3D detection modelsin real life. We presented a method for a multi-modal adversarial attack whichexpanded greatly beyond previous adversarial attacks on DNNs that had either tar-geted a single modality, or required very controlled environments to be effective.We learned from the weaknesses of previous approaches and decided to make anattack that is both realistic and universal. This attack does not perturb specific pix-els or points but a 3D objectâs shape and texture which in turn perturbs the input toa model in a realistic fashion. This attack is universal because a single object canhide any host vehicle in any 3D scene from detection by the model it was trainedon.We designed a multi-modal insertion attack where we place an adversarial ob-ject on a vehicle to make it evade detection by powerful DNN-based multi-modal3D detectors. This objectâs shape and texture are mutable properties, and chang-ing them is our way of introducing realistic perturbations to the image as well asthe point cloud input. We parametrize the shape deformation of the 3D adversar-ial object as vertex displacements of a 3D mesh. We also parametrize the textureas per-vertex colors that are interpolated to produce the meshâs texture. We useStochastic Gradient Descent (SGD) to optimize these parameters towards minimiz-ing our adversarial loss, which in turn maximizes the probability of the modelmissing host vehicles.To ensure our attack is realistic we utilized tools from the field of computergraphics to represent and render our 3D adversarial object to multiple modalities.We place the 3D mesh over a vehicle in 3D space and then render it to 2D imagesand 3D point clouds to realistically simulate the objectâs presence in real-life. Wemade a differentiable GPU implementation of a ray-triangle intersection algorithm[21] to simulate a LiDAR. We used a differentiable image renderer [27] where wesimulated sunlight and used Phong shading to realistically render our 3D mesh tobe as if it was captured by a camera outdoors.56We used this attack to study the vulnerabilities of multi-modal 3D detectionmodels to realistic adversarial attacks. We focused on attacking two common 3Ddetection architectures: cascaded and fusion. The cascaded model handles eachmodality independently with a pipeline for the image input first, and then anotherfor the point cloud input. So we designed a two-stage attack that perturbs each in-put separately. Nonetheless, we accounted for cross-modality consistency since werendered the adversarial shape of the mesh (learned using the point cloud pipeline)to the RGB image to train the texture using the image pipeline. On the other hand,fusion models learn from both input modalities simultaneously towards accurate3D detection that uses the complimentary information of semantically rich im-ages and geometrically accurate point clouds. We utilize the architecture of fusionmodels to train the shape and texture of our multi-modal adversarial object simul-taneously and end-to-end.We conducted many experiments to determine if our adversarial object wassuccessful in its attempt to hide its host vehicles from both victim models (at-tack success rate). It can be concluded from our attack that both models underconsideration were vulnerable to multi-modal adversarial attacks to a relativelyclose degree. Our adversarial object was able to hide 55.6% of its host vehiclesfrom the cascaded model and 63.19% of its host vehciles from the fusion model.Both models were very vulnerable to any image-perturbing attacks (including ourmulti-modal attack) with the cascaded model being slightly more robust than thefusion model. However, the fusion model was found to be nearly immune to shapeperturbations with only around 6% attack success rate. The cascaded model how-ever was found to be much more vulnerable to point cloud-based attacks than thefusion model which is most likely strengthened by its utilization of image fea-tures. Moreover, we found that the cascaded modelâs recall under low Intersection-over-Union (IoU) thresholds (small overlap between the predicted and the groundtruth bounding boxes indicating inaccurate 3D detections) depends almost entirelyon the 2D region proposal stage which represents the bottleneck of the cascadedmodelâs performance. If the 2D detector missed a vehicle then it will simply goundetected by the entire 3D detection model. The cascaded modelâs neglect ofimage features in the 3D detection process (beyond region proposals) means thatit wastes a significant amount of rich semantic information from images. Finally,57the cascaded modelâs limited 3D search space (post-region proposal) means thatsuch a model often misses essential contextual information especially around a ve-hicle. This could explain why placing a simple 3D sphere on a host vehicle (inour ablation study) made the cascaded model much less accurate and miss manyvehicles.We also applied our adversarial object to all vehicles and measured a modelâsAverage Precision (AP) under different levels of occlusion. This experiment showedoverall results that were consistent with the results from measuring recall. Our at-tack on the cascaded model dropped its AP (under the low occlusion scenario) from91.17% to 29.04%. The cascaded modelâs accuracy dropped with just a sphere ontop of vehicles and dropped further with our adversarial object. The cascadedmodel was also found to be slightly more robust to image-perturbations than thefusion model especially under low occlusion because of the cascaded modelâs uti-lization of mature 2D detectors which relatively protected it from our image per-turbations (compared to the fusion model) especially since our adversarial objectdoes not cover the host vehicle at all as it is placed above it.In the fusion modelâs case, we found that adversarial shape perturbations didnâtaffect AP much unlike texture perturbations which were very detrimental to themodelâs AP. However the combined shape and texture perturbations affected themodel by nearly 15% more than just texture perturbations. This can be attributedto the larger mesh which results from shape perturbations that compounds the ef-fects of image perturbations. These effects can include false positives that mostlikely occur due to the erroneous projection of image features to LiDAR points ina distance [40]. This happens because of the cameraâs perspective projection from3D scenes to 2D images that make faraway objects seem much closer to each other.It can also happen due to the abstractions of DNNs which can make the corrupt ex-tracted features of our adversarial object affect nearby benign features.5.2 Future WorkThis attacks opens the door to many more works in the future starting from the spe-cific field of adversarial attacks, to the broader fields of multi-modal deep learningand autonomous driving.58The world is multi-modal and if AI systems are to truly understand and interactwith the world they need to be multi-modal. Our work is among the first fewworks on multi-modal adversarial attacks (on image and point cloud processingmodels) which is a nascent field deserving of much deeper research. Previousadversarial attacks provided great insights into the inner workings of the black-boxthat is DNNs and gave us a better understanding of the decision boundaries thatDNNs learn [7, 38]. Our attack can contribute to these attempts in the context ofmulti-modal learning. Moreover, the field of deep learning interpretability usedadversarial examples (what fools a model) to understand what a model actuallylearns. So, our attack can contribute to opening up a sub-field of multi-modal deeplearning interpretability.The vulnerabilities that our attack exposed calls researchers to action on pro-tecting current models or proposing new robust models. There are many workson adversarial defense, however many of them remain unsatisfactory and unableto offer a true solution to the adversarial problem of DNNs [29]. There are numer-ous algorithms and methods of constructing adversarial attacks. Most adversarialdefense works could be successful at thwarting some attacks but not other unseenattacks [48]. The most successful defense so far seems to be adversarial trainingwhere the model is trained on perturbed inputs. However adversarial training isoften inefficient, especially with iterative attacks such as ours where the attack isrepeatedly trained over multiple input samples. Adversarial training is also oftenincapable of covering all possible adversarial attacks [29].An alternative to defense against adversarial attacks is to design entirely newmodels with this robustness built in. These models would probably be character-ized by being able to learn much more meaningful and robust features compared tothe current features learned by most DNN-based models. Until then, our attack canbe used to measure the robustness of a camera-LiDAR multi-modal model to real-istic adversarial attacks. Our attack method can also be used in adversarial trainingto at least protect some models from our type of insertion attacks.An adversarial attack is transferable when it is effective in fooling models otherthan the one it was trained to fool [45]. While the two models we attack are opensource and rely on powerful and common backbones which can make our resultsgeneralizable to an extent, there is still the open question of whether a single multi-59modal adversarial object can be effective in attacking multiple 3D detection mod-els. This transferability is behind the success of many black-box adversarial attacksthat attempt to clone a model using only its output and then generate adversarialexamples from the clone [23]. While our attack is a white-box attack, its simu-lation and perturbation methods can be utilized to launch a black-box adversarialattack. This form of attack as well as transferable attacks can pose a serious threatif they were proven to be effective on unseen models. This could mean that even ifan AVs company kept its AI models secret, these models may still be vulnerable tosuch an adversarial attack.60Bibliography[1] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robustadversarial examples. In Proceedings of the 35th International Conferenceon Machine Learning, volume 80 of Proceedings of Machine LearningResearch, pages 284â293, Stockholmsma¨ssan, Stockholm Sweden, 10â15Jul 2018. PMLR. URL http://proceedings.mlr.press/v80/athalye18b.html.â pages 17, 23[2] Y. Cao, C. Xiao, D. Yang, J. Fang, R. Yang, M. Liu, and B. Li. Adversarialobjects against lidar-based autonomous driving systems. arXiv preprintarXiv:1907.05418, 2019. â pages 9, 21, 24, 26, 55[3] Y. Cui, R. Chen, W. Chu, L. Chen, D. Tian, and D. Cao. Deep learning forimage and point cloud fusion in autonomous driving: A review. arXivpreprint arXiv:2004.05224, 2020. â pages 2, 3, 5, 7, 22[4] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, F. Trame`r,A. Prakash, T. Kohno, and D. Song. Physical adversarial examples for objectdetectors. In 12th USENIX Workshop on Offensive Technologies (WOOT18), Baltimore, MD, Aug. 2018. USENIX Association. URLhttps://www.usenix.org/conference/woot18/presentation/eykholt. â pages17, 19, 23[5] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao,A. Prakash, T. Kohno, and D. Song. Robust physical-world attacks on deeplearning visual classification. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 1625â1634, 2018. âpages x, 9, 17, 18, 23, 55[6] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving?the kitti vision benchmark suite. In 2012 IEEE Conference on ComputerVision and Pattern Recognition, pages 3354â3361. IEEE, 2012. â pagesx, 1, 2, 5, 6, 8, 39, 4061[7] J. Gilmer, L. Metz, F. Faghri, S. S. Schoenholz, M. Raghu, M. Wattenberg,and I. Goodfellow. Adversarial spheres. arXiv preprint arXiv:1801.02774,2018. â page 59[8] I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessingadversarial examples. In International Conference on LearningRepresentations, 2015. URL http://arxiv.org/abs/1412.6572. â pagesx, 8, 16, 23, 27[9] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun. Deep learningfor 3d point clouds: A survey. IEEE transactions on pattern analysis andmachine intelligence, 2020. â pages 3, 5[10] T. Huang, Z. Liu, X. Chen, and X. Bai. Epnet: Enhancing point featureswith image semantics for 3d object detection. In European Conference onComputer Vision, pages 35â52. Springer, 2020. â pagesx, 4, 7, 22, 34, 36, 38, 40, 47, 50, 51[11] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXivpreprint arXiv:1412.6980, 2014. â pages 41, 43[12] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in thephysical world. ICLR Workshop, 2017. URLhttps://arxiv.org/abs/1607.02533. â pages 8, 16, 18, 23[13] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436â444, 2015. doi:10.1038/nature14539. URLhttps://doi.org/10.1038/nature14539. â pages 3, 4[14] M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun. Multi-task multi-sensorfusion for 3d object detection. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 7345â7353, 2019. âpage 7[15] Z. C. Lipton. The mythos of model interpretability: In machine learning, theconcept of interpretability is both important and slippery. Queue, 16(3):31â57, 2018. â page 5[16] H.-T. D. Liu, M. Tao, C.-L. Li, D. Nowrouzezahrai, and A. Jacobson.Beyond pixel norm-balls: Parametric adversaries using an analyticallydifferentiable renderer. In International Conference on LearningRepresentations, 2019. URL https://openreview.net/forum?id=SJl2niR9KQ.â pages 20, 2362[17] S. Liu, T. Li, W. Chen, and H. Li. Soft rasterizer: A differentiable rendererfor image-based 3d reasoning. The IEEE International Conference onComputer Vision (ICCV), Oct 2019. â page 35[18] J. Lu, H. Sibai, E. Fabry, and D. Forsyth. Standard detectors arenât(currently) fooled by physical adversarial stop signs. arXiv preprintarXiv:1710.03337, 2017. â pages 14, 18, 19[19] J. Lu, H. Sibai, E. Fabry, and D. Forsyth. No need to worry about adversarialexamples in object detection in autonomous vehicles. arXiv preprintarXiv:1707.03501, 2017. â pages 17, 19[20] D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network forreal-time object recognition. In 2015 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), pages 922â928, 2015.doi:10.1109/IROS.2015.7353481. â page 3[21] T. Mo¨ller and B. Trumbore. Fast, minimum storage ray-triangle intersection.Journal of graphics tools, 2(1):21â28, 1997. â pages 27, 28, 56[22] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universaladversarial perturbations. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (CVPR), July 2017. â pages8, 9, 18, 23[23] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami.Practical black-box attacks against machine learning. In Proceedings of the2017 ACM on Asia Conference on Computer and Communications Security,ASIA CCS â17, page 506â519, New York, NY, USA, 2017. Association forComputing Machinery. ISBN 9781450349444.doi:10.1145/3052973.3053009. URLhttps://doi.org/10.1145/3052973.3053009. â page 60[24] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on pointsets for 3d classification and segmentation. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages 652â660,2017. â pages 3, 4, 6, 30, 32, 37[25] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchicalfeature learning on point sets in a metric space. In Proceedings of the 31stInternational Conference on Neural Information Processing Systems,NIPSâ17, page 5105â5114, Red Hook, NY, USA, 2017. Curran AssociatesInc. ISBN 9781510860964. â pages 4, 7, 3463[26] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas. Frustum pointnets for 3dobject detection from rgb-d data. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (CVPR), June 2018. â pagesx, 4, 5, 6, 30, 32, 33, 37, 38, 40, 47[27] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, andG. Gkioxari. Accelerating 3d deep learning with pytorch3d.arXiv:2007.08501, 2020. â pages 29, 30, 56[28] J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXivpreprint arXiv:1804.02767, 2018. â pages 32, 37[29] K. Ren, T. Zheng, Z. Qin, and X. Liu. Adversarial attacks and defenses indeep learning. Engineering, 6(3):346â360, 2020. â page 59[30] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter. Accessorize to acrime: Real and stealthy attacks on state-of-the-art face recognition. InProceedings of the 2016 acm sigsac conference on computer andcommunications security, pages 1528â1540, 2016. â pagesx, 8, 9, 17, 23, 35, 55[31] S. Shi, X. Wang, and H. Li. Pointrcnn: 3d object proposal generation anddetection from point cloud. In The IEEE Conference on Computer Visionand Pattern Recognition (CVPR), June 2019. â pages 4, 34, 35, 50, 51[32] K. Shin, Y. P. Kwon, and M. Tomizuka. Roarnet: A robust 3d objectdetection based on region approximation refinement. In 2019 IEEEIntelligent Vehicles Symposium (IV), pages 2510â2515, 2019.doi:10.1109/IVS.2019.8813895. â pages 4, 5, 30[33] A. Simonelli, S. R. Bulo, L. Porzi, M. Lo´pez-Antequera, andP. Kontschieder. Disentangling monocular 3d object detection. InProceedings of the IEEE/CVF International Conference on ComputerVision, pages 1991â1999, 2019. â page 40[34] V. A. Sindagi, Y. Zhou, and O. Tuzel. Mvx-net: Multimodal voxelnet for 3dobject detection. In 2019 International Conference on Robotics andAutomation (ICRA), pages 7276â7282. IEEE, 2019. â pages 7, 22[35] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller. Multi-viewconvolutional neural networks for 3d shape recognition. In Proceedings ofthe IEEE international conference on computer vision, pages 945â953,2015. â page 364[36] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow,and R. Fergus. Intriguing properties of neural networks. arXiv preprintarXiv:1312.6199, 2013. â pages 8, 16, 23, 55[37] S. Thys, W. Van Ranst, and T. Goedeme´. Fooling automated surveillancecameras: adversarial patches to attack person detection. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern RecognitionWorkshops, pages 0â0, 2019. â pages x, 9, 17, 19, 23, 25, 35, 55[38] F. Trame`r, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Thespace of transferable adversarial examples. arXiv preprintarXiv:1704.03453, 2017. â page 59[39] J. Tu, M. Ren, S. Manivasagam, M. Liang, B. Yang, R. Du, F. Cheng, andR. Urtasun. Physically realizable adversarial examples for lidar objectdetection. In Proceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition (CVPR), June 2020. â pages9, 21, 24, 25, 26, 27, 34, 35, 37, 38, 43, 50, 51, 55[40] J. Tu, H. Li, X. Yan, M. Ren, Y. Chen, M. Liang, E. Bitar, E. Yumer, andR. Urtasun. Exploring adversarial robustness of multi-sensor perceptionsystems in self driving. arXiv preprint arXiv:2101.06784, 2021. â pages22, 24, 27, 43, 54, 58[41] S. Wang, T. Wu, and Y. Vorobeychik. Towards robust sensor fusion in visualperception. arXiv preprint arXiv:2006.13192, 2020. â pages 22, 24[42] Z. Wang and K. Jia. Frustum convnet: Sliding frustums to aggregate localpoint-wise features for amodal 3d object detection. In 2019 IEEE/RSJInternational Conference on Intelligent Robots and Systems (IROS), pages1742â1749. IEEE, 2019. â pages 4, 5, 30[43] C. Xiang, C. R. Qi, and B. Li. Generating 3d adversarial point clouds. InProceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition (CVPR), June 2019. â pages 9, 20, 24, 55[44] C. Xiao, D. Yang, B. Li, J. Deng, and M. Liu. Meshadv: Adversarial meshesfor visual recognition. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 6898â6907, 2019. âpages 20, 23[45] H. Xu, Y. Ma, H.-C. Liu, D. Deb, H. Liu, J.-L. Tang, and A. K. Jain.Adversarial attacks and defenses in images, graphs and text: A review.65International Journal of Automation and Computing, 17(2):151â178, 2020.â page 59[46] J. Yang, Q. Zhang, R. Fang, B. Ni, J. Liu, and Q. Tian. Adversarial attackand defense on point sets. arXiv preprint arXiv:1902.10899, 2019. â pages9, 20, 24[47] Y. Yu, H. J. Lee, B. C. Kim, J. U. Kim, and Y. M. Ro. Investigatingvulnerability to adversarial examples on multimodal data fusion in deeplearning. arXiv preprint arXiv:2005.10987, 2020. â pages 21, 24[48] X. Yuan, P. He, Q. Zhu, and X. Li. Adversarial examples: Attacks anddefenses for deep learning. IEEE transactions on neural networks andlearning systems, 30(9):2805â2824, 2019. â page 59[49] X. Zeng, C. Liu, Y.-S. Wang, W. Qiu, L. Xie, Y.-W. Tai, C.-K. Tang, andA. L. Yuille. Adversarial attacks beyond the image space. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition, pages4302â4311, 2019. â page 2066Appendix AQualitative ExamplesIn the following figures we show some extra examples of our adversarial attackson the cascaded and fusion models, and how these attacks affect 3D detection.67Figure A.1: First example of the multi-modal attack on the fusion model. 3Dbounding boxes of detected vehicles are shown in LiDAR point cloudand camera image before (top) and after (bottom) our multi-modal ad-versarial attack on the fusion model. The point cloud rendering of theadversarial mesh is shown as red points.68Figure A.2: Second example of the multi-modal attack on the fusion model.3D bounding boxes of detected vehicles are shown in LiDAR pointcloud and camera image before (top) and after (bottom) our multi-modal adversarial attack on the fusion model. The point cloud ren-dering of the adversarial mesh is shown as red points.69Figure A.3: First example of the multi-modal attack on the cascaded model.3D bounding boxes of detected vehicles are shown in the image andtheir BEV projection in LiDAR point cloud before (top) and after (bot-tom) our multi-modal adversarial attack on the cascaded model. Thepoint cloud rendering of the adversarial mesh is shown as red points.We applied the mesh on all vehicles and use BEV to show distant misseddetections.70Figure A.4: Second example of the multi-modal attack on the cascadedmodel. 3D bounding boxes of detected vehicles are shown in the imageand their BEV projection in LiDAR point cloud before (top) and after(bottom) our multi-modal adversarial attack on the cascaded model.The point cloud rendering of the adversarial mesh is shown as redpoints. We applied the mesh on all vehicles and use BEV to show distantmissed detections.71
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Adversarial attacks on multi-modal 3D detection models
Text