Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Anjum, Taif
Master of Science - MSc
University of British Columbia
Globally, traffic accidents are the leading cause of death, according to reports published by the World Health Organization (WHO). Advanced Driver Assistance Systems (ADAS) effectively reduce the likelihood or severity of accidents. Time-To-Accident (TTA) forecasting is a critical component of ADAS; it can improve decision-making in traffic, dynamic path planning, and alert drivers to potential dangers. TTA is defined as the interval between the threat of an accident and the occurrence of the accident itself. Sensor or depth imaging-based TTA forecasting can be inaccurate and unaffordable to the mass. Existing vision-based traffic safety research in the literature focuses on accident anticipation with limited work on TTA prediction. We propose forecasting TTA by utilizing spatio-temporal features from dashboard camera videos and introducing a novel algorithm to compute ground truth TTA values. Additionally, we present two efficient architectures, a Light 3D Convolutional Neural Network (Li3D) and Hybrid CNN-Transformer (HyCT) network, for our application. Li3D uses significantly fewer parameters and larger kernels for efficient feature learning without sacrificing accuracy. We considered transfer learning to improve the model's performance further but faced limitations due to a lack of relevant large-scale video datasets and Li3D's input requirements. Thus we propose the HyCT network which is trained via transfer learning. HyCT outperformed a comprehensive list of networks, including Li3D on the Car Crash Dataset (CCD). Furthermore, with our approach, we can simultaneously forecast TTA and recognize accident and non-accident scenes. In real-world scenarios, it is vital to identify when a normal driving scene transitions into a potential accident scene. We assess our approach’s ongoing scene recognition ability using the TTAASiST@x (Time-to-Accident-based Accuracy at the xth frame Since Scene Transition) metric. With just 25% of the data, HyCT achieved over 85% accuracy in recognizing accident and non-accident scenes, further showcasing its performance on limited data. Our best-performing models can predict TTA values with an average prediction error of 0.25 and 0.79 seconds on the CCD and DoTA dataset, respectively.
Leveraging Spatio-temporal Featuresto Forecast Time-To-AccidentbyTaif AnjumB.Sc. University of the Fraser Valley, 2021A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)July 2023© Taif Anjum, 2023The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Leveraging Spatio-temporal Features to Forecast Time-To-Accidentsubmitted by Taif Anjum in partial fulfilment of the requirements of thedegree of Master of ScienceDr. Apurva Narayan, Irving. K. Barber Faculty of ScienceSupervisorDr. John Braun, Irving. K. Barber Faculty of ScienceCo-SupervisorDr. Mohammad Khaled Hasan, Irving. K. Barber Faculty of ScienceSupervisory Committee MemberDr. Jeffrey Andrews, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Anas Chaaban, Faculty of Applied ScienceUniversity ExamineriiAbstractGlobally, traffic accidents are the leading cause of death, according to re-ports published by the World Health Organization (WHO). Advanced DriverAssistance Systems (ADAS) effectively reduce the likelihood or severity ofaccidents. Time-To-Accident (TTA) forecasting is a critical component ofADAS; it can improve decision-making in traffic, dynamic path planning,and alert drivers to potential dangers. TTA is defined as the interval betweenthe threat of an accident and the occurrence of the accident itself. Sensor ordepth imaging-based TTA forecasting can be inaccurate and unaffordableto the mass. Existing vision-based traffic safety research in the literaturefocuses on accident anticipation with limited work on TTA prediction. Wepropose forecasting TTA by utilizing spatio-temporal features from dash-board camera videos and introducing a novel algorithm to compute groundtruth TTA values.Additionally, we present two efficient architectures, a Light 3D Con-volutional Neural Network (Li3D) and Hybrid CNN-Transformer (HyCT)network, for our application. Li3D uses significantly fewer parameters andlarger kernels for efficient feature learning without sacrificing accuracy. Weconsidered transfer learning to improve the model’s performance furtherbut faced limitations due to a lack of relevant large-scale video datasets andLi3D’s input requirements. Thus we propose the HyCT network which istrained via transfer learning. HyCT outperformed a comprehensive list ofnetworks, including Li3D on the Car Crash Dataset (CCD). Furthermore,with our approach, we can simultaneously forecast TTA and recognize acci-dent and non-accident scenes. In real-world scenarios, it is vital to identifywhen a normal driving scene transitions into a potential accident scene.We assess our approach’s ongoing scene recognition ability using the TTA-ASiST@x (Time-to-Accident-based Accuracy at the xth frame Since SceneTransition) metric. With just 25% of the data, HyCT achieved over 85% ac-curacy in recognizing accident and non-accident scenes, further showcasingits performance on limited data. Our best-performing models can predictTTA values with an average prediction error of 0.25 and 0.79 seconds on theCCD and DoTA dataset, respectively.iiiLay SummaryTraffic accidents are a leading cause of death worldwide, according to theWorld Health Organization (WHO). Advanced Driver Assistance Systems(ADAS) can help reduce accidents but are often unaffordable. A key ADASfeature is Time-To-Accident (TTA) forecasting, which improves decision-making, route planning, and danger alerts. TTA is the time between whenan accident might happen and when it occurs. Current methods using sen-sors or depth imaging for TTA forecasting can be inaccurate and expensive.In addition, there is a lack of research on predicting TTA using only video.We propose a new TTA forecasting method that utilizes spatio-temporalfeatures from dashboard camera videos and employs a novel algorithm tocompute TTA. Our approach can simultaneously forecast TTA and recog-nize accident scenarios. We also propose two new, efficient machine learningmodels (a Light 3D Convolutional Neural Network (Li3D) and Hybrid CNN-Transformer (HyCT) network) to help with our task.ivPrefaceThe research work in this thesis was conducted by the author under thesupervision of Dr. Apurva Narayan. The experimental designs, analysis,tables, and figures presented in this thesis are original works of the au-thor. Chapters 3, 4 and portions of chapter 6 are published as Anjum et al.[ACLN23, ALN22]. While Chapter 5 and the remaining portions of Chapter6 have been accepted for publication at ”International Joint Conference OnNeural Networks 2023 (IJCNN’23)”[AKN23]Parts of chapters 2, 3, 4 and 6 were presented at SIGSPATIAL’22 [ALN22],ICAART’23 [ACLN23], IJCNN’23 [AKN23] and re-printed with permissionfrom Anjum et al.My co-authors, Daya Kumar and Louis Chirade assisted me in conduct-ing some experiments in Chapter 6.Dr. Beiyu Lin assisted in shaping the research concepts and validatingmy methodologies.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Proposed Approach . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2.1 TTA via 3D CNN . . . . . . . . . . . . . . . . . . . . 41.2.2 TTA via Transformer . . . . . . . . . . . . . . . . . . 41.2.3 Experimental Analysis . . . . . . . . . . . . . . . . . . 51.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 5Chapter 2: Related Work . . . . . . . . . . . . . . . . . . . . . . 62.1 Time-Accident (TTA) . . . . . . . . . . . . . . . . . . . . . . 62.2 Accident Anticipation . . . . . . . . . . . . . . . . . . . . . . 72.3 Spatio-temporal Feature Learning . . . . . . . . . . . . . . . . 7Chapter 3: Time-To-Accident (TTA) Forecasting . . . . . . . 103.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . 123.2 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13viTABLE OF CONTENTS3.3 Pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . . 14Chapter 4: TTA Forecasting with 3D CNN . . . . . . . . . . 174.1 3D Convolutional Neural Network (CNN) . . . . . . . . . . . 174.2 Li3D: Light 3D CNN . . . . . . . . . . . . . . . . . . . . . . . 19Chapter 5: TTA Forecasting via Transformer . . . . . . . . . . 225.1 Vision Transformer . . . . . . . . . . . . . . . . . . . . . . . 225.2 HyCT: Hybrid CNN-Transformer model . . . . . . . . . . . . 235.3 Accident Scene Recognition with TTA-ASSiST@x Metric . . 25Chapter 6: Experiments & Results . . . . . . . . . . . . . . . . 296.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . 296.2 Spatio-temporal Analysis . . . . . . . . . . . . . . . . . . . . 296.3 Comparative Analysis . . . . . . . . . . . . . . . . . . . . . . 356.3.1 Static Image Architectures . . . . . . . . . . . . . . . 356.3.2 Video Learning Architectures . . . . . . . . . . . . . . 356.4 TTA-ASiST@x Analysis . . . . . . . . . . . . . . . . . . . . . 38Chapter 7: Conclusion & Future work . . . . . . . . . . . . . . 41Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43viiList of TablesTable 3.1 Original size and characteristics of traffic accident datasetswith an egocentric view. . . . . . . . . . . . . . . . . . 14Table 3.2 Size of the datasets at each stage of processing . . . . 15Table 6.1 Mean Absolute and Root Mean Squared Errors onCCD with varying Spatial Resolution and TemporalDepth . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Table 6.2 Mean Absolute and Root Mean Squared Errors onDoTA with varying Spatial Resolution and TemporalDepth . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Table 6.3 TTA estimation on CCD: Comparison of our workwith single frame VGG-16, multi-stream stream VGG-16 [MWOBK19], ResNet-8 [LMDBS18], CNN-RNNand C3D [TRS+17] . . . . . . . . . . . . . . . . . . . . 37Table 6.4 TTA estimation on DoTA: Comparison with singleframe VGG-16, N-stream VGG-16 [MWOBK19], ResNet-8 [LMDBS18], CNN-RNN and C3D [TRS+17] . . . . . 37viiiList of FiguresFigure 1.1 TTA prediction pipeline . . . . . . . . . . . . . . . . . 2Figure 1.2 Prediction samples from test data . . . . . . . . . . . 3Figure 3.1 Sample clip from the Car Crash Dataset . . . . . . . 10Figure 3.2 Ego Vehicle in Diverse Accident Scenarios . . . . . . 14Figure 4.1 Convolutional operations of 2D CNNs and 3D CNNs 18Figure 4.2 Light 3D CNN (Li3D) Architecture . . . . . . . . . . 21Figure 5.1 Transformer Encoder Architecture . . . . . . . . . . . 23Figure 5.2 Proposed Hybrid CNN-Transformer (HyCT) Network 28Figure 6.1 Mean Absolute Error Visualization for CCD . . . . . 31Figure 6.2 Root Mean Squared Error Visualization for CCD . . 32Figure 6.3 Mean Absolute Error Visualization for DoTA . . . . . 33Figure 6.4 Root Mean Squared Error Visualization for DoTA . . 33Figure 6.5 Comparative Analysis on CCD . . . . . . . . . . . . . 36Figure 6.6 Comparative Analysis DoTA . . . . . . . . . . . . . . 36Figure 6.7 TTA-ASiST@x Analysis on CCD . . . . . . . . . . . 38Figure 6.8 TTA-ASiST@x Analysis on DoTA . . . . . . . . . . . 39ixAcknowledgementsI wholeheartedly thank Dr. Apurva Narayan for his unwavering sup-port, encouragement, and kindness at every step of this journey. I am veryfortunate to have had the opportunity to work under his leadership. I willforever be indebted to him.Additionally, I would like to thank Dr. Beiyu Lin for her support andfeedback on my work.Finally, I thank Steven Lawrence and Abhijith Sharma for helping mebrainstorm research ideas and navigate through this degree.xDedicationTo my beloved parents and siblings, your love and support fuel my mo-tivation to keep moving forward in life.xiChapter 1IntroductionThe demand for robust and efficient advanced driver assistance systems(ADAS) has been growing in recent years due to their proven effectivenessin improving driver safety. A critical component of ADAS systems is time-to-accident (TTA) forecasting. TTA forecasting is essential in the field ofintelligent transportation systems as it can aid decision-making in traffic,improve dynamic path planning, and subsequently reduce the chances of anaccident. In addition, TTA forecasting can be used to improve the perfor-mance of advanced driver assistance systems such as automatic emergencybraking and lane departure warning systems. Recent studies have foundthat systems such as forward collision warning, automatic emergency brak-ing, and lane departure warning can significantly increase TTA and reducethe number of collisions. For example, a study [TII22] by the Insurance In-stitute for Highway Safety (IIHS) found that front crash prevention systemsreduced rear-end crashes by 50%. According to the World Health Organi-zation, road traffic accidents are a leading cause of death worldwide, withan estimated 1.35 million deaths each year [WHO18]. In addition to themortality rate, road accidents also have significant economic and societalimpacts [Bel22].Despite the benefits of metrics such as TTA, the literature lacks studieson forecasting TTA. The majority of previous research has proposed meth-ods for early risk anticipation in accident forecasting, however, they fail toprovide estimations for TTA. For example, Suzuki et al. [SKAS18] proposedan adaptive loss function and a Quasi-Recurrent Neural Network (QRNN)to learn spatio-temporal features for accident anticipation. The model gen-erates the probability of a possible accident with a prediction horizon of 3seconds, but it does not predict the exact time of the accident. Similarly, Baoet al. [BYK20] proposed a Graph Convolutional Network (GCN) with RNNcell to learn spatio-temporal features, followed by Bayesian Neural Network(BNNs) to generate accident probability. Another study [CCXS16] proposeda Dynamic-Spatial Attention Recurrent Neural Network (DSA-RNN) for an-ticipating accidents from dashboard camera videos. These technologies aimto predict an accident before it occurs, but only being able to anticipate or11.1. PROPOSED APPROACHdetect a possible accident is not sufficient. To effectively make decisions,plan paths, and avoid collisions, it is important to also have a temporalestimation of the accident.1.1 Proposed ApproachIn this thesis, we propose a novel TTA forecasting and accident recog-nition framework that can predict the exact time of an accident with a 3-6second prediction horizon. Studies recommend a 3-second TTA under opti-mal conditions and 4-6 seconds in poor visibility or slippery conditions foradequate accident avoidance [RD16, AvS06]. Our approach relies only onvisual data from inexpensive dashboard cameras, eliminating the need forany additional sensors.Figure 1.1: TTA prediction pipeline: If TTA is estimated to be less thanzero, scene is non-accident; otherwise, risk of accident exists.The core idea is to extract spatio-temporal features from car accidentvideos and train machine learning models to predict the time step at whichthe accident takes place. To achieve this, we require ground truth TTA val-ues which we calculate using annotations provided by the publicly availabledatasets used in our work. The videos in datasets include binary annota-tions indicating if a given is an accident (positive) or non-accident (negative)frame. The videos always begin with a period of non-accident frames, hencethe first positive frame indicates that an accident occurred at that point.Given framewise annotations, we can calculate the ground truth TTA valueusing our novel Algorithm introduced in Chapter 3. For non-accident videos,we label them with a TTA value of -1, indicating an infinite TTA. Ourapproach can predict the TTA value and also recognize accident and non-accident scenes as illustrated in Figure 1.1. Our approach is to predict theTTA of each video using only the first N-frames as history. Our best modelscan forecast TTA with a prediction error of 0.24 seconds and 0.79 seconds onthe CCD and DoTA dataset respectively. Prediction samples from our test21.2. CONTRIBUTIONFigure 1.2: Prediction samples from test data. First two rows representaccident scenes and the third row represents non-accident scenes.data are shown in 1.2. Our proposed model outperforms a comprehensivelist of state-of-the-art video learning architectures.1.2 ContributionThe aforementioned TTA forecasting method and the calculation algo-rithm in Chapter 3 are key contributions of this thesis. However, our ap-plication requires specific characteristics when it comes to machine learningmodels. The eventual goal of our work would be to apply the frameworkas a driver assistance system which will be used in real-time. This necessi-tates architectures with both high accuracy and minimal inference latency.The following sections briefly introduce our proposed architectures and our31.2. CONTRIBUTIONevaluation approach.1.2.1 TTA via 3D CNNSince its inception, Convolutional Neural Network (CNN) has revolu-tionized computer vision and image processing. However, the conventional2D CNN can only learn spatial features and fails to learn temporal context.Due to their superiority in learning spatial features, researchers combineCNNs with sequential models such as Recurrent Neural Networks (RNNs)for video learning tasks. However, they typically operate on fixed-lengthvideo clips and fail to learn long-term temporal dependencies and context[DAHG+15]. To address this researchers proposed 3D CNNs, end-to-endCNN-based architecture that can learn both spatial and temporal features.In 3D CNNs, kernels have three dimensions, height, width, and depth. Thedepth represents the number of frames, and the 3D kernels are directly ap-plied to videos to extract spatio-temporal features. Within the realm ofvideo learning architectures, the 3D CNN models are widely regarded assuperior owing to their exceptional level of accuracy. However, researchersoften avoid 3D CNN due to the large computational cost. As our appli-cation is safety critical, we require high accuracy and low latency. Hence,we propose Li3D: Light 3D CNN that requires over 50x fewer parametersthan the commonly used 3D CNN (C3D) architecture without compromisingaccuracy.1.2.2 TTA via TransformerCNN-based models are superior at learning spatial features due to thepresence of inductive bias. Inductive bias allows CNN models to learn lo-cal features and achieve better generalization. However, due to inductivebias CNN-based video learning models are poor at learning global featureswhich is important for understanding temporal information. Additionally,pre-training is also a challenge for video learning with CNN-based modelssuch as 3D CNN due to the limited availability of large-scale video datasets.One of the recent developments in image and video processing is the useof transformers. Initially developed for applications in NLP, transformersare experts at learning temporal information due to the absence of induc-tive biases. With this motivation, we propose a Hybrid CNN-Transformer(HyCT) network trained via transfer learning, consumes less memory, needsless data, and requires lower training time than its CNN or Transformer-based counterparts.41.3. THESIS OUTLINE1.2.3 Experimental AnalysisTo evaluate our proposed approach, we perform comprehensive experi-ments. The first set of experiments analyzes the impact of spatio-temporalfeatures on the prediction error. We report the Root Mean Squared Error(RMSE) and Mean Absolute Error (MAE) for various combinations of tem-poral depth and spatial resolution. More precisely, temporal depth valuesof 2, 4, 6, 8, and 10 and spatial resolutions of 36x64, 64x128, and 144x256.This set of experiments gives us insights into how much spatial and temporalinformation is required for adequate performance. In the second set of ex-periments, we compare our proposed models against static image and videolearning models used in relevant studies. In the final set of experiments,we evaluate the scene recognition performance of our approach using theTTA-ASiST@x (Time-To-Accident-based Accuracy at the xth frame SinceScene Transition). The original ASIST@x metric proposed by Gaurav et.,al [GTN21] provides frame-wise scene recognition accuracy as well as therecognition continuity of ongoing scenes. For our application, we modifythe ASiST@x metric and present TTA-ASiST@x in Chapter 5. Models thatcan deliver efficient performance with limited data are always sought af-ter, as video learning necessitates the processing of large amounts of framesusing computationally intensive architectures. Hence, to test our model’sperformance on limited data, we perform our scene recognition experimentswith only 1/4th of the dataset.1.3 Thesis OutlineThis thesis is organized as follows, Chapter 2 presents the related work,chapter 3 describes the proposed approach, chapter 4 introduces the Li3Darchitecture, Chapter 5 introduces the HyCT architecture, Chapter 6 detailsthe experiments and presents results, and finally, Chapter 7 summarizes thethesis, and discusses future research directions.5Chapter 2Related Work2.1 Time-Accident (TTA)Time-to-Accident (TTA) refers to the period before two (or more) ob-jects will collide as defined by [Hay72]. They proposed time-measure-to-collision (TMTC) as a measure of danger to an accident which was es-timated using the velocity and distance between vehicles. The study by[JNG13] builds on that work and proposes a more computationally efficientand accurate calculation of TTA between two vehicles colliding at constantspeed along a straight path. These studies provide mathematical equationsfor TTA estimation; however, they do not discuss the application side of it.With the emergence of deep learning and advancements in computer vision,TTA calculation, and estimation techniques have evolved. TTA estimationscan now be done using object detection, tracking, instance segmentation,and trajectory prediction [TSSPdF22]. TTA is not only limited to auto-motive vehicles but is a critical component for navigation in robotics, ves-sels, and Unmanned Aerial Vehicles. One study [TSSPdF22] introduced aframework that utilizes object detection to detect objects around a ship andgenerate bounding boxes, which are used to track the objects and producevelocity vectors. In robot navigation, TTA is estimated by tracking thetrajectory and measuring the velocity of surrounding objects or pedestri-ans [BGO+16, SAMK18]. A study by [MWOBK19] proposes time-to-nearcollision prediction between a suitcase-shaped robot and nearby pedestriansusing a monocular camera and lidar sensors. They combine video and lidardata to predict when pedestrians will be within one meter of the robot froma sequence of frames. The aforementioned approaches rely on high-qualitysensors and depth imaging devices to detect and track objects. However,sensor noise and error in object detection can easily cause such approachesto fail. For example, inaccuracies in-depth estimation or 2D bounding boxdetection can result in significant velocity changes and inaccurate trajectoryestimates [MWOBK19].62.2. ACCIDENT ANTICIPATION2.2 Accident AnticipationAccident anticipation methods seek to predict an accident before it takesplace. We require a first-person or ego-centric view for vision-based ap-proaches, such as the view from dashboard cameras. Several works proposedSpatio-temporal learning frameworks and car accident datasets comprisedof videos from dashboard cameras. One study [SKAS18] offered a Near-miss Incident DataBase (NIDB) for near-miss traffic accident anticipation.They present an Adaptive Loss for Early Anticipation (AdaLEA) and Quasi-Recurrent Neural Network (QRNN) to evaluate their dataset. AdaLEA isa loss function that aims to learn earlier anticipation as training progresses.The QRNN is an efficient alternative to LSTM for temporal feature learn-ing. Their system outputs a probability of a possible accident in the futureand can anticipate a near-miss incident or accident about 3 seconds in ad-vance. Similarly, [BYK20] proposes a Graph Convolutional Network (GCN)with RNN cell to learn Spatio-temporal features followed by Bayesian neu-ral network (BNNs) to generate accident probability. [CCXS16] proposeda Dynamic-Spatial Attention Recurrent Neural Network (DSA-RNN) foranticipating accidents from dashcam videos. Using their model, they useobject detection to identify candidate objects and incorporate spatial andtemporal features from sequential images. The works mentioned above cananticipate a possible accident; however, they fail to predict the exact timeof the accident.2.3 Spatio-temporal Feature LearningSpatio-temporal feature learning is necessary for video tasks, includingaction recognition, scene recognition, accident anticipation, pose estima-tion, and more. Spatio-temporal features provide the motion informationfrom a sequence of images which can be used to recognize activities in se-quential data such as videos. Before the success of CNN and its variants,Spatio-temporal features were handcrafted using algorithms such as SIFT-3D, HOG-3D, and Motion Boundary Histogram. However, in recent yearsautomated Spatio-temporal feature learning has gained tremendous successdue to the emergence of Deep Learning (DL) algorithms. Spatio-temporalfeature learning in the DL domain can be split into two categories, (i) atwo-stream method where spatial and temporal features are extracted sep-arately and then fused (ii) Spatio-temporal kernels are applied directly tothe videos.72.3. SPATIO-TEMPORAL FEATURE LEARNINGCombinations of CNNs and RNNs belong to the first category. Like thework by [YHNHV+15] that uses GoogleLet to extract the spatial features,an LSTM for temporal context and fuses them before feeding to the fullyconnected layers.3D CNNs fall into the second category, where 3D convolutional kernelsare directly applied sequences of images or videos to capture spatial andtemporal features. 3D CNN was first introduced by [JXYY12] for humanaction recognition from video data. Their work was the first deep learning-based approach for learning Spatio-temporal features from multiple sequen-tial frames. Following their work, [TBF+15] conducted an empirical study tofind the best kernel dimensions and introduced the C3D architecture. Theirstudy showed that the C3D architecture could excel at various video taskswithout fine-tuning. C3D demonstrates superior performance but comes ata high computational cost; as an efficient alternative [TRS+17] proposed aResNet style architecture named Res3D which is faster and more compactcompared to C3D. C3D and Res3D are limited to short clips, at most 16frames long. This limitation was addressed by [VLS17] who developed along-term temporal convolution (LTC) architecture that can apply 3D con-volution on videos of up to 100 frames in length. However, one drawback of3D CNN is that it requires a large amount of training data and computa-tional resources.CNNs were considered state-of-the-art for image or video analysis untilrecently when Vision Transformers took over the title. The introductionof the transformer model is considered revolutionary in the field of NLPand Vision, as it alleviated the limitations of sequence modeling architec-tures by handling entire sequences at once, greatly parallelizing the process.Consequently, the model can capture interactions between non-local con-texts with far greater ease than its sequential counterparts. This ability toparallelize, along with the lack of inductive biases, makes the transformermodel highly versatile, leading to its application for different modalities[DCLT18, BZMA20, DBK+20, ADH+21].The original transformer model [VSP+17] is based on the principle ofSelf-Attention. This involves generating representations by considering theinteraction among all elements in an input sequence. The ‘attention’ thatan element should give another is computed using a pair-wise affinity func-tion. This ability to capture interactions between all elements in an inputsequence uniquely suits transformers for video data, allowing the modelto learn long-range temporal dependencies and capture motion cues. De-spite this inherent compatibility of transformer models and video data, sim-ply utilizing an image transformer for this task is not sufficient, as mod-82.3. SPATIO-TEMPORAL FEATURE LEARNINGeling temporal interactions requires additional considerations than simplymodeling appearances. This led to the introduction of the Video VisionTransformer (ViViTs) [ADH+21], a pure transformer-based model for videoclassification, capable of extracting spatio-temporal tokens from the inputvideo. The model achieved state-of-the-art results on multiple video classi-fication benchmarks outperforming existing models. Despite the success ofViViTs, these models still need to improve on the limitations of the orig-inal transformer model. The transformer model scales quadratically withthe length of the input sequence, making these models computationallyexpensive. This complexity is increased further when dealing with videodata due to its high dimensionality. Additionally, while allowing versatility,the lack of inductive bias also leads to the model requiring large amountsof data to learn the highly redundant spatio-temporal structure of videos.Many transformer-based architectures have been proposed to try to alleviatethe above limitations in vision transformers. Multiscale vision transformers(MViTs) introduced by [FXM+21] combine the idea of multiscale featurehierarchies with transformer models. MViTs reduce the complexity of themodel by utilizing inductive biases built into the model itself due to its multi-scale design. Frozen Embedding Networks [SJE+22] has also been suggestedto reduce the data requirements by infusing task-specific biases.The success of these models has led to their application in safety-criticalscenarios, particularly in the domain of road and traffic safety. Recent stud-ies have focused on the uses of variants of vision transformers for road safetyapplications such as - traffic accident risk forecasting [SGM22], automatedvehicle safety assessment [KLHY22], traffic congestion prediction [RSK+23],driver attention prediction [GZL22], and many more. However, none of theabove studies have attempted to predict the time-to-accident, which canhave critical applications for road safety and application to autonomous ve-hicles.9Chapter 3Time-To-Accident (TTA)ForecastingCalculating the ground truth TTA value can be challenging withoutdepth sensing capability. Most traffic accident datasets provide annotationsindicating the point of the accident in the videos. We use two publiclyavailable datasets: the Car Crash Dataset (CCD) [YWX+22] and Detectionof Traffic Anomaly (DoTA) [BYK20]. These datasets provide annotationsindicating the exact frame where an accident occurs. The annotations arebinary labels associated with each frame, indicating whether the frame isan accident (positive) or non-accident (negative) scene. Given a sequence ofN consecutive frames {F1, F2, F3...FN}, our goal is to use this sequence ashistory to estimate the time-to-accident. If the estimation is less than 0, wecan say that there will be no accidents in the near future.Figure 3.1: Sample clip from the Car Crash DatasetFor a given accident clip, we have M binary labels {label1, label2, label3...labelM},where M is the total number of frames in the video. The binary labels areassociated with each video frame; for our case, each frame represents 0.1seconds as there are 10 fps. According to the annotations, the first posi-10tive label in this sequence is the time step where the accident has started.We denote the first accident label in this sequence of labels as T, then ourground truth time-to-accident (TTA) is t = T/10 seconds. 3.1 depicts a clipfrom the CCD dataset and illustrates how the TTA value is calculated.Algorithm 1: Time-to-Accident CalculationInput: V and LjOutput: TTA and LTTAjInitialization:FPS (frames per second);TTA = -1;LTTAj = -1 for all j;for each frame j in the video doif Lj = 1 then// First positive label foundset TTA← j/FPS; // Calculate TTAendendif TTA = −1 then// No positive labels are foundset all LTTAj as -1; // set all frames as -1endelsefor each frame j in the video doif j ≤ TTA× FPS thenset LTj ← T − j/FPS; // Compute TTA forpre-accident framesendelseset LTTAj ← 0; // Post-accident frames are setto 0endendendReturn TTA, LTTAj113.1. PROBLEM FORMULATIONFor automated TTA calculation, we present Algorithm 1 that providesTTA for the entire video and each frame. The algorithm takes a video, V,and binary labels Lj , which indicate the accident and non-accident frames.First, Frames Per Second (FPS) value is initialized, which depends on thevideo. Then we initialize the TTA value for the video and frame-wise TTAvalues as -1 for all frames. A value of -1 indicates an infinite TTA value,as no positive labels were found in the video. The first For loop labels thevideos with their calculated TTA value. The second For loop labels eachframe with the corresponding TTA value. The frames after the point of anaccident are labeled as 0, indicating that an accident has already occurred.The algorithm returns TTA for the video and framewise TTA labels (LTTA)for all frames. It is to be noted that this algorithm can be applied to anytraffic accident dataset, a machine learning model can easily be trained tolabel accident and non-accident frames, and this is the only pre-requisite forour algorithm.3.1 Problem FormulationTime-to-accident prediction can be formulated in two ways - multi-classclassification or regression. Multi-class classification penalizes mispredic-tions equally regardless of the magnitude of the error. For example, for theground truth of 3.0 seconds, predictions of 0.5 seconds and 2.9 seconds willbe penalized equally with the standard categorical cross-entropy loss func-tion. Whereas the latter is a much more accurate prediction. One way toapproach the problem would be to create a differentiable loss function thatconsiders the magnitude of error [MWOBK19]. Another solution would beformulating it as a regression problem with mean-squared error as the lossfunction. Regression is often preferred over multi-class classification for sim-ilar applications because it provides more precise and continuous predictionswithout requiring a custom loss function. In contrast, multi-class classifica-tion is more suitable for discrete predictions such image or text classification.Hence, we opted to train a regression model with the following loss function.Lossmse =12||tgt − f(l1, l2, ..., lN )||2 (3.1)The Lossmse is the mean squared loss between our ground truth time tgtand predicted time f(l1, l2, ..., lN ).123.2. DATASETS3.2 DatasetsOur objective is to forecast the time-to-accident for automotive vehiclesbased on only visual data. For this, we require video data from the driver’sfield of view, such as videos from dashboard cameras. However, such data arescarce in the literature. To the best of our knowledge, there are four publiclyavailable datasets, namely, Dashcam Accident Dataset (DAD) [CCXS16],AnAn Accident Detection (A3D) [YXW+19], Detection of Traffic Anomaly(DoTA) [YWX+22] and the Car Crash Dataset (CCD) [BYK20].DAD consists of high-resolution videos with complex road scenes, crowdedstreets, and diverse accidents. However, the dataset has not included anyannotations that provide information such as weather conditions, accidentstart or end time, type of collisions, etc. Hence, we deem this dataset un-suitable for our purpose.In A3D, the accidents are categorized according to their type, e.g., col-lisions with pedestrians or vehicles that are either stopped, moving, or sta-tionary. Each video has 10 fps, and the lengths of videos are from 2.3 secondsto 20.8 seconds, as shown in Table 3.1. DoTA is an extension of A3D wherethe authors increased the number of accident clips to 4,677 and added morecategories of accidents. Therefore, the authors suggest using DoTA insteadof A3D, as DoTA is an improved version of A3D.Similar to DoTA and A3D, the Car Crash Dataset (CCD) is constructedfrom accident videos collected from YouTube. The videos are structuredsuch that the accident occurs within the last two seconds of the video. BothCCD and DoTA have frame-wise annotations that allow calculation for TTA.All videos begin with a normal period of driving before the accident occurs.The annotations were performed with the consensus of multiple human an-notators. For our particular application of forecasting time-to-accident, werequire annotations indicating the start time of accidents. CCD and DoTAprovide such annotations, so they are appropriate datasets for our experi-ments. In addition to accident clips, we require non-accident scenes to cre-ate a model robust to false alarms. CCD includes 3,000 randomly samplednormal driving video clips from the Berkley Driving Dataset (BDD100K)[YCW+20], which are also recorded via dashboard cameras. These clipsare used as negative or non-accident samples for our experiments. Eachnon-accident clip is 5 seconds long with 10 frames per second.133.3. PRE-PROCESSINGName Positive Samples Length (in s) Fps AnnotationsDAD 1,130 5 20 NoA3D 1,500 2.3-20.8 10 YesDoTA 4,677 2.3-20.8 10 YesCCD 1,500 5 10 YesTable 3.1: Original size and characteristics of traffic accident datasets withan egocentric view.3.3 Pre-processingThe videos in both datasets have annotations indicating if the ego vehiclewas involved in the crash or not. An ego vehicle is defined as a subject whosebehavior is of primary interest. In our case, the vehicle on which the camerais mounted will be referred to as the ego vehicle. The videos where theego vehicle was not involved in the crash included accidents between otherroad users captured by the ego vehicle’s dashboard camera. The ego vehiclebeing involved in the crash means there was a direct collision between theego vehicle and other road users. These two scenarios are illustrated inFigure 3.2. Our goal is to develop a system that will warn the ego vehicle ofa potential danger to itself. Considering that, we remove the videos wherethe ego vehicle is not involved in the crash.Figure 3.2: The vehicle of interest (i.e., ego vehicle) is the red car. a)Scenario where the ego vehicle is involved in the crash. b) Scenario wherethe ego vehicle witnesses an accident.The videos in the DoTA dataset are categorized into nine types of ac-cidents—for example, a collision with a pedestrian or vehicle that turns143.3. PRE-PROCESSINGinto or crosses a road. Among the nine categories, two categories did notapply to our purpose and were discarded. The first category is where thevehicle loses control and leaves the roadway; such situations can occur dueto mechanical failures or poor road conditions. If the vehicle loses control,a time-to-accident estimation will not help the driver or an autonomoussystem perform path planning. The other category was categorized as un-known. These videos included scenarios such as a rock or an object suddenlyhitting the windshield. Such scenarios occur with minimal warning and can-not be anticipated from the initial spatial and temporal features. The finaldata processing step was data augmentation. To perform the augmentation,we deconstructed the videos and applied augmentation to each frame beforere-compiling them as a video. Various combinations of augmentation tech-niques were applied to the data, such as rotation, horizontal flip, gaussianblur, gaussian noise, scaling, and random crop. Out of which, using onlyhorizontal flip provided the best results. Table 3.3 shows how the size of thedatasets evolved as data was processed. With the accident and non-accidentdata combined, CCD and DoTA have 4,500 and 7,677 videos, respectively.After removing videos that were irrelevant to our application, sizes werereduced to 3,801 and 5,353. Data augmentation was only applied to thetraining data, resulting in 6,080 videos in the training set and 761 in thetest set. For DoTA, the training set has 8,564 videos and 1,070 videos inthe test set.Name Original Post-processing Post-augmentationCCD 4,500 3,801 6,841DoTA 7,677 5,353 9,634Table 3.2: Size of the datasets at each stage of processing. Original refersto the combined count of accident and non-accident videos. The post-processing count refers to the size after irrelevant videos were discarded.And post-augmentation is the total dataset size after augmentation was ap-plied to the training data.In summary, we provide a novel algorithm for TTA forecasting. In orderto evaluate our approach we choose two datasets that provide video feed fromthe driver’s field of view. We perform three stages of pre-processing to thedata before passing them to our machine learning models. The processingtechniques clean the data by discarding irrelevant videos. To compensate forthe discarded videos we perform the final stage of pre-processing by applyingdata augmentation to enrich the dataset. In the following two chapters, we153.3. PRE-PROCESSINGpropose two deep-learning architectures which are used to learn from theprocessed video data.16Chapter 4TTA Forecasting with 3DCNNIn this chapter, we present our 3D CNN architecture for TTA forecasting.3D CNNs have proven to be a compelling end-to-end solution for videolearning; however, their performance comes with a high computational cost.For our application, we require a lightweight architecture with low inferencelatency. Such an architecture will allow our approach to be feasible andused in real-time. Hence, in this section, we propose Light 3D CNN (Li3D),a simple yet robust architecture. We first provide a brief overview of 3DCNNs in this chapter before delving into our proposed architecture.4.1 3D Convolutional Neural Network (CNN)3D CNNs are an extension of conventional (2D) CNNs designed for volu-metric data. They were developed for processing spatio-temporal data suchas videos, 3D medical images, or other applications involving a sequence offrames [SMB19]. The architecture is similar to the conventional CNN butwith an additional dimension. The three-dimensional input includes the in-put volume’s width, height, and depth. The 3D convolution operation isdirectly applied to the video to extract spatio-temporal features. Equation4.1 represents the convolution operation, where ϕ is the activation function,w represents the weights, and P,Q, and R are the three dimensions of thekernel. The output, vx,yi,l , is the activation value at (x,y) position of the lthfeature map in the ith layer. The value is obtained by applying the acti-vation function ϕ to the weighted sum of input values, where the weightswp,q,ri,j,m correspond to the kernel filters. The bias term bi,j is added to theweighted sum to provide a bias to the activation function. This 3D convo-lution operation is applied as a sliding window to a sequence of frames, andthe output is a 3D feature map.174.1. 3D CONVOLUTIONAL NEURAL NETWORK (CNN)kkoutputoutputoutputLLkkWWWHHHA)B)C)kkdFigure 4.1: Convolutional operations of 2D CNNs and 3D CNNsvx,yi,l = ϕbi,j +∑mPi−1∑p=0Qi−1∑q=0Ri−1∑r=0wp,q,ri,j,mv(x+p),(y+q),(z+r)(i−1),m (4.1)Figure 4.1 illustrates the process and output of 2D and 3D kernels. PartA represents a basic 2D feature map where a KxK kernel is slid across, andthe output is a 2D feature map. Part B illustrates the process of applyinga 2D CNN kernel over a sequence of frames where the output is still a 2Dfeature map that does not provide any temporal context. Finally, in part C,we can see a 3D kernel is applied to a sequence of frames, where the kernelis slid across the length of the sequence of frames resulting in a 3D featuremap. This 3D feature map captures the motion of the features over time,allowing the model to learn both the spatial and temporal features.184.2. LI3D: LIGHT 3D CNN4.2 Li3D: Light 3D CNNAs our approach requires high performance and efficiency for real-timeusage, we perform rigorous experiments to develop the parameters for anefficient 3D CNN architecture for our application that does not compromiseperformance. For example, our proposed architecture has exactly 1,843,489trainable parameters for image sizes of 72x128 and temporal depth of 4,whereas C3D has 128,328,065 parameters. This equates to 69.6x fewer pa-rameters for Li3D. Our experiments show that even with significantly fewerparameters, our architecture outperforms C3D. Figure 4.2, our proposedLight 3D CNN architecture (Li3D). The input to the model is a 3D tensorwhere width, height, and depth are the three dimensions. Depth refers tothe number of sequential frames in the video. The first two 3D convolutionallayers have 32 filters, a kernel size of 3x3x3, zero padding, a stride of 1, andReLU as the activation function. The convolutional layers are followed bya Max Pooling layer with a pool size of (3,3,3) and zero padding. Next, adropout layer with a rate of 0.25 follows the pooling layer. Dropout layersdiscard input weights with a frequency of 0.25 to prevent overfitting. Theoutput is then normalized before being passed to the next set of layers. Thesame sequence of convolution, Max Pooling, dropout, and normalizationis performed twice with 64 and 128 filters. After the final dropout layer,the output is flattened to convert the 3D tensor into a 1d tensor. The 1Dtensor is then passed to a fully connected layer with 512 neurons and an ac-tivation function of ReLU. Batch normalization is performed after the fullyconnected layer, followed by a dropout of 0.5. Finally, the output layer isfully connected with a single neuron and linear activation function.Compared to the state-of-the-art C3D architecture, Li3D is a straight-forward design that is easier to train and optimize. C3D has 8 convolutionallayers 5 pooling layers and 2 fully connected layers. The number of filtersfor the convolutional layers ranges from 64 to 512, and each fully connectedlayer has 4096 neurons. Such characteristics make the architecture complexand challenging to optimize due to large training times. Furthermore, C3Dapplies a 2x2x2 max pooling compared to 3x3x3 of Li3D. A larger pool sizecovers a larger area of the feature map, allowing the network to capturebetter global patterns and suppress irrelevant features [HZRS16]. Addition-ally, a larger pool size reduces the spatial dimension of feature maps andsubsequently reduces computational complexity. Finally, we used normal-ization in regular intervals; however, the original work on C3D does not usenormalization layers.Overall, our architecture focuses on reducing training time and compu-194.2. LI3D: LIGHT 3D CNNtational complexity as those characteristics are essential for our applicationand, at the same time, retain the high accuracy that comes with 3D convo-lution operations. However, our model still has limitations with regards tooperating with limited data and poor ability to learn long-range temporalcontext. We address these issues in the next chapter by proposing an ar-chitecture that can leverage transfer learning and learn long-range temporalcontext.204.2. LI3D: LIGHT 3D CNNWx Hx Dx 32Wx Hx Dx 643x3x31 x 1 x 1 x 5123x3x33x3x33x3x33x3x3Conv 3DMaxPoolingWx Hx Dx 1283x3x33x3x33x3x33x3x3Fully ConnectedLinearKernelFigure 4.2: Light 3D CNN (Li3D) Architecture21Chapter 5TTA Forecasting viaTransformerCNNs excel at learning spatial features, whereas Transformers excel atlearning temporal features. The benefit of 2D CNN over 3D CNN is theavailability of large-scale datasets (e.g., ImageNet) that allows for transferlearning. Hence, to combine the benefits of 2D CNN and transformers, wepresent a Hybrid Cnn-Transformer model in this chapter. The followingsection provides a brief overview of Transformers.5.1 Vision TransformerThe transformer architecture is designed based on the concept of self-attention. Self-attention enables the model to focus on particular parts ofthe input sequence. In other words, self-attention computes the weightedsum of the input sequence, where the learned weights reflect the impor-tance of each element in the sequence based on the task. The originalTransformer architecture proposed by Vaswani et al. [VSP+17] consists ofan encoder and decoder composed of self-attention layers and feed-forwardneural networks. The architecture was later adopted by [DBK+20] to de-velop Vision Transformer (ViT). ViT consists of a patch embedding layer, atransformer encoder, and a classification head. The patch embedding layerdivides the input image into a grid of non-overlapping patches, each treatedlike a token, resulting in a sequence of patch vectors. Positional embeddingis applied to the patches so they can be reconstructed to their original form.Linear projection is then applied to the patch vectors to a fixed-dimensionalembedding where each dimension is a learned feature. The weights of theprojection matrix are updated during training through backpropagation andgradient descent. Finally, the embeddings are passed as input to the Trans-former encoder. The transformer encoder is composed of Multi-ahead self-attention, an MLP network, and normalization layers as illustrated in Figure5.1. Multi-head self-attention is a variant of self-attention where each head is225.2. HYCT: HYBRID CNN-TRANSFORMER MODELFigure 5.1: Transformer Encoder Architecturea separate self-attention mechanism with individual learnable parameters.The output of each head is concatenated and passed to the feed-forwardneural network.5.2 HyCT: Hybrid CNN-Transformer modelFor our application, combining local spatial features and global tempo-ral features would be the ideal setup. To overcome the aforementioned chal-lenges with CNNs and Transformers, we propose a CNN-Transformer hybrid235.2. HYCT: HYBRID CNN-TRANSFORMER MODEL(CTH) architecture trained via transfer learning. By combining CNNs andTransformers, we utilize the local spatial features from the frames throughthe CNN and global temporal features via the Transformer. The addition oftransfer learning reduces the training time; for example, the HyCT networkrequires only 50 epochs for training, whereas C3D, and Li3D were trainedfor 150 and 200 epochs, respectively. Additionally, transfer learning allowsthe model to perform well with limited data, as shown in Chapter 6.As illustrated in Figure 5.2, we first use a ResNet50 network that lever-ages transfer learning from ImageNet to extract 3D features from our se-quences of frames. The trainable linear projection in ViT architecture learnsboth spatial and positional information. However, in our case, the CNNmodel already learns the spatial features, so we only require the positionalinformation. Hence, Positional embedding is applied to the feature mapsusing an embedding layer, allowing the model to remember the position ofeach frame. The features are then fed to our transformer encoder, whichconsists of a multi-head self-attention with eight heads and a dropout of0.40. The output is normalized and followed by two dense projection lay-ers, the first of which is an eight-dimensional layer and applies a GaussianError Linear Unit (Gelu) activation function to the input. The conventionis to use Rectified Linear Unit (ReLU) as the activation function. ReLU ismathematically defined as f(x) = max(0, x), which returns input for posi-tive values and otherwise returns zero. This causes a well know issue knownas the ”dying ReLU” problem. The problem arises when many inputs arenegative or extremely small positive numbers which then causes the neu-rons not to update their weights, and the model stops learning. GELU wasproposed as an alternative ReLU and is mathematically defined as f(x) = x* P( X ≤ x), where P(X ≤ x) is the cumulative distribution function of astandard Gaussian distribution. GELU has a non-zero gradient for negativeinputs, so it does not suffer from the dying ReLU problem. Additionally,for our experiments, GELU provided better results than ReLU. The outputfrom dense layers passed through a Global Max Pooling layer, resulting in afixed-length representation of the video. A dropout layer follows the poolinglayer and drops a fraction of the output to prevent overfitting. Finally, adense layer with a linear activation function takes the pooled output andproduces a scalar result. Layer normalization is performed after each stepto improve the stability of training.245.3. ACCIDENT SCENE RECOGNITION WITH TTA-ASSIST@X METRIC5.3 Accident Scene Recognition withTTA-ASSiST@x MetricThe ASiST@x (Accuracy at the xth frame Since Scene Transition) metric,introduced by [GTN21], is a novel accuracy measure explicitly designed forassessing the accuracy of models for driving scene recognition. When appliedto imbalanced datasets, the traditional accuracy metric for comparing thetrue and predicted labels is highly deceptive. For example, if a datasetconsists of 90% accident videos and 10% non-accident videos and the modelincorrectly predicts all non-accident videos, then the model accuracy will be90%. In contrast, the ASiST@x metric measures the percentage of framesaccurately predicted by the model from the scene transition until the arrivalof the xth frame. The original algorithm was designed for driving scenerecognition and relied on discrete classification labels. As our labels andoutputs are continuous values, we modify the original algorithm and proposeTTA-ASiST@x (Time-to-Accident-based Accuracy at the xth frame SinceScene Transition). Algorithm 2 presents our modified algorithm.Algorithm 2: TTA-ASiST@x Metric Calculationinput : K, Array [ti], Array [pi]output: lmatched at x, fcount at xInitialization: lent ← length of [ti] ;x← 0 // Index of next frame since true transitionlmatched at x← [0, ..., 0] // Zero array of size K+1fcount at x← [0, ..., 0] // Zero array of size K+1for i← 1 to lent doif (ti > 0 ∧ ti−1 ≤ 0) ∨ (ti < 0 ∧ ti−1 ≥ 0) thenx← 0 ; // If scene transition detected, set x to 0endif x ≤ K thenfcount at x[x]← fcount at x[x] + 1if (ti > 0 ∧ pi > 0) ∨ (ti < 0 ∧ pi < 0) thenlmatched at x[x]← lmatched at x[x] + 1endendx← x+ 1 ;endreturn lmatched at x, fcount at x255.3. ACCIDENT SCENE RECOGNITION WITH TTA-ASSIST@X METRICThe inputs to this algorithm are the arrays of true and predicted la-bels of each frame fi denoted by [ti] and [pi] respectively, along with thenumber of next frames since scene transition, denoted by K. TTA valuesless than zero are labeled as non-accident, and values greater than zero arelabeled as an accident. The length of the input arrays specifies the numberof frames in a driving session. Each session consists of multiple accidentand non-accident scenes. For a sequence of true labels, scene transition issaid to occur at frame index i when the TTA of ti is less than zero, andt1 − 1 is more than zero or vice versa. Based on this formulation, an in-dividual scene in a frame sequence Fi,j can be represented by a sequenceof the same valued true labels tm where m ∈ [i, . . . , j]. The algorithm’soutput is in the form of two arrays, lmatched at x and fcount at x, eachof size K + 1. Here, a size of K + 1 is selected to include fi, the scenetransition frame, along with the K next frames. For a particular index x,the array lmatched at x stores the number of frames since scene transitionwhere the predicted scene is the same as the true scene, while the arrayfcount at x stores the number of frames at index i + x, for x ∈ [0, . . . , n],since scene transition. For a session, the TTA-ASiST@x metric is defined asTTA − ASiST@x = (lmatched at x)/(fcount at x), where the two arrayslmatched at x and fcount at x are computed for each scene.The unique aspect of this metric is that along with measuring per-frameaccuracy, it can also quantify the model’s ability for continued scene recog-nition. This is particularly useful in the context of TTA as the model mustmaintain an accurate understanding of the occurring scene.As mentioned above, TTA-ASiST@x computes frame-wise accuracy. Weutilize a sliding window approach to apply this metric to evaluate the pro-posed TTA model’s performance. For example, for a contiguous sequenceof frames Fi,j , a sliding window Fi+l−1 of l frames can be built where l isthe temporal depth. This window is moved across the sequence of frames,one frame at a time, comparing the predicted TTA value with the groundtruth for that frame. This facilitates the evaluation of per-frame accuracyby utilizing the proposed metric.To summarize, this chapter presents HyCT, a TTA forecasting archi-tecture, and a scene recognition accuracy metric. HyCT compensates forthe limitations of Li3D by performing transfer learning that allows it toachieve high performance with limited data. Furthermore, HyCT combinesthe strong suits of CNNs and Transformers by providing a model that canefficiently learn both spatial and long-range temporal data. The final sectionof this chapter provides a scene recognition accuracy metric that measuresthe ongoing scene recognition accuracy. Such a metric is essential for our265.3. ACCIDENT SCENE RECOGNITION WITH TTA-ASSIST@X METRICapproach as it allows us to mitigate the risk of false alarms while also eval-uating how the model performs during a scene transition. The followingchapter outlines the experiments performed to evaluate our approach andpresents the empirical results.275.3. ACCIDENT SCENE RECOGNITION WITH TTA-ASSIST@X METRICFigure 5.2: Proposed Hybrid CNN-Transformer (HyCT) Network28Chapter 6Experiments & ResultsThis chapter presents the empirical analysis of our proposed approachand architectures. The following section explains the parameters of ourexperimental setup. The subsequent segments present our comprehensiveset of experiments and results. Finally, we culminate the thesis with aconcise summary of our findings and a brief discussion of our future work.6.1 Experimental SetupWe aim to investigate whether the Spatio-temporal information from Nframes can predict the time-to-accident and identify accident scenes. Tomake our method applicable to real-world scenarios, we limit the numberof frames to 10, corresponding to 1 second. Considering additional frameswould not give the driver enough time to take action to prevent the accident,given that the prediction horizon is between 30 to 60 frames. Additionally,we vary the frame resolution to test our approach further by consideringframe resolutions of 36x64, 72x128, and 144x256. We want to note thatexperiments with a resolution of 144x256 were not conducted for C3D andLi3D due to limited computational resources, as C3D is extremely computa-tionally heavy. Although Transformers are computationally expensive, thetransfer learning component allowed us to conduct experiments with framesizes of 144x256. All experiments were conducted on a system with an IntelXeon processor, 51 GB of RAM, and an Nvidia V100 GPU. Normalizationis applied to the images and labels before being passed to the networks. Dueto the stochastic nature of neural network training, we present the averageof 5 runs for each experiment.6.2 Spatio-temporal AnalysisWe compare the prediction errors with different spatial resolutions andtemporal depths for C3D, Li3D, and HyCT models. Some experiments in296.2. SPATIO-TEMPORAL ANALYSISthis section were conducted with the assistance of Lious Chirade 1 and DayaKumar 2. We report the Root Mean Squared Error (RMSE) and Mean Ab-solute Error (MAE) values. RMSE represents the variation in error, andMAE represents the absolute average prediction error. C3D will act as thebase model for comparison. Our implementation of the C3D architecture isidentical to the original paper [TRS+17]. For C3D, the experiments weretrained for 150 epochs with a batch size of 64, and Stochastic GradientDescent (SGD) was used for optimization. The Li3D model was trainedfor 200 epochs with a batch size of 64 and Adam as the optimizer. HyCTwas trained for only 50 epochs with a batch size of 32 and optimized withthe Adam algorithm. All three models had a learning rate of 0.001, MeanSquared Error as the loss function, and used two callbacks: Early Stoppingand Learning Rate Scheduler. Early stopping prevents over-fitting by halt-ing training when validation loss stops improving for a certain number ofepochs. The learning rate scheduler reduces the learning rate by a factor of0.20 if the validation loss does not improve. As discussed in the previoussection, we increase our training data by two folds through horizontal fliptransformation.Tables 6.1 6.2 present the RMSE and MAE values for our experimentson CCD and DoTA, respectively. Regarding temporal depth, C3D providespoor results compared to Li3D and HyCT for 2-6 frames. This holds forboth datasets as shown in tables 6.1 and 6.2. However, for C3D, we noticea significant improvement as temporal depth increases across both datasets.For example, for a spatial resolution of 36x64 on CCD, C3D shows a 45%improvement in MAE as frames are increased from 6 to 8. The improvementis 49% for the higher resolution as frames are increased from 4 to 6. Theimprovement in RMSE is also significant but not as drastic. A similartrend is seen for DoTA. We notice an improvement in Li3D across the twodatasets for both MAE and RMSE as temporal depth increases, but thechange is not substantial. However, we see that for 2 and 4 frames, Li3Dperforms significantly better than the other two models. The performanceimprovement may be due to larger kernel sizes that allow the model tocapture global patterns and ignore irrelevant features.1Taif Anjum., Louis Chirade., Beiyu Lin., and Apurva Narayan. Learning spatio-temporal features via 3d cnns to forecast time-to-accident. In Proceedings of the 15thInter- national Conference on Agents and Artificial Intelligence - Volume 3: ICAART,,pages 532–540. INSTICC, SciTePress, 20232Taif Anjum., Daya Kumar., and Apurva Narayan. Spatio-temporal analysis of dash-board camera videos for time-to- accident forecasting. In 2023 International Joint Con-ference on Neural Networks (IJCNN), pages 1–8, 2023.306.2. SPATIO-TEMPORAL ANALYSISHyCT HyCT C3D C3D Li3D Li3DResolution Frames RMSE MAE RMSE MAE RMSE MAE36x64 2 1.126 0.495 1.229 0.802 0.906 0.38636x64 4 1.102 0.553 1.215 0.769 0.872 0.31936x64 6 0.898 0.524 1.060 0.614 0.885 0.32536x64 8 1.106 0.556 0.902 0.333 0.900 0.33436x64 10 1.000 0.445 0.941 0.353 0.915 0.33772x128 2 1.109 0.523 1.130 0.703 0.858 0.36972x128 4 1.195 0.741 1.127 0.676 0.842 0.31272x128 6 1.096 0.519 0.952 0.347 0.871 0.34072x128 8 1.010 0.443 0.926 0.338 0.822 0.31572x128 10 0.798 0.449 0.990 0.365 0.819 0.300144x256 2 1.006 0.647 - - - -144x256 4 0.896 0.364 - - - -144x256 6 0.814 0.359 - - - -144x256 8 0.603 0.248 - - - -144x256 10 0.651 0.281 - - - -Table 6.1: Mean Absolute and Root Mean Squared Errors on CCD withvarying Spatial Resolution and Temporal DepthFigure 6.1: Mean Absolute Error Visualization for CCD316.2. SPATIO-TEMPORAL ANALYSISFigure 6.2: Root Mean Squared Error Visualization for CCDHyCT HyCT C3D C3D Li3D Li3DResolution Frames RMSE MAE RMSE MAE RMSE MAE36x64 2 1.805 1.141 2.081 1.610 1.545 0.96936x64 4 1.824 1.125 2.047 1.560 1.521 0.88636x64 6 1.844 1.195 2.193 1.515 1.505 0.83136x64 8 1.742 1.174 1.479 0.811 1.496 0.80936x64 10 1.863 1.117 1.467 0.801 1.498 0.82172x128 2 1.704 1.228 2.138 1.660 1.535 0.87972x128 4 1.604 1.026 2.003 1.413 1.510 0.80272x128 6 1.587 1.091 1.903 1.262 1.517 0.81972x128 8 1.634 1.002 1.473 0.800 1.457 0.78672x128 10 1.535 0.981 1.514 0.834 1.490 0.822144x256 2 1.523 0.886 - - - -144x256 4 1.607 0.911 - - - -144x256 6 1.506 0.876 - - - -144x256 8 1.456 0.832 - - - -144x256 10 1.352 0.822 - - - -Table 6.2: Mean Absolute and Root Mean Squared Errors on DoTA withvarying Spatial Resolution and Temporal Depth326.2. SPATIO-TEMPORAL ANALYSISFigure 6.3: Mean Absolute Error Visualization for DoTAFigure 6.4: Root Mean Squared Error Visualization for DoTAThe spatial resolution also affects the prediction error, but similar to thetrend with the number of frames that change is not particularly substantial.Regardless, with a resolution of 72x128 and 10 frames, Li3D outperformsC3D on CCD with an MAE and RMSE value of 0.819s and 0.300s, respec-tively. For DoTA the best Li3D model (72x128,8) outperforms both C3Dand HyCT.336.2. SPATIO-TEMPORAL ANALYSISHyCT also shows that temporal context improves prediction error assignificant improvement is seen across both datasets. HyCT provided thebest results with 8 and 10 frames and outperformed C3D for the lower frameexperiments. Regarding spatial resolution, for HyCT, there is no significantimprovement between 36x64 and 72x128 for CCD. However, as the resolutionis increased to 144x256, we notice that the (144x256,8) model outperformsall C3D and Li3D models by a significant margin. For the DoTA dataset,HyCT consistently improves as the spatial resolution is increased to 144x256.However, for DoTA, the best C3D model outperformed HyCT in terms ofMAE by a small margin, but HyCT’s RMSE was lower.In summary, the best HyCTmodel outperforms all C3D and Li3D modelson the CCD dataset with MAE and RMSE values of 0.248s and 0.603s,respectively. For DoTA, the best Li3D model outperforms HyCT and C3Dfor all experiments.The discrepancy between the results on the two datasets can be becauseCCD has fewer data samples than DoTA, which prevented C3D and Li3Dfrom performing as well as HyCT. Both Li3D and C3D is trained fromscratch and can tend to overfit with a smaller dataset. In contrast, DoTAis a far richer dataset, due to which the pre-trained weights did not helpHyCT, enabling C3D and Li3D to learn better representations by trainingfrom scratch. Another observation is that C3D and Li3D perform betterwith lower resolution in some cases than HyCT. This is because the imagesin ImageNet are of size 224x224, as we leveraged transfer learning from Im-ageNet, it could not extract valuable features from lower-resolution frames.Another observation is that the prediction errors for DoTA are significantlyhigher than CCD. This is because DoTA has longer videos with an averageprediction error of 6 seconds compared to 3 seconds for CCD.We also notice in Figures 6.1 6.2 6.3 and 6.4 that RMSE values aresignificantly higher than MAE values. RMSE is sensitive to large errors aswe square the errors before averaging them. That means when are somelarge errors on RMSE, emphasize them. MAE is more interpretable thanRMSE as it directly calculates the average of absolute errors, and the unitstays the same. RMSE on the other hand is less interpretable as it squaresthe values before taking root; the error value is not the same unit as theoriginal data.We note that there isn’t a piece-wise monotonic relation between tem-poral depth and mean absolute error. Our findings align with the studies[MWOBK19] where they conclude that length of temporal history does notnecessarily increase or decrease error in prediction for applications such astrajectory prediction or robot-pedestrian collision. Overall our experiments346.3. COMPARATIVE ANALYSISprovide clear evidence that higher spatial resolution and longer temporalcontext improve both MAE and RMSE.6.3 Comparative AnalysisWe perform extensive experiments against state-of-the-art CNN archi-tectures proposed for similar applications to examine the robustness of ourproposed method. For a fair comparison, we train and fine-tune the CNNarchitectures on CCD and DoTA.6.3.1 Static Image ArchitecturesWe compare our work against two types of 2D CNN architectures: VGGand ResNet. For a baseline, we implement the conventional VGG-16 [SZ14]to explore the impact of spatial features compared to Spatio-temporal fea-tures. The implementation was identical to the original work, except wereplaced the softmax-based output with a dense linear layer and used MSEas the loss function. The model passed 224x224 RGB images as input. Thetraining was conducted for 100 epochs with a batch size of 32 and a learningrate of 0.001.The ResNet-8 architecture was proposed for collision avoidance in drones[LMDBS18]. In the original work, the model is fed a single image, generatinga steering angle and a collision probability to recognize and avoid collisions.In our implementation, we replaced the two output layers with a singleregression layer that produces the time-to-accident estimation. Similar tothe original work, 224x224 greyscale images were used with a learning rateof 0.0001, with MSE as the loss function and Adam as the optimizer.6.3.2 Video Learning ArchitecturesThe work by [MWOBK19] proposes a multi-stream VGG-16 for robot-pedestrian near-collision scenarios. Their model extracts spatial featuresfrom N-frames, concatenates them to learn the motion information, andfeeds them to a fully connected layer before being passed to the outputlayer. As the authors found 6 frames to perform the best, we compareour model against 6-Stream VGG-16. The model was initialized with pre-trained weights from ImageNet, similar to the original work. The authorsfine-tuned the model on PASCAL VOC as ImageNet does not have a Personclass. However, as our application is based on traffic accidents and ImageNetcontains a vehicle class, we skipped the fine-tuning step. Similar to the356.3. COMPARATIVE ANALYSISFigure 6.5: Comparative Analysis on CCDFigure 6.6: Comparative Analysis DoTA366.3. COMPARATIVE ANALYSISMethod RMSE (s) MAE (s)VGG-16 (1 frame) 1.279 0.882VGG-16 (6 frames) 1.456 0.750ResNet-8 (1 frame) 1.029 0.637CNN-RNN (16 frames) 1.078 0.388C3D (8 frames) 0.902 0.333Li3D (10 frames) 0.819 0.300HyCT (8 frames) 0.603 0.248Table 6.3: TTA estimation on CCD: Comparison of our work with sin-gle frame VGG-16, multi-stream stream VGG-16 [MWOBK19], ResNet-8[LMDBS18], CNN-RNN and C3D [TRS+17]Method RMSE (s) MAE (s)VGG-16 (1 frame) 1.62 1.46VGG-16 (6 frames) 1.33 1.35ResNet-8 (1 frame) 1.64 1.01CNN-RNN (16 frames) 1.51 0.95C3D (8 frames) 1.47 0.80HyCT (10 frames) 1.34 0.82Li3D (8 frames) 1.46 0.79Table 6.4: TTA estimation on DoTA: Comparison with single frame VGG-16, N-stream VGG-16 [MWOBK19], ResNet-8 [LMDBS18], CNN-RNN andC3D [TRS+17]original work, we used 224x224 RGB images, SGD as the optimizer, andMSE as the loss function. A learning rate of 0.001 and the model wastrained for 50 epochs as these parameters performed the best.A combination of CNN with an RNN variant is typically used for videoclassification apart from 3D-CNNs due to their computational efficiency.RNN models have variations such as Long Short-Term Memory (LSTM),Gated Recurrent Unit (GRU), Initialized RNNs, and Convolutional LSTM.For our purpose, we use a GRU-based RNN; GRU was considered overLSTM because it controls the flow of the information, does not require amemory unit, and is better in terms of computational efficiency and perfor-mance [BBA22]. For the spatial features, we used an Inception V3 modelpre-trained on ImageNet to extract the features from our traffic accidentdatasets. The features are then fed to a sequence model with two GatedRecurrent Unit layers (GRU) with 16 and 8 neurons, respectively, followedby a dropout layer and a fully connected layer with 8 neurons; finally, all376.4. TTA-ASIST@X ANALYSISthe features are fed to the output layer with a linear activation function.RNN-CNN model performed best with 16 frames; hence, we report thoseresults in Tables 6.3 and 6.4.We report the results of our comparative analysis in Tables 6.3, 6.4 andillustrate them in Figures 6.5, 6.6. In almost every case, the multi-framemodels outperform the single-frame models. The only exception is the single-frame ResNet-8 which surpassed the multi-frame VGG-16 in MAE for bothdatasets. As a shallower, less complex model, ResNet-8 may have preventedoverfitting compared to VGG-16. However, the lower RMSE of the VGG-16model suggests more stable training.On the smaller dataset, CCD, HyCT was the clear winner comparedto all the other models. Whereas on the more complex and larger DoTAdataset, Li3D came out on top. Our results indicate that leveraging Spatio-temporal features can provide better performance than only using spatialfeatures. Adding to the results of the spatio-temporal analysis, it is evidentthat spatio-temporal are effective for TTA forecasting.6.4 TTA-ASiST@x AnalysisFigure 6.7: TTA-ASiST@x Analysis on CCD386.4. TTA-ASIST@X ANALYSISFigure 6.8: TTA-ASiST@x Analysis on DoTAFigures 6.7 and 6.8 present our analysis of the CCD and DoTA datasets,respectively. We perform our TTA-ASiST@x analysis on only 1/4th of thedata. As video learning requires large amounts of frames to be processedwith computationally heavy architectures, models that can provide highperformance with limited data are always desired. Additionally, we aredealing with only two classes, so using a quarter of the data will make thetask more challenging. We also note that we only report TTA-ASSiST@xresults using the HyCT model. C3D and Li3D models could not learn bothclasses with only 1/4 of the data. Hence we discarded the results.Our analysis shows that the models with lower temporal depth performedsignificantly well for lower values of x. An x value of 0 indicates the truetransition, meaning the model detected a transition as soon as it occurred.The model (144x256, 2) could recognize the scene at the transition point with63% and 42% accuracy on CCD and DoTA, respectively. In comparison,(144x256, 10) achieved a 30% and 28% accuracy. This is because when weuse a sliding window of 10 frames to predict the 10th frame, which is the truetransition point, the remaining 9 frames are from the previous scene. Thismakes it challenging for the model to recognize the scene at the transitionpoint. However, with a temporal depth of 2, one frame is from the previousscene, and the other is from the current scene. This is more apparent when396.4. TTA-ASIST@X ANALYSISwe notice that when x is 10, the best performing models are 144x256, 10with 85% (CCD) and 90% (DOTA) accuracy as they have the most spatio-temporal context. For both datasets, we notice similar trends and a clearindication that spatial features and temporal context influence the model’soutcome. Our model performed exceptionally well since we used only 1/4thof the data. TTA-ASiST@x metric is essential for our work as it allowsus to measure the models’ performance when dealing with scene transition.Additionally, it gives us an insight on how many frames are required for thenew scene to correctly predict the transition.In this chapter, the findings strongly support our hypothesis that spatio-temporal features are effective for predicting TTA. Furthermore, it is evidentthat the performance of the model relies on both spatial and temporal fea-tures.40Chapter 7Conclusion & Future workWe propose a novel approach to forecast time-to-accident (TTA) byleveraging Spatio-temporal features extracted from traffic accident videos.We present Algorithm 1 for calculating TTA. Our approach relies on inex-pensive and easy-to-install dashboard cameras instead of expensive depthimaging devices or sensors that require expert installation. Our methodonly uses the first N-frames where N is at most 10 (1 second); this allowsthe driver enough time to take action to mitigate the risks, given the pre-diction horizon is between 3-6 seconds.Additionally, we present two efficient architectures, i.e., Li3D and HyCT,where each has characteristics that lower computational complexity andtraining time. Li3D (Light 3D CNN) has fewer parameters and larger ker-nels, allowing it to learn important features efficiently. Our experimentsshow that even with fewer parameters, the model does not compromise onaccuracy. To further improve our model’s performance and efficiency, trans-fer learning is one of the most viable approaches. However, due to thelack of large-scale video datasets relevant to our task, we could not utilizetransfer learning. Furthermore, 3D CNNs require three-dimensional input,preventing us from using large-scale image datasets. That inspired us to pro-pose the HyCT (Hybrid CNN-Transformer) network, which outperformed allother deep learning models on the smaller CCD dataset. The transfer learn-ing component enabled the use of pre-trained ImageNet weights, making itwell-suited for our dataset. While estimating TTA, our model can simultane-ously recognize accident and non-accident scenes. In real-world situations,it is crucial to recognize when a normal driving scenario transitions intoa potential accident situation. Hence, we evaluate our approach’s ongoingscene recognition ability using the TTA-ASiST@x metric. With only a quar-ter of the data, the HyCT model could recognize accident and non-accidentscenes with over 85% accuracy. This set of experiments further highlightedthe HyCT network’s performance capability on limited data. The proposedarchitectures can enable our approach to be implemented in real-time sce-narios where minimum inference latency, low computational cost, and highaccuracy are necessary. Also, lightweight architectures can make finding the41optimal spatio-temporal parameters easier.Comparing the results of our multi-frame experiments against the single-frame experiments, there is clear evidence that Spatio-temporal featuresperform better than using only spatial features. Our comprehensive ex-periments suggest that spatial and temporal features play a vital role inforecasting TTA.As a part of future work, we will work on the interpretability of our modelto analyze the features that impact the prediction error. To achieve this var-ious interpretability techniques can be applied such as feature importanceanalysis, partial dependence plots [HTF09], and SHAP (Shapley AdditiveexPlanations) [LEC+20] values. These methods will allow us to identify themost influential features in the prediction process. By gaining insights intothese features, we can not only explain the model’s decision-making processbut also identify potential areas for improvement and fine-tuning. Addition-ally, we may explore the use of advanced visualization techniques, such asheatmaps or attention mechanisms, to provide a more intuitive representa-tion of the model’s reasoning and highlight the specific regions or factorswithin the input data that have the most impact on the prediction error.The next phase of our future work would be to integrate our modelwith an accident localization framework to detect various road users thatpose a collision threat. To achieve this, we will leverage techniques fromcomputer vision and object detection algorithms. By analyzing real-timevideo or image data captured from cameras mounted on vehicles or trafficinfrastructure, our integrated system will be able to identify and track dif-ferent road users, such as pedestrians, cyclists, and vehicles. The accidentlocalization framework will provide contextual information, such as the po-sitions, velocities, and trajectories of road users, which can then be fed intoour model for collision threat prediction. This integration will enable oursystem to not only predict potential collision risks but also provide valu-able insights into the specific objects or individuals that are contributing tothose risks. It will facilitate proactive measures for accident prevention andenhance overall road safety.Finally, we would like to implement our approach in real-world scenariosand assess the feasibility of our solution in real-time. While our currentresearch has primarily focused on model development and validation, it isessential to assess the practicality and effectiveness of our solution in real-world environments.42Bibliography[ACLN23] Taif Anjum., Louis Chirade., Beiyu Lin., and ApurvaNarayan. Learning spatio-temporal features via 3d cnns toforecast time-to-accident. In Proceedings of the 15th Inter-national Conference on Agents and Artificial Intelligence -Volume 3: ICAART,, pages 532–540. INSTICC, SciTePress,2023. → pages v[ADH+21] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun,Mario Lucˇic´, and Cordelia Schmid. Vivit: A video visiontransformer. In Proceedings of the IEEE/CVF InternationalConference on Computer Vision, pages 6836–6846, 2021. →pages 8, 9[AKN23] Taif Anjum., Daya Kumar., and Apurva Narayan. Spatio-temporal analysis of dashboard camera videos for time-to-accident forecasting. In 2023 International Joint Conferenceon Neural Networks (IJCNN), pages 1–8, 2023. → pages v[ALN22] Taif Anjum, Beiyu Lin, and Apurva Narayan. Leveragingspatio-temporal features to forecast time-to-accident. In Pro-ceedings of the 30th International Conference on Advances inGeographic Information Systems, pages 1–2, 2022. → pagesv[AvS06] Letty Aarts and Ingrid van Schagen. Driving speed and therisk of road crashes: A review. Accident Analysis & Preven-tion, 38:215–224, 2006. → pages 2[BBA22] Behnaz Bahmei, Elina Birmingham, and Siamak Arzanpour.Cnn-rnn and data augmentation using deep convolutionalgenerative adversarial network for environmental sound clas-sification. IEEE Signal Processing Letters, 29:682–686, 2022.→ pages 3743Bibliography[Bel22] B. Beltz. 100+ car accident statistics. https://safer-america.com/car-accident-statistics/impact,2022. Accessed on March 25, 2023. → pages 1[BGO+16] Alex Bewley, ZongYuan Ge, Lionel Ott, Fabio Ramos, andBen Upcroft. Simple online and realtime tracking. CoRR,abs/1602.00763, 2016. → pages 6[BYK20] Wentao Bao, Qi Yu, and Yu Kong. Uncertainty-based trafficaccident anticipation with spatio-temporal relational learn-ing. In Proceedings of the 28th ACM International Confer-ence on Multimedia, pages 2682–2690, 2020. → pages 1, 7,10, 13[BZMA20] Alexei Baevski, Yiming Zhou, Abdelrahman Mohamed, andMichael Auli. Wav2vec 2.0: A framework for self-supervisedlearning of speech representations. In Advances in NeuralInformation Processing Systems, volume 33, pages 12449–12460, 2020. → pages 8[CCXS16] Fu-Hsiang Chan, Yu-Ting Chen, Yu Xiang, and Min Sun.Anticipating accidents in dashcam videos. In Asian Confer-ence on Computer Vision, pages 136–153. Springer, 2016. →pages 1, 7, 13[DAHG+15] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama,Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko,and Trevor Darrell. Long-term recurrent convolutional net-works for visual recognition and description. In Proceedingsof the IEEE conference on computer vision and pattern recog-nition, pages 2625–2634, 2015. → pages 4[DBK+20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, DirkWeissenborn, Xiaohua Zhai, Thomas Unterthiner, ..., andNeil Houlsby. An image is worth 16x16 words: Trans-formers for image recognition at scale. arXiv preprintarXiv:2010.11929, 2020. → pages 8, 22[DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and KristinaToutanova. Bert: Pre-training of deep bidirectionaltransformers for language understanding. arXiv preprintarXiv:1810.04805, 2018. → pages 844Bibliography[FXM+21] Haofeng Fan, Boqing Xiong, Karttikeya Mangalam, YanghaoLi, Zhicheng Yan, Jitendra Malik, and Christoph Feichten-hofer. Multiscale vision transformers. In Proceedings of theIEEE/CVF International Conference on Computer Vision,pages 6824–6835, 2021. → pages 9[GTN21] Ramashish Gaurav, Bryan Tripp, and Apurva Narayan. Driv-ing scene understanding: How much temporal context andspatial resolution is necessary? In Canadian Conference onAI, 2021. → pages 5, 25[GZL22] Chengyuan Gou, Yubin Zhou, and Dong Li. Driver atten-tion prediction based on convolution and transformers. TheJournal of Supercomputing, 78(6):8268–8284, 2022. → pages9[Hay72] John C Hayward. Near miss determination through use of ascale of danger. 1972. → pages 6[HTF09] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. TheElements of Statistical Learning: Data Mining, Inference,and Prediction. Springer, 2nd edition, 2009. → pages 42[HZRS16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.Deep residual learning for image recognition. In Proceedingsof the IEEE conference on computer vision and pattern recog-nition, pages 770–778. IEEE, 2016. → pages 19[JNG13] Felipe Jime´nez, Jose´ Eugenio Naranjo, and Fernando Garc´ıa.An improved method to calculate the time-to-collision of twovehicles. International Journal of Intelligent TransportationSystems Research, 11(1):34–42, 2013. → pages 6[JXYY12] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolu-tional neural networks for human action recognition. IEEEtransactions on pattern analysis and machine intelligence,35(1):221–231, 2012. → pages 8[KLHY22] Min-Gi Kang, Won-Yong Lee, Kyungho Hwang, and Young-Sup Yoon. Vision transformer for detecting critical situa-tions and extracting functional scenario for automated vehi-cle safety assessment. Sustainability, 14(15):9680, 2022. →pages 945Bibliography[LEC+20] Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex De-Grave, Jordan M. Prutkin, Bharath Nair, ..., and Su-In Lee.Explainable ai for trees: From local explanations to global un-derstanding. Nature Machine Intelligence, 2(1):56–67, 2020.→ pages 42[LMDBS18] Antonio Loquercio, Ana I Maqueda, Carlos R Del-Blanco,and Davide Scaramuzza. Dronet: Learning to fly by driv-ing. IEEE Robotics and Automation Letters, 3(2):1088–1095,2018. → pages viii, 35, 37[MWOBK19] Aashi Manglik, Xinshuo Weng, Eshed Ohn-Bar, and Kris MKitanil. Forecasting time-to-collision from monocular video:Feasibility, dataset, and challenges. In 2019 IEEE/RSJ In-ternational Conference on Intelligent Robots and Systems(IROS), pages 8081–8088. IEEE, 2019. → pages viii, 6, 12,34, 35, 37[RD16] Hani Rakha and Thomas A Dingus. The impact of the 85thpercentile rule on time headways. Transportation ResearchPart F: Traffic Psychology and Behaviour, 42:194–205, 2016.→ pages 2[RSK+23] Katta Ramana, Gaurav Srivastava, Manoj R Kumar,Thippa Reddy Gadekallu, Jerry Chun-Wei Lin, MamounAlazab, and Cosmos Iwendi. A vision transformer approachfor traffic congestion prediction in urban areas. IEEE Trans-actions on Intelligent Transportation Systems, 2023. → pages9[SAMK18] Sarthak Sharma, Junaid Ahmed Ansari, J. Krishna Murthy,and K. Madhava Krishna. Beyond pixels: Leveraging geom-etry and shape cues for online multi-object tracking. CoRR,abs/1802.09298, 2018. → pages 6[SGM22] Karim Saleh, Alexander Grigorev, and Alexandru Sorin Mi-haita. Traffic accident risk forecasting using contextual visiontransformers. In 2022 IEEE 25th International Conference onIntelligent Transportation Systems (ITSC), pages 2086–2092.IEEE, October 2022. → pages 946Bibliography[SJE+22] Josep Selva, Alexander Skovsbo Johansen, Sergio Es-calera, Kamal Nasrollahi, Thomas B Moeslund, and Al-bert Clape´s. Video transformers: A survey. arXiv preprintarXiv:2201.05991, 2022. → pages 9[SKAS18] Tomoyuki Suzuki, Hirokatsu Kataoka, Yoshimitsu Aoki, andYutaka Satoh. Anticipating traffic accidents with adaptiveloss and large-scale incident db. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages3521–3529, 2018. → pages 1, 7[SMB19] Rahul Dev Singh, Ajay Mittal, and Rajesh K Bhatia. 3dconvolutional neural network for object recognition: a review.Multimedia Tools and Applications, 78:15951–15995, 2019. →pages 17[SZ14] Karen Simonyan and Andrew Zisserman. Very deep convo-lutional networks for large-scale image recognition. arXivpreprint arXiv:1409.1556, 2014. → pages 35[TBF+15] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani,and Manohar Paluri. Learning spatiotemporal features with3d convolutional networks. In Proceedings of the IEEE in-ternational conference on computer vision, pages 4489–4497,2015. → pages 8[TII22] Highway Safety The Insurance Institute. Real-world benefitsof crash avoidance technologies, 2022. → pages 1[TRS+17] Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, andManohar Paluri. Convnet architecture search for spatiotem-poral feature learning. arXiv preprint arXiv:1708.05038,2017. → pages viii, 8, 30, 37[TSSPdF22] Daniel Tøttrup, Stinus Lykke Skovgaard, Jonas le Fevre Se-jersen, and Rui Pimentel de Figueiredo. A real-time methodfor time-to-collision estimation from aerial images. Journalof Imaging, 8(3):62, 2022. → pages 6[VLS17] Gu¨l Varol, Ivan Laptev, and Cordelia Schmid. Long-term temporal convolutions for action recognition. IEEEtransactions on pattern analysis and machine intelligence,40(6):1510–1517, 2017. → pages 847Bibliography[VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-reit, Llion Jones, Aidan N Gomez, ..., and Illia Polosukhin.Attention is all you need. In Advances in Neural InformationProcessing Systems, pages 5998–6008, 2017. → pages 8, 22[WHO18] WHO World Health Organization. Global status report onroad safety, 2018. → pages 1[YCW+20] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, YingyingChen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.Bdd100k: A diverse driving dataset for heterogeneous mul-titask learning. In Proceedings of the IEEE/CVF conferenceon computer vision and pattern recognition, pages 2636–2645,2020. → pages 13[YHNHV+15] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vi-jayanarasimhan, Oriol Vinyals, Rajat Monga, and GeorgeToderici. Beyond short snippets: Deep networks for videoclassification. In Proceedings of the IEEE conference on com-puter vision and pattern recognition, pages 4694–4702, 2015.→ pages 8[YWX+22] Yu Yao, Xizi Wang, Mingze Xu, Zelin Pu, Yuchen Wang, EllaAtkins, and David Crandall. Dota: unsupervised detectionof traffic anomaly in driving videos. IEEE transactions onpattern analysis and machine intelligence, 2022. → pages 10,13[YXW+19] Yu Yao, Mingze Xu, Yuchen Wang, David J Crandall, andElla M Atkins. Unsupervised traffic accident detection infirst-person videos. In 2019 IEEE/RSJ International Confer-ence on Intelligent Robots and Systems (IROS), pages 273–280. IEEE, 2019. → pages 1348
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Narayan, Apurva
Braun, John, 1963-
Leveraging spatio-temporal features to forecast time-to-accident
Text