Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Shojaei Hashemi, Anahita
Doctor of Philosophy - PhD
University of British Columbia
Remote health monitoring is evolving from a luxury to a necessity in residential buildings and smart homes. As the early detection of fall incidents can prevent mortality and alleviate morbidity, automatic fall detection systems are crucial for remote health monitoring. Providing rich content at an affordable cost, video cameras have become prevalent for fall detection. Most video-based fall detection models use classic machine learning, relying on handcrafted that limit their performance and generalization capability. In this thesis, we propose three novel video-based fall detection methods to solve the mentioned weaknesses, while considering privacy concerns associated with video data. Using deep learning, our models automatically learn the most discriminative features without resorting to definitions and assumptions. The first method is among the early video-based fall detection algorithms that employ deep learning. Consisting of a long short-term memory (LSTM) neural network, it works with depth cameras and uses the three-dimensional (3D) locations of major body joints to address privacy concerns. The second method is designed to operate with RGB cameras, targeting applications that require the identity of the fallen person to provide personalized medical help. It is comprised of a two-dimensional (2D) convolutional neural network (CNN) that extracts features from the frames and feeds them to an LSTM neural network. To address privacy concerns, edge computing is essential for models working with RGB videos to avoid video transmission. In designing our third method, we paid specific attention to this aspect by keeping the computational complexity of the model so low that it can be implemented on cameras built-in boards without losing its real-time performance. Spatial features extracted from the frames by a lightweight, yet powerful 2D CNN are sequentially processed by a transformer. Evaluations prove the competing performance of our models with regards to the state-of-the-art methods. We also present an embedded system prototype we have built to verify the capability of our RGB models to perform in real time on limited computational resource. Finally, we introduce the UBC-DML Fall Dataset, a large comprehensive video dataset we have captured to address the shortcomings in the current public datasets for fall detection.
VIDEO-BASED HUMAN FALL DETECTION IN INDOOR SPACES FOR HEALTH MONITORING by Anahita Shojaei Hashemi A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Electrical and Computer Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) April 2023 © Anahita Shojaei Hashemi, 2023 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: Video-based Human Fall Detection in Indoor Spaces for Health Monitoring submitted by Anahita Shojaei Hashemi in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Electrical and Computer Engineering Examining Committee: Dr. Panos Nasiopoulos, Professor, Department of Electrical and Computer Engineering, UBC Supervisor Dr. James J. Little, Professor, Department of Computer Science, UBC Co-supervisor Dr. Shahriar Mirabbasi, Professor, Department of Electrical and Computer Engineering, UBC Supervisory Committee Member Antony Hodgson, Professor, Department of Mechanical Engineering, UBC University Examiner Michael Van De Panne, Professor, Department of Computer Science, UBC University Examiner Additional Supervisory Committee Members: Dr. Victor Leung, Professor, Department of Electrical and Computer Engineering, UBC Supervisory Committee Member iii Abstract Remote health monitoring is evolving from a luxury to a necessity in residential buildings and smart homes. As the early detection of fall incidents can prevent mortality and alleviate morbidity, automatic fall detection systems are crucial for remote health monitoring. Providing rich content at an affordable cost, video cameras have become prevalent for fall detection. Most video-based fall detection models use classic machine learning, relying on handcrafted that limit their performance and generalization capability. In this thesis, we propose three novel video-based fall detection methods to solve the mentioned weaknesses, while considering privacy concerns associated with video data. Using deep learning, our models automatically learn the most discriminative features without resorting to definitions and assumptions. The first method is among the early video-based fall detection algorithms that employ deep learning. Consisting of a long short-term memory (LSTM) neural network, it works with depth cameras and uses the three-dimensional (3D) locations of major body joints to address privacy concerns. The second method is designed to operate with RGB cameras, targeting applications that require the identity of the fallen person to provide personalized medical help. It is comprised of a two-dimensional (2D) convolutional neural network (CNN) that extracts features from the frames and feeds them to an LSTM neural network. To address privacy concerns, edge computing is essential for models working with RGB videos to avoid video transmission. In designing our third method, we paid specific attention to this aspect by keeping the computational complexity of the model so low that it can be implemented on cameras built-in boards without losing its real-time performance. Spatial features extracted from the frames by a lightweight, yet iv powerful 2D CNN are sequentially processed by a transformer. Evaluations prove the competing performance of our models with regards to the state-of-the-art methods. We also present an embedded system prototype we have built to verify the capability of our RGB models to perform in real time on limited computational resource. Finally, we introduce the UBC-DML Fall Dataset, a large comprehensive video dataset we have captured to address the shortcomings in the current public datasets for fall detection. v Lay Summary Falling is a common cause of domestic injuries with potentially fatal consequences. The early detection of fall incidents can mitigate subsequent injuries and traumas and reduce casualties. Hence, an automatic fall detection system is essential for remote health monitoring in residences to promote wellness and support independent living. Video cameras provide an affordable and efficient solution for fall detection. However, the performance of most existing models is limited to particular conditions, while some of them provoke privacy concerns. To solve these issues, in this thesis, we propose three novel methods using deep learning that can generalize well to different conditions. We address the privacy concerns by either using depth cameras or performing the processing on the camera. We also present a system prototype we have built and introduce a large video dataset we have captured for human fall detection to address the need for a comprehensive and representative public dataset. vi Preface This thesis presents research conducted by Anahita Shojaei Hashemi under the guidance of Dr. Panos Nasiopoulos and Dr. James J. Little. A list of publications resulting from the work presented in this thesis is provided below. Versions of Chapter 2 appear in the published conference paper [P3] and the submitted journal paper [P2]. The work presented in this chapter was performed by Anahita Shojaei Hashemi. It includes performing the literature review, designing and implementing the proposed method, conducting evaluations, Analyzing the results, and writing the manuscripts. Dr. Panos Nasiopoulos and Dr. James J. Little provided technical guidance and editorial input into the manuscript writing, and Dr. Mahsa T. Pourazad advised on the practical aspects of the work. The research findings from Chapter 3 have been published in the conference paper [P5] and was primarily accomplished by Anahita Shojaei Hashemi. She performed the literature review, designed and implemented the proposed method, conducted evaluations, analyzed the results, and prepared the initial manuscript. Dr. Hamid Reza Tohidypour wrote the final manuscript, and Dr. Panos Nasiopoulos provided technical guidance and editorial input into the manuscript writing. Dr. Mahsa T. Pourazad advised on the practical aspects of the work. Chapter 4 contains work presented in the submitted journal paper [P1]. The work was mainly carried out by Anahita Shojaei Hashemi, including the literature review, designing and implementing the proposed method, performing evaluations, analyzing the results and writing the manuscript. Dr. Hamid Reza Tohidypour helped with the implementation and evaluation of the proposed method. Dr. Panos Nasiopoulos and Dr. James J. Little provided technical guidance and vii editorial input into the manuscript writing. Dr. Mahsa T. Pourazad advised on the practical aspects of the work. The content of Chapter 6 appears in the submitted journal paper [P1], the workshop presentation [P4], and the published conference paper [P6]. Anahita Shojaei Hashemi led the data capturing project. She designed the data collection procedure, the video scenarios, and the arrangement of the scenes. She was in charge of capturing the videos and received assistance of Michelle Mak, Jennifer Grace, Doojin Jung, and Taran Bains. Dr. Mahsa T. Pourazad and Dr. Panos Nasiopoulos provided technical guidance throughout the project, as well as editorial input into writing the manuscripts. Dr. Maryam Azimi helped with writing the manuscript of [P6]. The data capturing in this work was covered under the UBC Ethics Board (H12-00308). List of Publications Based on Work Presented in this Thesis [P1] A. Shojaei-Hashemi, H. R. Tohidypour, P. Nasiopoulos, J.J. Little, and M.T. Pourazad, “A Transformer Convolutional Neural Network for Real-Time Video-based Human Fall Detection”, submitted [P2] A. Shojaei-Hashemi, P. Nasiopoulos, J.J. Little, and M.T. Pourazad, “Video-based Human Fall Detection Using Long Short-Term Memory Neural Network on the 3D Location of Body Joints”, submitted [P3] A. Shojaei-Hashemi, P. Nasiopoulos, J.J. Little, and M.T. Pourazad, “Video-based Human Fall Detection in Smart Homes Using Deep Learning,” in Proceedings of 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-5, IEEE, 2018 viii [P4] A. Shojaei-Hashemi, P. Nasiopoulos, and M.T. Pourazad, “A New Large-scale Video Dataset for Human Fall Detection,” in NeurIPS 2020 Workshop on Dataset Curation and Security, December 2020 [P5] H. R. Tohidypour, A. Shojaei-Hashemi, P. Nasiopoulos, and M.T. Pourazad, “A Deep Learning Based Human Fall Detection Solution”, in Proceedings of the 15th PErvasive Technologies Related to Assistive Environments Conference (PETRA), pp. 89-92, ACM, 2022 [P6] M.T. Pourazad, A. Shojaei-Hashemi, P. Nasiopoulos, M. Azimi, M. Mak, J. Grace, D. Jung, and T. Bains, “A Non-Intrusive Deep Learning Based Fall Detection Scheme Using Video Cameras,” in Proceedings of 2020 International Conference on Information Networking (ICOIN), pp. 443-446, IEEE, 2020 ix Table of Contents Abstract ......................................................................................................................................... iii Lay Summary .................................................................................................................................v Preface ........................................................................................................................................... vi Table of Contents ......................................................................................................................... ix List of Tables .............................................................................................................................. xiii List of Figures ...............................................................................................................................xv List of Abbreviations ................................................................................................................. xvi Acknowledgements .................................................................................................................... xix Dedication .....................................................................................................................................xx Chapter 1: Introduction ................................................................................................................1 1.1 Literature Review............................................................................................................ 4 1.1.1 Classic Machine Learning-based Methods ................................................................. 4 1.1.2 Deep Learning-based Methods ................................................................................... 7 1.2 Thesis Outline ................................................................................................................. 9 Chapter 2: A Long Short-Term Memory (LSTM) Neural Network for Fall Detection Using the 3D Location of Body Joints ...................................................................................................12 2.1 Proposed Method .......................................................................................................... 12 2.1.1 Input of the Model .................................................................................................... 13 2.1.2 Model Architecture ................................................................................................... 15 2.2 Experiments .................................................................................................................. 20 2.2.1 Datasets ..................................................................................................................... 20 x 2.2.2 Training Procedure.................................................................................................... 21 2.3 Performance Evaluation and Results ............................................................................ 22 2.4 Conclusions ................................................................................................................... 28 Chapter 3: A Long-Term Recurrent Convolutional Network (LRCN) for Fall Detection Using RGB Videos........................................................................................................................30 3.1 Proposed Method .......................................................................................................... 30 3.1.1 Two-dimensional Convolutional Neural Network (2D CNN) Module .................... 32 3.1.2 Long-Term Recurrent Convolutional Network (LRCN) Model ............................... 34 3.2 Performance Evaluation and Results ............................................................................ 34 3.3 Conclusions ................................................................................................................... 38 Chapter 4: A Transformer-Based Convolutional Neural Network for Real-Time Fall Detection Using RGB Videos ......................................................................................................40 4.1 Proposed Method .......................................................................................................... 40 4.1.1 Two-dimensional Convolutional Neural Network (2D CNN) Module .................... 42 4.1.2 Transformer Module ................................................................................................. 45 4.1.3 Transformer Convolutional Neural Network (TrCNN) Model ................................ 48 4.2 Performance Evaluation and Discussion ...................................................................... 49 4.3 Conclusions ................................................................................................................... 54 Chapter 5: An Embedded System Prototype Development for Real-Time Fall Detection via RGB Camera ................................................................................................................................56 5.1 Hardware Options ......................................................................................................... 56 5.1.1 NVIDIA Jetson Series Developer Kits ..................................................................... 56 5.1.2 Amazon Web Services (AWS) DeepLens ................................................................ 58 xi 5.1.3 Raspberry Pi 4 B combined with Intel Neural Compute Stick 2 or Google’s Coral USB Accelerator ................................................................................................................... 58 5.2 Prototype Design ........................................................................................................... 60 5.2.1 NVIDIA Jetson Nano Developer Kit Setup .............................................................. 61 5.2.2 Hardware and Software Optimization ...................................................................... 62 5.3 Live Tests and Results .................................................................................................. 64 5.4 Conclusions ................................................................................................................... 65 Chapter 6: UBC-DML Fall Dataset ...........................................................................................66 6.1 Existing Public Video Datasets on Human Fall ............................................................ 67 6.1.1 Le2i Fall Detection Dataset (FDD) ........................................................................... 67 6.1.2 UR Fall Detection (URFD) Dataset .......................................................................... 67 6.1.3 Multiple Cameras (Multicam) Fall Dataset .............................................................. 67 6.1.4 ACT42 Dataset .......................................................................................................... 68 6.1.5 TST Fall Detection Dataset v2 .................................................................................. 68 6.1.6 NTU RGB+D Action Recognition Dataset .............................................................. 68 6.1.7 SDUFall Dataset ....................................................................................................... 69 6.1.8 Shortcomings of Existing Datasets ........................................................................... 69 6.2 Capturing Procedure ..................................................................................................... 70 6.2.1 First Round of Capturing .......................................................................................... 72 6.2.2 Second Round of Capturing ...................................................................................... 73 6.2.3 Third Round of Capturing ......................................................................................... 77 6.3 Conclusions ................................................................................................................... 77 Chapter 7: Conclusions ...............................................................................................................79 xii 7.1 Summary of the Contributions ...................................................................................... 79 7.2 Significance and Potential Applications of the Research ............................................. 81 7.3 Future Work .................................................................................................................. 82 Bibliography .................................................................................................................................85 xiii List of Tables Table 2.1 Performance of our model at optimal hyperparameters (trained and tested on NTU RGB+D Dataset) ........................................................................................................................... 24 Table 2.2 Performance of our model compared to the state-of-the-art (trained and tested on NTU RGB+D Dataset) ........................................................................................................................... 25 Table 2.3 Performance of our model when trained on NTU RGB+D Dataset and tested on SDUFall Dataset ........................................................................................................................... 26 Table 2.4 Performance of our model compared to the state-of-the-art (trained on NTU RGB+D Dataset and tested on SDU Dataset without fine-tuning) ............................................................. 27 Table 2.5 Performance of our model (N1=60, N2=60) in live test ................................................ 28 Table 3.1 Validation results of different CNNs on the auxiliary action classification task .......... 33 Table 3.2 UBC-DML Fall Dataset labels and their corresponding number of samples used for action classification and fall detection tasks ................................................................................. 36 Table 3.3 Performance of our proposed model on the validation subset ...................................... 37 Table 3.4 Performance of the LRCNs with different classification layers on the test subset ....... 38 Table 4.1 Validation results of different CNNs for the auxiliary action classification task ......... 43 Table 4.2 Validation results of the reduced features for the auxiliary action classification task . 48 Table 4.3 TrCNN validation results for different hyperparameters values .................................. 52 Table 4.4 Number of parameters for the transformer module and the overall TrCNN model ..... 53 Table 4.5 Comparison of TrCNN and LRCN performance on test subset ................................... 54 Table 5.1 Specifications of NVIDIA Jetson series developer kits................................................ 57 Table 5.2 Specifications of AWS DeepLens ................................................................................ 59 xiv Table 5.3 Specifications of Raspberry Pi 4 B ............................................................................... 60 Table 5.4 Specifications of Raspberry Pi Camera V2 .................................................................. 61 Table 6.1 Frame labels for the first round of video capturing ...................................................... 72 Table 6.2 Number of videos in each category for the first round of video capturing ................... 73 Table 6.3 Cameras arrangement in the second round of video capturing ..................................... 76 xv List of Figures Figure 2.1 Overall structure of our proposed model ..................................................................... 13 Figure 2.2 25 major body joints used by our network [50] .......................................................... 14 Figure 2.3 Structure of a general RNN. (a) Compact illustration. (b) Unfolded illustration. (c) Hidden layer l at time t. (d) Structure of the same hidden layer, consisting of Nl hidden units. .. 16 Figure 2.4 Structure of an LSTM cell ........................................................................................... 17 Figure 2.5 Setup for live-testing our proposed model .................................................................. 28 Figure 3.1 Overview of our proposed LRCN model .................................................................... 31 Figure 3.2 Architecture of our custom CNN ................................................................................ 32 Figure 4.1 Overview of our proposed model (TrCNN) ................................................................ 41 Figure 4.2 DenseNet121. (a) Network architecture. (b) A 5-layer dense block [57]. .................. 44 Figure 4.3 Structure of the encoder in a vanilla transformer ........................................................ 46 Figure 5.1 Hardware components of our prototype ...................................................................... 61 Figure 5.2 Our prototype. (a) Temporary 3D-printed mount. (b) Metal box and mount.............. 64 Figure 6.1 Camera setup for capturing the UBC-DML Fall Dataset. (a) Round 1 setup. (b) Round 2 setup. (c) Round 3 setup............................................................................................................. 71 Figure 6.2 Sample frames of the first group of labels in round 1 video capturing. (a) Falling. (b) Fallen. (c) Recovering. (d) Recovered to sitting. (e) Recovered to standing. ............................... 74 Figure 6.3 Sample frames of the second group of labels in round 1 video capturing. (a) Upright. (b) Sitting. (c) Lying down on the floor. (d) Sitting on the floor. (e), (f) Other. .......................... 75 Figure 6.4 Sample frames of the round 2 video capturing. (a) Standing. (b) Falling. .................. 76 Figure 6.5 Sample frames of the round 3 video capturing. (a) Falling. (b) Standing. .................. 77 xvi List of Abbreviations 2D Two-Dimensional 3D Three-Dimensional ADL Activities of Daily Life AI Artificial Intelligence API Application Programming Interface ASIC Application-specific Integrated Circuit AUC Area Under the Curve AWS Amazon Web Services CAD Canadian Dollar CNN Convolutional Neural Network CPU Central Processing Unit CSI Camera System Interface DSTCAE Deep Spatiotemporal Convolutional Autoencoder DML Digital Multimedia Lab DI Dynamic Image FDD Fall Detection Dataset FPR False Positive Rate fps Frames Per Second GELU Gaussian Error Linear Unit GB Giga Byte xvii GHz Giga Hertz GPU Graphics Processing Unit HOG Histogram of Oriented Gradient HOOF Histogram of Oriented Optical Flow IR Infrared IoT Internet of Things KNN K Nearest Neighbor LPDDR Low-Power Double Data Rate LSTM Long Short-Term Memory LRCN Long-Term Recurrent Convolutional Network MHI Motion History Image MIPI Mobile Industry Processor Interface MLP Multilayer Perceptron MP Megapixel OS Operating System RAM Random Access Memory ROC Receiver Operating Characteristic RNN Recurrent Neural Network SBC Single-Board Computer SD Secure Digital SI Silhouette Image SDK Software Developer Kit xviii ST-GCN Spatial Temporal Graph Convolutional Network SVM Support Vector Machine UBC University of British Columbia TrCNN Transformer Convolutional Neural Network TPR True Positive Rate URFD University of Rzeszów Fall Detection USB Universal Serial Bus xix Acknowledgements The work presented in this thesis was made possible through the assistance and support of several individuals. I wish to take this opportunity to express my appreciation to all those who helped me in my academic journey. Firstly, I express my utmost gratitude to my supervisor, Dr. Panos Nasiopoulos, and my co-supervisor, Dr. James J. Little, for their enduring guidance and support throughout my Ph.D. program. They have played a crucial role in shaping me as a researcher through their patient mentorship and insightful advice. I would also like to extend my heartfelt thanks to Dr. Mahsa T. Pourazad for her advice and assistance during my studies. I was fortunate to benefit from her technical expertise and professional experience. Furthermore, I am grateful to my Ph.D. committee members, Dr. Shahriar Mirabbasi and Dr. Victor Leung, for their time and effort in guiding me throughout my graduate studies. I must also acknowledge the collaborative and supportive research environment cultivated by my colleagues at the UBC Digital Multimedia Lab that helped me grow as a researcher and for which I am immensely grateful. Additionally, I would like to express my profound gratitude to my caring friends, who provided unwavering support that sustained me through the inevitable challenges of this academic journey. Finally, I express my deepest appreciation to my family for their constant love and support throughout my life, without which this academic achievement would not have been possible. xx Dedication To my beloved parents in recognition of their unwavering love and countless sacrifices 1 Chapter 1: Introduction Remote health and wellness monitoring is quite a new technology in the field of public healthcare, with research focused on developing solutions that can be easily incorporated into existing homes as well as future smart homes. Enabled by the internet of things (IoT), it aims at improving the wellness and comfort of the occupants, while reducing the imposed cost on healthcare systems [1]. One application of such integrated health monitoring system is detecting critical health situations and sending automatic notifications to family, friends, or nearby contacts, and if necessary, to the emergency medical services. In this way, immediate help or medical response can be provided when time is of vital importance, which results in reduced morbidity and mortality rates. Falling is a common domestic incident. Although anyone may experience a fall incident, older adults, patients discharged from the hospital, and people with chronic diseases are more prone to it, and it makes a barrier against independent living for them. Falling may be caused by or result in critical health conditions such as heart attack, stroke, epilepsy attack, pelvic fraction, and concussion [2]. In such cases, a delay in discovering the incident can result in serious health complications and even death. Even when no injury is occurred, half of older adults who fall are not able to get up without assistance, which can lead to dehydration, muscle damage, and fear of future falls [3]. Therefore, early detection of fall incidents is the key to preventing fatality and reducing physical and mental impacts. This is where the value of an automatic fall detection system resides [4]. Hence, an automatic fall detection system can significantly contribute to the wellbeing of residents, support independent living and aging in place, and facilitate healthcare at home, making it beneficial to people as well as the healthcare systems and governments. 2 Fall detection systems are categorized into wearable devices, ambient sensors, and video cameras, each having their own advantages and disadvantages [5]–[7]. Wearable devices use accelerometers, gyroscopes, and magnetic sensors to constantly measure the changes in the height, orientation, and velocity of the person wearing them [5],[8]. Although they are the cheapest option and do not need installation, they should be worn the whole time that makes them intrusive and prone to being forgotten [9]. Moreover, they cannot capture the details of complex motions and interactions with the environment. Ambient sensors, on the other hand, are not intrusive, but they have low signal-to-noise ratio and they cannot pick up details. Furthermore, their limited coverage necessitates the use of several sensors for the sufficient coverage of space, making this solution less affordable. Unlike wearable devices and ambient sensors, video cameras can produce very rich content and provide good coverage at an affordable cost. In addition to increasing the accuracy of fall detection, the rich video content can be used for person identification, upon the occupants’ consent, to provide more personalized services. However, one disadvantage is that video cameras may raise privacy concerns. Using edge computing to implement the detection algorithm on the camera, or employing depth cameras instead of regular RGB cameras address this issue to some extent, making video cameras a popular solution for fall detection [10]. Plenty of algorithms exist for video-based fall detection, most of which exert classic machine learning techniques. In this approach, background knowledge is used to engineer video traits that can expectedly discriminate between videos with and without fall incidents and are referred to as handcrafted features. Majority of such fall detection methods are based on the standard definition of falling, i.e. going quickly from a standing to a lying position. They detect falls by tracking head, monitoring body pose, analyzing the shape of body silhouette, or examining the transformation of 3 body bounding box [11]–[18]. Others rely on the assumption that falling is succeeded by lying on the ground with little or no motion [19]–[22]. There are a few disadvantages for handcrafted features. Most importantly, they are based on particular definitions and assumptions and do not generalize well. For instance, the pose change from standing to lying may not occur at a fast pace, such as when the person experiences a heart attack, or the fallen person may not end up lying still on the ground. Consequently, the performance of the method is guaranteed only in situations where those definitions and assumptions hold, which constrains the application of the system and highly deteriorates its performance in practice. Moreover, handcrafted are not necessarily the most discriminative traits for the intended task, i.e., spotting fall events. Finally, they are problem-specific, and the slightest modification to the task requires redesigning the features. Hence, classic machine learning approach in video-based human fall detection is not good enough for real-life use. Considering the capabilities of deep learning compared to those of classic machine learning and to address the above shortcomings, we were among the first research groups to employ deep learning for video-based human fall detection. Using an abundant amount of data, our deep learning-based methods reckon the most discriminative traits of the data for the assigned task, i.e., fall detection, without relying on definitions or assumptions. The learnt features are not confined to specific conditions. However, for scenarios not included in the training data or applications moderately different from the original one, fine tuning the models is needed to maintain their performance. The generalization capacity of features learnt through deep learning depends on the size and diversity of the training data. Larger and more diverse data results in more comprehensive features and higher performance in practice. Existing public video datasets for human fall detection 4 that are large enough for deep learning have a few deficiencies which degenerate the system performance in practice. They lack diversity in fall scenarios and consist of very short videos. To address these issues, we have captured and made publicly available a comprehensive and representative video dataset for human fall detection. In this thesis, we propose three novel deep learning solutions for privacy-preserving video-based human fall detection. Chapter 2 explains the first model that uses depth cameras to protect the occupants’ identities in addition to their privacy. It takes the locations of key body joints extracted from depth videos to distinguish videos containing fall events. In Chapter 3, we explicate the second model operating with RGB cameras for applications where the identity of the fallen person is needed to provide better help. Chapter 4 goes over our third model, which is an improvement on the second one. It has substantially lower computational complexity to better support the real-time performance of the model on cameras built-in processors for privacy concerns. Chapter 5 describes the embedded system prototype we have made to assess the performance of our second and third models on boards with limited computational resources. Chapter 6 elaborates on our video dataset and its capturing procedure. Section 1.1 of this chapter provide a literature review of video-based human fall detection. Classic machine learning and deep learning approaches are covered in Sub-Sections 1.1.1 and 1.1.2, respectively. Section 1.2 presents a summary of the scientific contributions of this thesis. 1.1 Literature Review 1.1.1 Classic Machine Learning-based Methods A lot of research has been done on classic machine learning approach for video-based fall detection, out of which numerous models have been proposed [5],[10]. In this section, we review 5 the performance of some recent ones and go over the handcrafted features, classification algorithms, and input data they use. Yun and Gu’s method uses RGB video and is based on the assumption that the shape and motion of human body change rapidly when a person is falling [23]. Assuming the bounding box tracking the person throughout the video is available, they extract velocity statistics of dynamic shape points and motion points moving on two Riemannian manifolds as features and apply support vector machine (SVM) on them. The shape points and motion points are based on the histogram of oriented gradient (HOG) and the histogram of oriented optical flow (HOOF), respectively. They have obtained the average true positive rate (TPR) of 99.38% and false positive rate (FPR) of 1.84% on three datasets, including Multiple Cameras (Multicam) Fall Dataset [24], UR Fall Detection (URFD) Dataset [25], and ACT42 Dataset [26]. Yang and Lin use RGB and depth videos together to identify fall events [27]. Foreground objects are segmented according to the depth, and humans are detected using the skin color. A fall incident is announced if the vertical displacement of body centroid and the change in the height of body silhouette and tilt angle of body are higher than a presser threshold in less than a pre-determined number of frames. The method has achieved precision of 94.31% and recall of 85.57% on their captured small dataset. Stone and Skubic’s method operating on depth video is composed of two main stages [18]. In the first stage, after three-dimensional (3D) object segmentation and tracking, the vertical state of each 3D object is calculated for individual frames. On-ground events for each object are identified through temporal segmentation of the vertical state time series. In the second stage, features are extracted from the on-ground periods, using the vertical state and its derivatives. An ensemble of decision trees is applied on the features extracted for each on-ground event to compute the confidence that a fall preceded it. They have evaluated the performance of their system on nine years of continuous data collected in 13 apartments of older adults. Marzahl et al.’s 6 system [28] requires a manual pre-segmentation after installation to identify areas of the frame allowing and not allowing a fall. The pre-segmentation has to be updated every time the room arrangement is changed. Their proposed method segments human objects on depth images and applies rule-based classification through a decision tree on them to detect falls. The rules are defined on the human object contour and its relative location to the pre-segmented areas. The system has been evaluated on a small dataset collected in a laboratory environment and has shown 93% recall. Using depth video, Mastorakis and Markis [16] extract the 3D bounding box of the person by OpenNI framework [29] and exercise rule-based classification on the dimensions of the bounding box and their first derivatives to detect fall incidents. Evaluating on a small dataset captured in a laboratory environment, their method has detected all falls without any false alarm. Bian et al.’s algorithm [12] extracts key body joints from depth video and calculates the distance of the head from the floor in each frame. The head distances for T consecutive frames comprise the input vector to an SVM classifier. Evaluation on a small dataset collected in a laboratory environment has shown 95.3% sensitivity, 100% specificity, and 97.6% accuracy. Rougier et al. employ depth images for fall detection [30]. The procedure consists of detecting the ground plane of the room, segmenting persons from background, localizing and tracking the 3D centroid of the body, and thresholding the distance of the centroid from the ground. In case of an occlusion. i.e. a sudden disappearance of the body silhouette, the 3D velocity of the body centroid just before the occlusion is thresholded. Out of 25 fall and 54 non-fall videos captured in a laboratory environment, one fall has not been detected and no false alarm has been made. Planinc and Kampel [31] use body joints extracted from depth video to obtain body’s major axis. They apply thresholds on the orientation of this axis with regards to and its distance from the ground to recognize fall events. They have evaluated this method on a small dataset collected in a laboratory environment 7 and reported 92.5% recall, 100% precision, and 95.8% accuracy. Ramirez et. al. [32] apply an off-the-shelf algorithm on RGB videos to extract the 3D locations of body joints, and process the obtained vectors of locations using one of the SVM, K nearest neighbor (KNN), random forest, or multilayer perceptron (MLP). 1.1.2 Deep Learning-based Methods After deep learning became the mainstream in computer vision, video-based fall detection methods using deep learning emerged in 2018 and have substantially grown since then. We review the performance of a few popular and some recent ones in this section and brief their model architectures and input data. Nunez et al. [33] apply a two-dimensional (2D) convolutional neural network (CNN) on the optical flow of 20-frame video segments to detect the segments containing falls. Their model has achieved accuracies of 95%, 97%, and 97%, sensitivities of 100%, 99%, and 99%, and specificities of 92%, 96%, and 97% on Multiple Cameras Fall Dataset, URFD Dataset, and Le2i Fall Detection Dataset (FDD) [34], respectively. DeepFall, proposed by Nogas et. Al. [35], is composed of a 3D convolutional autoencoder, named deep spatiotemporal convolutional autoencoder (DSTCAE) that takes overlapping windows of consecutive frames and reconstructs the frames. Falling is considered as anomaly and is detected if the reconstruction error is higher than a threshold determined through validation process. The area under the curve (AUC) for the receiver operating characteristic (ROC) curve has been reported as 97%, 89%, and 95% for the Thermal Simulated Fall Dataset [36], URFD Dataset, and SDUFall Dataset [37], respectively. In Ge et al.’s method [38], SVM takes the outputs of a two-stream 2D CNN to spot fall videos. One stream consists of the key frames and the other is composed of their corresponding optical flows. The outputs of the CNN are sparsely encoded and merged before going to the SVM. 8 Evaluations on Multicam Fall Dataset and URFD Dataset have demonstrated 83.7% and 100% TPR and 2.57% and 5.5% FPR, respectively. Liu et al. [39] apply spatial temporal graph convolutional network (ST-GCN) on 2D coordinates of body joints for videos of 150 frames. The model proposed by Rahnemoonfar et al. [40] consists of a 3D CNN that takes videos of 120 frames and outputs the predicted class. Chen et al. [41] have designed a dilated 1D CNN with residual skips. They take a video of 300 frames, concatenate the 3D coordinates of body joints for each frame, and stack the obtained vectors to construct the multi-channel 1D input of the CNN. Tsai et al. [42] also employ the concatenated and stacked 3D coordinates of body joints. Their model is composed of a 2D CNN, a customized version of AlexNet [43], which takes 30 frames as input. In order to reduce the number of the CNN parameters, they use only seven body joints, which they believe are the most distinctive ones for the purpose of fall detection. Lu et al. [44] have proposed a 3D CNN to process the video in segments of 16 frames. They have incorporated a long short-term memory (LSTM) neural network into their model for spatial attention. In Kong et al.’s three-stream convolutional model [45], video segments of eight frames are compressed into motion history images (MHIs) and dynamic images (DIs) to form two streams, and silhouette images (SIs) comprise the third stream. The streams are sampled with random strides of one to eight, and windows of five frames are taken in by three 3D CNNs that are built upon ResNet18 [46]. Similarly, Ding et al. [47] compact video segments of 16 frames into DIs and feed the DIs to a shallow 2D CNN, ResNet18. Feng et al. have addressed a little different problem, fall detection in a crowded scene, by employing YOLOv3 [48] to extract bounding boxes and deep sort tracking to obtain motion trajectories of different people. The 9 trajectories are given to a VGG16 [49] neural network and an attention mechanism as applied on the extracted features by an LSTM neural network for classification. 1.2 Thesis Outline In this thesis, we propose three novel deep learning-based methods for indoor human fall detection that use video cameras, while preserving the residents’ privacy. We also present the hardware prototype with limited computational power that we have designed to test our models for implementation on cameras built-in boards. Additionally, we introduce a large comprehensive video dataset we have captured to be used in developing video-based human fall detection solutions, especially when abundant data is needed. In Chapter 2, we present our first proposed model that is one of the earliest video-based fall detection methods to exert deep learning and the first deep learning-based one to employ depth cameras and require solely the 3D locations of major body joints. Working with depth cameras and relying only on the body joints, which are extracted on the fly by cameras SDKs or off-the-shelf software in real time, our model provides high privacy preservation and conserves occupants’ identities. It is composed of an LSTM neural network, which was the state-of-the-art at the time in processing sequential data, to analyze the joints locations throughout the video for spotting fall incidents. In some applications, knowing the identity of the fallen person results in providing better service. For instance, the response action can be taken based on the medical profile of the person, and more personalized medical care can be provided. In such applications, RGB cameras are preferred, so that the occupants can be identified with high accuracy. Therefore, in Chapter 3, we propose another LSTM-based model that works with RGB video. Using a 2D CNN, a vector representation 10 composed of spatial features is generated for each RGB frame. The LSTM network processes these vectors in sequence to detect any fall incident. Because of privacy concerns associated with RGB videos, edge computing is crucial for fall detection models that use RGB cameras, with the model implemented on the camera's built-in board. This approach eliminates video transmission, which may introduce privacy and security issues during video processing for fall detection. However, the limited computational resources of these boards necessitate that the model is lightweight to ensure real-time performance. Meeting this criterion is challenging for deep learning, and existing deep learning-based fall detection methods using RGB videos do not fulfill it. While the model we propose in Chapter 3 has been shown to perform in real-time on these boards, it only marginally satisfies the criterion. To improve this, we propose a deep neural network in Chapter 4 that is composed of a DenseNet121 and a transformer. DenseNet121 is a lightweight, yet high-performing 2D CNN, and transformer is the state-of-the-art deep learning architecture for sequence analysis. Using a lightweight CNN and applying feature reduction on its output result in a low-complexity model that can reliably perform in real-time on the built-in boards of RGB video cameras. To assess the performance of our proposed RGB video-based models on cameras built-in boards, we have designed and built a prototype of an RGB camera with limited built-in computational resources. It is composed of a Raspberry Pi camera and an NVIDIA single-board computer. We present this prototype in Chapter 5, including the details of the designing procedure and the live tests conducted on the proposed models. In Chapter 6, we introduce a large video dataset, named UBC-DML Fall Dataset, that we have captured to be publicly used for research on human fall detection, addressing the shortcomings in 11 the current public datasets. The performance of models, especially the ones based on deep learning, in practice depends considerably on the breadth of the data they are trained on. Moreover, a lot of model restrictions and weaknesses are identified through evaluations on the test data, and more comprehensive data supports more thorough and more efficient testing. To have a comprehensive and representative dataset for human fall detection, we have included various falling scenarios and occluded fall events in the UBC-DML Fall Dataset, which are missing from the existing large public datasets. It also consists of longer videos compared to those datasets. The capturing procedure and the detailed specifications of our dataset are explicated in the chapter. 12 Chapter 2: A Long Short-Term Memory (LSTM) Neural Network for Fall Detection Using the 3D Location of Body Joints Since the adoption of deep learning in the field of computer vision, it has rapidly advanced and gained popularity in video applications, including video-based human fall detection. However, most of the current deep learning-based methods for video-based fall detection rely on neural networks that are not intrinsically sequential. Therefore, they can capture long-term dependencies among video frames only at the cost of having a very large number of parameters. Considering the limited amount of video data available on falling, the great number of parameters in these models restricts their generalization capability to any dataset other than the one used for training. To address this problem, in this chapter we have proposed a model consisting of a long short-term memory (LSTM) neural network. To accommodate for videos of various lengths, we have used dynamic-length LSTM rather than the common regular LSTM. The input of our end-to-end model is the 3D coordinates of major body joints, extracted by the depth camera on-the-fly. As a result, the number of parameters is not huge. Thus, our network is successfully trained on a limited number of staged fall videos and yet, generalizes well to real-life falls. To accommodate for videos of various lengths, we have used dynamic-length LSTM rather than the common regular LSTM. The rest of this chapter is structured as follows. The proposed method is explained in Section 2.1. Section 2.2 contains the details of the conducted experiments, and the results are discussed in Section 2.3. Section 2.4 concludes the chapter. 2.1 Proposed Method Our proposed model is composed of an LSTM neural network, a deep learning model designed specifically for sequential data. Figure 2.1 shows the overall structure of the model. At each time 13 step, the network takes in the 3D coordinates of the body joints extracted from the frame and merges them with the processed information from previous frames. After getting the last frame, the information aggregated from all the frames goes to a fully-connected layer to make the decision on whether a fall incident is occurring in the video or not. The following sub-sections explain the model, including the input and the network architecture, in detail. 2.1.1 Input of the Model The 3D coordinates of the major body joints encompass the dynamics of the body. Hence, they contain rich information to detect falls. Although the whole image and the optical flow include more details that may help with detecting fall incidents, employing them as an input substantially increases the number of neural network parameters. This reduces the generalization capability of Figure 2.1 Overall structure of our proposed model 14 the network, considering the limited amount of fall data available for training, and consequently, deteriorates its performance in practical real-life scenarios. For these reasons, we decided to use the 3D coordinates of body joints as the input to our proposed neural network. The 3D coordinates of the joints can be extracted directly from depth maps, while RGB videos can provide only the 2D coordinates and the 3D coordinates need to be estimated. Depth modality is also better in preserving the privacy than RGB. Therefore, we opted for depth modality. Several off-the-shelf algorithms are available to extract body joints coordinates in real time. Also, the software developer kits (SDKs) of some cameras provide real-time joint extraction. We employ the 25 body joints depicted in Figure 2.2 [50], where their 3D coordinates are merged to form a vector of 75 elements. Figure 2.2 25 major body joints used by our network [50] 15 2.1.2 Model Architecture The sequential aspect of video, as a set of frames, is substantial to the illustration of the actions taking place in the video. Thus, to accurately detect fall incidents, both short and long-term temporal correlations among the frames should be captured. In most of the existing deep learning models for video-based fall detection, one of the following three approaches, or a combination of them, is employed to process the information carried by the time dimension of videos: optical flow, sliding window, and CNN. Optical flow encodes the temporal information of consecutive frames into 2D maps. Despite being rich in information, it is computationally expensive, which is a bottleneck for real-time applications. It also has a very short time scope, i.e., only two frames, unless it is used in conjunction with another method. In the sliding window approach, short overlapping segments of videos are processed separately and the outputs are aggregated to make the final decision. Although it has longer time scope than optical flow, it cannot capture long-term dependencies. The CNN approach processes temporal information by applying convolutional filters on the time dimension. The temporal receptive field increases layer by layer, so to capture longer-term dependencies, more layers are required, which means a greater number of parameters. Since videos available on real-life fall incidents are scarce, fall detection models are trained and tested on videos of controlled staged falling. To avoid a drastic drop in their performance in real-life applications, it is desirable to have high generalization capability with regards to input data. This is not achieved in models with a large number of parameters that are trained on a limited amount of data, which is the case for CNN-based fall detection models. To address the mentioned shortcomings, we have used Recurrent neural network (RNN) [51], a deep neural network specifically designed to handle sequential data. As shown by Figure 2.3.(a), 16 Figure 2.3 Structure of a general RNN. (a) Compact illustration. (b) Unfolded illustration. (c) Hidden layer l at time t. (d) Structure of the same hidden layer, consisting of Nl hidden units. Unfolding17 an RNN consists of a feed-forward neural network that works in a looped procedure. Each element of the input sequence is processed by the feed-forward neural network and the output of each layer is used in processing the next element at the same layer. This enables the information flow throughout the sequence, so that each element can be processed with regards to its preceding ones. For the better understanding, an RNN can be unfolded as illustrated in Figure 2.3.(b). All the feed-forward neural networks shown in this figure are identical, i.e., replicated only for demonstration purposes. Thus, they have the same structure and weights. The feed-forward network has L hidden layers of 𝑁𝑙 hidden units, for 𝑙 ∈ {1, 2, … , 𝐿}. The input to the hidden layer 𝑙 at the step 𝑡 is compromised of the output the previous layer, 𝒉𝑡𝑙 , and the output of the same layer for the (𝑡 − 1)𝑡ℎ input, i.e., 𝒉𝑡−1𝑙 . Although the processed information is carried forward throughout the sequence, the flow decays substantially in long sequences, because of a phenomenon called “vanishing gradient”. To solve this problem, modified versions of RNN have been developed, a popular one of which is LSTM neural network. In LSTM literature, each layer of the feed-forward neural network is called a Figure 2.4 Structure of an LSTM cell 18 “cell”. Figure 2.4 demonstrates the structure of the cell at the 𝑙𝑡ℎ layer of the feed-forward network at step 𝑡. Equation (2.1) explains the mechanism for obtaining 𝒉𝑡𝑙 , the output of the displayed cell, [𝒊𝒇𝒐𝒈] = [𝑠𝑖𝑔𝑚𝑠𝑖𝑔𝑚𝑠𝑖𝑔𝑚𝑡𝑎𝑛ℎ] (𝑊𝑙 [𝒉𝑡𝑙−1𝒉𝑡−1𝑙 ]) 𝒄𝑡𝑙 = 𝒇 ∙ 𝒄𝑡−1𝑙 + 𝒊 ∙ 𝒈 𝒉𝑡𝑙 = 𝑜 ∙ 𝑡𝑎𝑛ℎ(𝒄𝑡𝑙 ) (2.1) where 𝑖, 𝑓, 𝑜, and 𝑔 are called the input, forget, output and control gates, respectively. 𝒄𝑡𝑙 , which is referred to as the cell state, represents the information flow throughout the blocks. A portion of the cell state determined by the output gate forms the output of the cell. The forget gate filters a portion of the cell state to be carried over to the next block, while the input and control gates select a portion of the cell input to be added to the cell state. Like RNN, all the feed-forward networks are identical and share the same weights. That is, the weight matrices 𝑊𝑙, where 𝑙 ∈ {1, 2, … , 𝐿}, are independent of 𝑡. The number of the layers, 𝐿, and the number of the hidden units for each layer, i.e., 𝑁𝑙 for 𝑙 ∈ {1, 2, … , 𝐿}, are determined through experiments, such that the overall model performance is optimized. The details of this process are explained in the next section. It is common practice in training LSTM neural networks to use input sequences of equal length. The motivation is a simpler and faster implementation of the network when the training batches consist of the same length sequences. In cases like ours, where the sequences have different lengths, a specific length is used for all the training sequences, which is determined based on the statistical features derived from the lengths of the sequences, e.g., the average, minimum, or 19 maximum length. The sequences are adjusted by cropping the longer sequences and zero-padding the shorter ones. For inference, no length adjustment is required as long as the sequences are given to the network one by one rather than in batches. Nonetheless, the length adjustment is usually done at the inference, too, since the network has only been exposed to and optimized for sequences of a specific length and its performance may drop if the sequence has a different length. Zero-padding the training sequences introduces noise to the input data and cropping the sequences results in partial information loss. Therefore, the input adjustment technique adversely affects the performance of the neural network and prevents it from reaching its full capacity. To avoid this pitfall, we propose to keep the sequences intact and let the LSTM network be trained on sequences with various lengths. To achieve this, we modify the common implementation of LSTM network, so that the training batches can consist of sequences of different lengths. We call this a “dynamic-length” LSTM neural network. Sequences in the inference phase also remain intact, with no concern about performance drop, considering that the network is exposed to sequences of various lengths during the training phase. Depending on the application, the outputs of the feed-forward neural network either at all the steps or only at the last step are taken as the output of the whole LSTM network. In our proposed model, the output of the last step is considered as the LSTM output. It is fed to a fully connected layer to obtain the scores of the fall and non-fall classes. If the higher score belongs to the fall class, the input video will be declared to contain a fall incident. Applying Softmax on the two scores gives the predicted probabilities of the classes. 20 2.2 Experiments We have conducted extensive experiments to find the optimal structure and the best training practice for our proposed model and have evaluated its performance against the state-of-the-art video-based fall detection methods. To this end, we have selected four of the top recent deep-learning-based methods, proposed in [39]–[42], [21]–[24], and the latest LSTM-based method, proposed in [52]. Moreover, we have tested the generalization capability of our model, to scenarios with new setups, against that of the state of the art. Finally, we have assessed the performance of our model in real life by running live tests. 2.2.1 Datasets We have employed two datasets for our experiments, the NTU RGB+D Action Recognition Dataset and the SDUFall Dataset. They used to be the largest video datasets on human fall that were publicly available at the time, before we captured UBC-DML Fall Dataset. In these two datasets, human skeletons have been extracted on the fly using the SDK of the cameras capturing the videos, i.e., Microsoft Kinect. Each skeleton in the NTU RGB+D Dataset is composed of 25 major body joints depicted in Figure 2.2. The same body joints except for joint 3 and joints 22 to 25 comprise the skeletons in the SDUFall Dataset. As the input to our model requires 25 joints, we have replicated joints 2, 7, 8, 11, and 12 in the SDUFall skeleton data to compensate for the unavailable joints, considering their proximity. More details on the two datasets can be found in the Section 6.1 of Chapter 6. Since the NTU dataset is substantially bigger than the SDUFall dataset, all the experiments, except for the generalization assessment, have been performed on the NTU dataset. 21 2.2.2 Training Procedure The proposed model has been implemented with Pytorch v1.9. To keep the number of model parameters low, we started with two layers, i.e., L=2, which was then proved through our experiments to be sufficient. The number of the hidden units for each layer, i.e., 𝑁𝑙 for 𝑙 ∈ {1, 2}, has been determined through grid search on 𝑁𝑙 ∈ {1, … , 60}, while 𝑁2 ≤ 𝑁1. Five-fold cross-validation on the NTU RGB+D dataset has been used for hyperparameter optimization and for selecting the best training approach. The training portion of each fold has been further split into training and validation subsets at the ratio of 0.8 to 0.2, where the validation subset has been employed only for the purpose of early stopping. We have tried two setups for each of the training and test subsets. In the namely balanced setup, the original ratio of the ADL to fall samples has been kept unchanged, while the other setup, named imbalanced, benefits from an equal number of samples from each class. We have examined three combined scenarios, excluding the case where both subsets are balanced. The batch size and the maximum number of epochs have been set to 128 and 1000, respectively. Cross-entropy with Softmax has been employed as the loss function, and both regular and weighted loss have been tried. The weights have been inversely proportional to the number of each class samples. Adaptive Moment Estimation (Adam), with initial learning rate of 10-3, has been picked for optimization to provide an adaptive learning rate throughout the training. Early stopping has been triggered if the average loss on the validation subsample did not decrease for 100 consecutive iterations. To prevent overfitting, we have applied dropout with the probability of 0.5 on the LSTM layers. 22 In the next step of our experiments and to evaluate the generalization capability of our proposed model, we have trained the model on the NTU dataset and tested it on the SDUFall dataset All the samples in the NTU dataset has been used for training, so the dataset has been split into training and validation subsets at the ratio of 0.8 and 0.2, respectively. The testing on the SDUFall dataset has been performed twice, once without fine-tuning and once with fine-tuning on the 20% of the dataset. Considering the results from the previous step, only imbalanced data setup has been used for training. However, both balanced and imbalanced setups have been investigated for fine-tuning and test. The input has been normalized before being fed to the neural network. In the experiments for the optimization and evaluation of the model, all samples have been normalized to the average mean and standard deviation of the training subset. The same has been done in the training stage of the experiments for generalization assessment. As the training and test datasets are different in the second group of the experiments, and the statistics of the two datasets are not the same, the average mean and standard deviation for normalizing the test samples have been derived from the test data in these experiments. For this purpose, 10 random ADL samples from the test dataset have been picked in the imbalanced setting, while the ADL samples of the test dataset excluded from the test set have been employed for the balanced setting. 2.3 Performance Evaluation and Results The performance of our model has been measured by and optimized based on different metrics, including the Area Under the Curve (AUC) for both the Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves, accuracy, and precision and recall as a pair. F1-score indicates the performance of the model with regards to the precision and recall together. The training and 23 test setups demonstrate whether a balanced or imbalanced data subset has been used for, respectively, training and testing the model. Except for the AUCs, the other metrics depend on the classifier threshold. All metrics have been calculated for all possible threshold values. The optimal threshold value is the one that maximizes the corresponding metric. These optimal thresholds may vary for each metric and may also change for different training and test setups. In practice, the model operates at a fixed threshold, and thus, it may perform optimally regarding to only one metric. The desired metric can be selected according to the user’s preference or needs, i.e., acceptance levels of false alarm or undetected fall, which then, determines what threshold to use. Tables 2.1.(a), 2.1.(b), and 2.1.(c) contain the performance of the model with the optimal hyperparameters regarding accuracy, F1-score, and AUCs, respectively, for the various training and test data setups and loss settings. We observe that the model with 60 hidden units in both layers, i.e., 𝑁1 = 𝑁2 = 60, has the best performance with regards to all the four metrics. It has achieved its optimal performance regarding accuracy when trained and tested on imbalanced data with weighted training loss and unweighted validation loss. Its optimal performance with regards to F1-score and AUCs is obtained when trained on an imbalanced training set with weighted training loss and unweighted validation loss and tested on a balanced test set. The performance of our model compared to that of the state of the art is depicted in Table 2.2. The test sets used by the rival methods are either balanced or slightly imbalanced. To have a fair comparison, we have included, in this table, the performance of our model only on the balanced test data. For some methods, we did not have access to the original implementation of the models to replicate the results reported by the original papers. In these cases, we have taken the values from the original papers, which might lack some of the metrics. Thus, some values in the table are 24 Table 2.1 Performance of our model at optimal hyperparameters (trained and tested on NTU RGB+D Dataset) (a) with regards to accuracy Training Mode Test Mode Weighted Loss for Training Weighted Loss for Testing (N1, N2) Maximum Accuracy Precision Recall F1-score ROC_AUC PR_AUC imbalanced imbalanced   (60, 60) 99.863 0.9652 0.9663 0.9658 0.9955 0.9794 ✓  (60, 60) 99.872 0.9738 0.9618 0.9678 0.9982 0.9796 ✓ ✓ (60, 50) 99.851 0.9671 0.9584 0.9628 0.9957 0.9696 imbalanced balanced   (60, 40) 98.890 0.9977 0.9798 0.9887 0.9967 0.9977 ✓  (60, 60) 99.556 0.9977 0.9933 0.9955 0.9991 0.9994 ✓ ✓ (60, 40) 99.334 0.9966 0.9899 0.9932 0.9993 0.9994 balanced imbalanced  ✓ (60, 30) 99.076 0.7281 0.8607 0.7889 0.9897 0.7390 ✓ ✓ (60, 50) 99.056 0.7590 0.7753 0.7671 0.9935 0.7672 (b) with regards to F1-score Training Mode Test Mode Weighted Loss for Training Weighted Loss for Testing (N1, N2) Maximum F1-core Accuracy Precision Recall ROC_AUC PR_AUC imbalanced imbalanced   (60, 60) 0.9658 99.863 0.9652 0.9663 0.9955 0.9794 ✓  (60, 60) 0.9678 99.872 0.9738 0.9618 0.9982 0.9796 ✓ ✓ (60, 50) 0.9628 99.851 0.9671 0.9584 0.9957 0.9696 imbalanced balanced   (60, 40) 0.9887 98.890 0.9977 0.9798 0.9967 0.9977 ✓  (60, 60) 0.9955 99.556 0.9977 0.9933 0.9991 0.9994 ✓ ✓ (60, 40) 0.9932 99.334 0.9966 0.9899 0.9993 0.9994 balanced imbalanced  ✓ (60, 30) 0.7914 99.062 0.7147 0.8865 0.9897 0.7390 ✓ ✓ (60, 50) 0.7682 99.002 0.7189 0.8247 0.9935 0.7672 (c) with regards to AUCs of ROC and precision-recall curves Training Mode Test Mode Weighted Loss for Training Weighted Loss for Testing (N1, N2) Maximum ROC_AUC (N1, N2) Maximum PR_AUC imbalanced imbalanced   (60, 60) 0.9955 (60, 60) 0.9794 ✓  (60, 60) 0.9982 (60, 60) 0.9796 ✓ ✓ (60, 40) 0.9975 (60, 40) 0.9732 imbalanced balanced   (60, 40) 0.9967 (60, 40) 0.9977 ✓  (60, 60) 0.9991 (60, 60) 0.9994 ✓ ✓ (60, 40) 0.9993 (60, 40) 0.9994 balanced imbalanced  ✓ (40, 40) 0.9926 (40, 40) 0.7890 ✓ ✓ (60, 50) 0.9935 (60, 50) 0.7672 25 void. The results show that our proposed method outperforms the state of the art not only regarding precision and recall together, i.e., F1-score, but also with regards to each of them separately. Furthermore, it is the runner up in the accuracy with only 0.3% less accuracy than [41]. Table 2.3 summarizes the assessment conducted on the generalization of our model respecting dataset, i.e., when the test dataset considerably differs from the training dataset in several aspects. The model has been trained and tested on the NTU RGB+D and the SDUFall datasets, respectively. Following the approach taken for Table 2.2, we have reported the model performance only on the balanced test data. Tables 2.3(a) and 2.3(b) contain the hyperparameters and setups that result in the optimal model performance, when testing the model on the new dataset with and without fine-tuning, respectively. As expected, the performance of our model declines when the test data belongs to a drastically different dataset. With fine-tuning, the drops in the maximum accuracy, F1-score, precision, recall, area under the ROC curve, and area under the precision-recall curve have been about 8%, 7%, 9%, 6%, 3%, and 3%, respectively, which are not drastic, and the model has kept its high performance. Without fine-tuning, on the other hand, the model performance has deteriorated more substantially, though it has still remained at an acceptable level. The maximum Table 2.2 Performance of our model compared to the state-of-the-art (trained and tested on NTU RGB+D Dataset) Method Accuracy Precision Recall F1-Score #parameters ST-GCN [39] 80.78 -- -- 0.9745 2.6M 3D CNN [40] 97.58 -- -- -- 10M Dilated 1D CNN [41] 99.83 0.9747 0.9425 0.9583 4.3M Modified AlexNet [42] 99.20 0.9910 0.9890 0.9899 4M LSTM with 2D joints [52] 98.31 0.9829 0.9829 0.9829 82k Our Proposed Model (N1=60, N2=60) 99.56 0.9977 0.9933 0.9955 62K 26 accuracy, F1-score, precision, recall, area under the ROC curve, and area under the precision-recall curve have fallen about 13%, 11%, 19%, 3%, 11%, and 17%, respectively. Table 2.3(c) illustrates the performance of the best model from the previous stage, i.e., the one with 60 hidden units in both layers, when trained on the NTU dataset and tested on the SDUFall dataset. Comparing the Table 2.3 Performance of our model when trained on NTU RGB+D Dataset and tested on SDUFall Dataset (a) Performance at optimal hyperparameters with fine-tuning at testing Optimized (maximized) Metric Fine-tuning Mode Weighted Loss for Training Weighted Loss for Validation (N1, N2) Accuracy Precision Recall F1-Score ROC_AUC PR_AUC Accuracy balanced   (40, 40) 91.688 0.9033 0.9338 0.9183 0.9439 0.9221 F1-Score balanced   (40, 40) 91.688 0.9033 0.9338 0.9183 0.9439 0.9221 AUC of ROC Curve imbalanced ✓ ✓ (50, 20) N/A N/A N/A N/A 0.9623 0.9642 AUC of PR Curve imbalanced   (50, 40) N/A N/A N/A N/A 0.9599 0.9656 (b) Performance at optimal hyperparameters without fine-tuning at testing Optimized (maximized) Metric Weighted Loss for Training Weighted Loss for Validation (N1, N2) Accuracy Precision Recall F1-Score ROC_AUC PR_AUC Accuracy ✓  (50, 30) 86.600 0.8045 0.9670 0.8783 0.8782 0.8105 F1-Score ✓  (50, 30) 86.600 0.8045 0.9670 0.8783 0.8782 0.8105 AUC of ROC Curve   (50, 40) N/A N/A N/A N/A 0.8855 0.8209 AUC of PR Curve   (50, 40) N/A N/A N/A N/A 0.8855 0.8209 (c) Performance at N1 = N2 = 60 with and without fine-tuning Fine-Tuned Fine-tuning Mode Weighted Loss for Training Weighted Loss for Validation Maximum Accuracy Maximum F1-Score ROC_AUC PR_AUC Acc. Prec. Rec. F1-Score Acc. Prec. Rec. F1-Score ✓ imbalanced   88.688 0.9626 0.8050 0.8768 88.688 0.9626 0.8050 0.8768 0.9383 0.9497 ✓  90.125 0.9707 0.8275 0.8934 90.125 0.9707 0.8275 0.8934 0.9349 0.9543 ✓ ✓ 89.500 0.9593 0.8250 0.8871 89.500 0.9593 0.8250 0.8871 0.9294 0.9485 balanced   90.000 0.8801 0.9263 0.9026 90.000 0.8801 0.9263 0.9026 0.9370 0.9224 ✓  90.063 0.8913 0.9125 0.9018 90.063 0.8913 0.9125 0.9018 0.9419 0.9337 ✓ ✓ 91.125 0.8871 0.9425 0.9139 91.125 0.8871 0.9425 0.9139 0.9490 0.9135  N/A   85.150 0.7941 0.9490 0.8647 85.150 0.7941 0.9490 0.8647 0.8707 0.8050 ✓  84.650 0.7866 0.9510 0.8610 84.650 0.7866 0.9510 0.8610 0.8755 0.8063 ✓ ✓ 84.500 0.7833 0.9540 0.8602 84.500 0.7833 0.9540 0.8602 0.8706 0.7936 27 maximum values of the metrics in this table with the values provided for this model in Table 2.1, we observe that the performance deterioration has also been moderate in this model when a slight fine-tuning has been applied, with 8%, 11%, 5%, 8%, 5%, and 4% drop in the accuracy, F1-score, precision, recall, area under the ROC curve, and area under the precision-recall curve, respectively. Without fine-tuning, the decrements are respectively 14%, 13%, 20%, 5%, 12%, and 19%, which are expectedly larger, but still acceptable. Thus, we can claim that our proposed model generalizes very well for new and unseen settings. To compare the generalization capability of our model with that of the state of the art, we have trained the models proposed in [41] and [52] on the NTU dataset and tested them on the SDUFall dataset. As depicted in Table 2.4, the performance of both models has drastically declined, which implies their poor generalization when it comes to a new setup. To assess the performance of our proposed model in real life, we have conducted a live test on the model with N1=N2=60. As displayed by Figure 2.5, the setup consisted of a Kinect V2 camera facing the area where the participants were supposed to fall. There were five participants, each performing four falls and four activities of daily life, resulting in a total of 40 test cases. Out of the Table 2.4 Performance of our model compared to the state-of-the-art (trained on NTU RGB+D Dataset and tested on SDU Dataset without fine-tuning) Method Accuracy Precision Recall F1-Score Dilated 1D CNN [41] 70.05 0.6326 0.7210 0.6739 LSTM with 2D joints [52] 85.150 0.7941 0.9490 0.8647 Our Proposed Model (N1=60, N2=60) 70.05 0.6326 0.7210 0.6739 28 20 fall and 20 ADL cases, two falls were missed and six ADLs were mistakenly detected as falls. In summary, there was only a slight drop in the model’s real-life performance compared to staged scenarios, as illustrated in Table 2.5. 2.4 Conclusions In this chapter, we proposed a dynamic-length Long Short-Term Memory (LSTM) neural network for video-based human fall detection with applications in remote health monitoring. Our model Figure 2.5 Setup for live-testing our proposed model Table 2.5 Performance of our model (N1=60, N2=60) in live test Accuracy Precision Recall F1-Score 80 0.75 0.9 0.82 29 takes in the 3D coordinates of the major body joints, extracted from the depth video by the camera or an off-the-shelf module. Using these coordinates, it determines, in real time, whether any fall incident is happening in the video or not. It can handle videos of different lengths, without modifying the video input. With two hidden layers of 60 units and no pre-training, our model achieves 99.87% accuracy, 0.996 F1-score, 0.998 precision, 0.993 recall, 0.999 area under ROC curve, and 0.999 area under precision-recall curve on NTU RGB+D Action Recognition Dataset, beating the state of the art. By minor fine-tuning, it also generalizes well to SDUFall Dataset, a substantially different dataset, experiencing 8%, 7%, 9%, 6%, 3%, and 3% decrease in the respective metrics. Our model has shown to perform extremely well in real-life scenarios. 30 Chapter 3: A Long-Term Recurrent Convolutional Network (LRCN) for Fall Detection Using RGB Videos More personalized service can be offered if the fallen person can be recognized. For example, knowing the identity of the person to access their medical record and health profile, proper medical decisions can be made more quickly and the appropriate help can be delivered to the person. Recognizing the fallen person is particularly important for places with more than one occupant. Although people can be identified in depth videos using specific biometrics, it is complicated and the results are not as accurate as those achieved through RGB videos. Hence, RGB cameras are better options for fall detection where person recognition is also involved to provide personalized service. To cover these applications, we have proposed a model in this chapter that works with RGB cameras. Similar to the model proposed in Chapter 2, it consists of an LSTM neural network. However, instead of the body joints locations, spatial features extracted by a 2D CNN from the RGB frames are the inputs to the LSTM neural network. In the rest of this chapter, the proposed model is explained in Section 3.1, and Section 3.2 includes the performance evaluations and results. Section 3.3 concludes the chapter. 3.1 Proposed Method Our proposed model consists of two main modules, a 2D CNN and an LSTM neural network, making for an overall long-term recurrent convolutional network (LRCN) architecture. Figure 3.1 illustrates the structure of our model. The 2D CNN processes the spatial information of the video by taking the frames one by one and converting each frame into a feature vector. The processed frames are fed to the LSTM neural network as sequences, using a sliding window, to analyze the 31 temporal information of the video. The classification layer, composed of a fully-connected layer with Softmax, takes the output of the LSTM network and determines whether the window of frames processed by the LSTM network include a fall incident or not. In practice, the video goes to the model as a stream, and the CNN processes each frame upon receipt. Assuming the sliding window consists of 𝐿 frames, the output of the CNN for the received frame is then given to the LSTM network, along with the outputs of the 𝐿 − 1 previous frames, for temporal processing. Since the length of all the input sequences to the LSTM network is the same, i.e. equals 𝐿, there is no need for a dynamic-length LSTM network. Thus, we have employed a regular LSTM neural Figure 3.1 Overview of our proposed LRCN model LSTM Neural Network2D CNN2D CNN2D CNNFCSoftmaxClassification LayerInput (Window of Frames)Output (Fall or non-fall)T-(L-1) T-(L-2) T32 network, which is composed of a single layer for low model complexity. The details of the CNN module and the overall LRCN model are explained in the following sub-sections. 3.1.1 Two-dimensional Convolutional Neural Network (2D CNN) Module We have employed the 2D CNN in our model to encode the spatial information of each frame into a feature vector. In our quest for designing an efficient network architecture for our task, we have tested three prevalent CNNs, namely VGG16 [49], Inception-v3 [53], and MobileNetV2 [54]. These CNNs are primarily used for image classification and are pre-trained on the ImageNet dataset [55]. We have also built a custom 2D CNN, depicted in Figure 3.2, specifically designed for this task. Figure 3.2 Architecture of our custom CNN 3 3 Conv. (32)Max Pooling3 3 Conv. (64)Max Pooling3 3 Conv. (128)FC(128)FC(6)SoftmaxClassification Layer256 256 3 (input)254 254 32127 127 32125 125 6462 62 6460 60 12812833 The ideal strategy to select the best one of these CNNs for our task is to base the comparison on the performance of the whole LRCN model, i.e. to choose the CNN that results in the highest performance of the LRCN. However, to the advantage of time, data, and computational cost, we have optimized the CNN individually, using an auxiliary classification task relevant to the original fall detection task. This technique will be a good approximate of the ideal strategy if the auxiliary task is designed properly. We have defined this task as action classification, with special focus on actions associated with falling. The CNN should classify input frames into one of the six classes, namely walking, sitting, falling, fallen, recovering, and recovered. All of the VGG16, Inception-v3, and MobileNetV2 neural networks, pre-trained on the ImageNet dataset, have a 1000-unit fully-connected classification layer, corresponding to the 1000 image classes in the ImageNet. To train them on the auxiliary task, we have replaced this layer with one consisting of six units to match the number of classes in the new task. Moreover, we have added a dense layer of 2048 units before the classification layer in the Inception-v3 and have replaced the last two dense layers of the VGG16 with one dense layer of 128 units. Details of the training and validation procedures are explicated in Section 3.2. Table 3.1 illustrates the numbers of parameters and the validation accuracies of the four evaluated CNNs. Our custom CNN and VGG16 beat Inception-v3 and MobileNetV2 with a high margin. In addition to higher Table 3.1 Validation results of different CNNs on the auxiliary action classification task CNN Model # Parameters Validation Accuracy VGG16 138.4M 78% Inception-v3 24M 69% MobileNetV2 3.4M 65% Our Custom CNN 59M 80% 34 accuracy, our custom CNN has half the number of parameters as VGG16. Therefore, we decided to proceed with our custom CNN as the spatial encoder in our model. 3.1.2 Long-Term Recurrent Convolutional Network (LRCN) Model To build the LRCN, the classification layer of the 2D CNN, i.e. the last fully-connected layer with Softmax, is removed, so that the spatial features extracted by the CNN can be directly fed to the LSTM network. With our custom CNN selected, the input sequence to the LSTM network is comprised of vectors of 128 feature values. As we have used only one layer for the LSTM network, it has two hyperparameters that have been determined by optimizing the performance LRCN on the fall detection task. They include the number of the units in the hidden layer and the size of the sliding window, i.e. the length of the input sequence to the LSTM network. The procedure and results are discussed in the next section. The output of the last block of the LSTM network, which is actually the output of the hidden layer of the 𝐿𝑡ℎ block, goes to the fully-connected classification layer. Softmax is applied on the output of this layer, indicating the class scores, to obtain the probability of each class. 3.2 Performance Evaluation and Results We have figured out the optimal architecture for our model and optimized its hyperparameters through conducting several experiments. These experiments and the training procedure are explicated in this section, in addition to the performance evaluation and results. As mentioned in Section 3.1.1, we have trained the modules of our model sequentially from the bottom to the top. First, the 2D CNN was trained on the auxiliary task. Then, the whole LRCN was trained on the main fall detection task, while the CNN was kept frozen, so that the LSTM 35 network could get trained. To investigate the performance of the four 2D CNNs to select the best one, for the VGG16, Inception-v3, and MobileNetV2, we have employed the publicly available pre-trained weights for the base layers and have trained the modified top layers on the auxiliary task, while keeping the base layers frozen. On the other hand, our custom CNN has been trained from scratch. The weights obtained for the CNN in this step have been kept frozen when training the rest of the LRCN. We have employed cross entropy loss and have applied dropout with ratio 0.5 and L2 regularization to prevent overfitting. Adam optimizer [56], with the initial learning rate of 0.001 has been used for gradient descent. The learning rate has been divided by 10 whenever the validation loss has plateaued for 10 and 7 consecutive epochs in the CNN and LSTM training, respectively. The batch size has been selected as 128 and the number of epochs has been set to 100, with early stopping defined for the LSTM training if the validation loss does not decrease for 6 consecutive epochs. All the training and evaluations have been done on our UBC-DML Fall Dataset, which is explained in detail in Chapter 6. The data labeling has been different for the auxiliary and main tasks. For the auxiliary task, i.e. action classification, the original frame labels, encompassing six classes, have been used as they are, whereas for the main fall detection task, windows of frames have been labeled as either fall or non-fall. A window is labeled as fall if at least one the first 75% of its frames has the fall label. Otherwise, the window inherits the label of the frame before the last. Table 3.2 shows the number of the frames for each of the six action labels and the number of fall and non-fall windows. The counts of the windows are based on the window length of 𝐿 = 16. In the early stages of developing the model, we noticed through live testing that it was very sensitive to the placement of the mat in the image frame. To overcome this weakness, we 36 introduced data augmentation into the training of the CNNs, where we applied random zooming or horizontal flipping on the training data frames and added the obtained frames to the training subset. To keep the complexity of our model low, we have used only one hidden layer for the LSTM module. The number of the units for this layer is a hyperparameter determined through experiment. We have also evaluated the performance of the LRCN with one dense layer added between the LSTM network and the classification layer, with the same number of units as the LSTM hidden layer. Out of the tested 30, 60, 100, 128, 256, and 512 units, the 60-unit hidden layer for the LSTM network with a 60-unit dense layer added on top the LSTM network achieved the highest validation Table 3.2 UBC-DML Fall Dataset labels and their corresponding number of samples used for action classification and fall detection tasks (a) For auxiliary task of action classification (frame samples) Label # Frames Walking 231476 Falling 26126 Fallen 112250 Recovering 55180 Recovered 13718 Sitting 50231 (b) For main task of fall detection (window* samples) Label # Windows Fall 144295 Non-fall 324846 * Numbers are for window length 𝑳 = 𝟏𝟔 37 accuracy. The 512-unit hidden layer for the LSTM network with no dense layer after the LSTM network followed closely. The length of the sliding window is another hyperparameter optimized through experiment. It is the number of frames the LSTM network takes in for understanding what goes on within that specific time interval. A very short window misses the time scope needed to understand the actions, while a long window causes lag in the processing, disrupting the real-time performance of the system. We have considered windows of 8, 16, 20, 30, and 60 frames, with the 16-frame window resulting in the highest validation accuracy. The performance of our proposed model with the optimal hyperparameters is depicted in Table 3.3. In all the experiments described so far, the dataset has been randomly split into the train and validation subsets at the ratio of 80% to 20%, respectively. The same subsets have been used for both the main and auxiliary task. In the last step of our experiments, we have modified the classification layer of the model and investigated its performance in comparison to our initial LRCN. In these experiments, the dataset has been split into the train, validation, and test subsets at the ratios of 60%, 20%, and 20%, respectively. The validation subset is the same as the one used in the previous experiments. In fact, the previous train subset has been split into two subsets, namely the train and test subset. Furthermore, at this stage of our experiments, we have applied data normalization on the inputs of the models, i.e. the video frames fed to the CNN. To have a Table 3.3 Performance of our proposed model on the validation subset Accuracy F1-score Precision Recall # Total Parameters 85.7 0.77 81 73 57.31 M 38 fair comparison, our initial LRCN has been re-trained on the new train subset using data normalization. The classification layer of our initial LRCN consists of a fully-connected layer with six units, corresponding to the six action classes used for the auxiliary task. To generate a binary output, matching the two classes associated with the main fall detection task, the windows detected by the LRCN as either “Falling” or “Fallen” are announced as fall and the others are categorized as non-fall. In the modified classification layer, the six-unit fully-connected layer has been replaced with one composed of two units to generate a binary output directly. Table 3.4 shows the performance of the two models on the test subset. For the modified LRCN, the precision, recall, and accuracy have been reported at the threshold that results in the maximal F1-score. There is no threshold involved in the classification layer of the initial LRCN. Therefore, the ROC and PR curves are not defined for it to report. According to Table 3.4, the LRCN with two-unit fully-connected classification layer performs substantially better, with 0.08, 6.5%, 9.8%, and 4.7% increase in the F1-score, precision, recall, and accuracy, respectively. 3.3 Conclusions In this chapter, we proposed a custom long-term recurrent convolutional network (LRCN) for video-based human fall detection. It is designed to work with RGB videos, so that it can be used Table 3.4 Performance of the LRCNs with different classification layers on the test subset # Units in the FC Classification Layer F1-score Precision Recall Accuracy ROC_AUC PR_AUC # Total Parameters 6 0.7741 81.14 74.27 86.31 N/A N/A 57.31 M 2 0.8585 87.67 84.1 91.06 0.9518 0.9153 57.31 M 39 in the systems where person identification is required for personalized service. Our model consists of a custom CNN and an LSTM network. The CNN takes in the video stream frame by frame to extract the spatial features. The output of the CNN goes to the LSTM, using a sliding window, for temporal processing. A score is given out for each window of frames as the model estimation for the probability of the window containing a fall incident. Based on the experiments we ran on the UBC-DML Fall Dataset, a single-layer LSTM network with 60 hidden units, followed by a 60-unit dense layer and a direct binary classification layer is the optimal structure for our proposed model. It achieved 91% accuracy, 0.86 F1-score, 0.88 precision, 0.84 recall, 0.99 area under ROC curve, and 0.98 area under precision-recall curve on this dataset. 40 Chapter 4: A Transformer-Based Convolutional Neural Network for Real-Time Fall Detection Using RGB Videos Privacy and security have been major concerns when using video cameras, especially RGB cameras, for human monitoring in households. Industry is trying to address this issue by avoiding sending video content to a central processing unit and rather perform the entire detection process on the camera itself in real time. With this solution in mind, in this chapter we propose another deep neural network for fall detection using RGB videos. Different from the model proposed in Chapter 3, being light-weight, along with high accuracy, is the top criterion in designing this model, so that it can function smoothly in real time on cameras embedded boards. It is comprised of a transformer on top of a two-dimensional convolutional neural network. Transformers are a breakthrough in sequential data analysis. Unlike LSTM neural networks, transformers allow for parallel computation that decreases inference time, supporting the real-time performance of the system. The rest of this chapter is structured as follows. Section 4.1 explains the proposed method. The conducted experiments for performance evaluation are elaborated in Section 4.2, and the results are discussed in detail in the same section. Section 4.3 concludes the chapter. 4.1 Proposed Method Our proposed model consists of two main modules, a 2D CNN and a transformer, which process the spatial and the temporal information of the video, respectively. Figure 4.1 illustrates the structure of our model, named TrCNN. The CNN module takes in the frames one by one and converts each frame into a feature vector. Sets of processed frames formed by a sliding window 41 are fed to the transformer module to determine if they contain a human fall or not. In practice, the video stream is processed by the CNN frame-by-frame, as each frame is received. Assuming the window length of 𝐿, the CNN output for the current frame together with CNN outputs for the 𝐿 − 1 previous frames go through the transformer. The output of the transformer indicates whether a fall incident is occurring or not. The modules of our model are explained in detail in the following sub-sections. Figure 4.1 Overview of our proposed model (TrCNN) Transformer2D CNN2D CNN2D CNNFC FC FCPoolingFCSoftmaxClassification LayerInput (Window of Frames)Output (Fall or non-fall)T-(L-1) T-(L-2) T42 4.1.1 Two-dimensional Convolutional Neural Network (2D CNN) Module The 2D CNN module extracts spatial features from the frames and produces a compressed representation of each frame in the form of a feature vector. Since there is no particular requirement for the CNN in our model, except for the common computational complexity constraint, we had a whole variety of neural networks to choose from. We have tested a few of the prevailing 2D CNNs that are employed for image classification tasks to select the one with the best performance. We have experimented with VGG16 [49], ResNet50 [46], ResNet101, ResNet152, DenseNet121, DenseNet169, and DenseNet201 to investigate both classic and state-of-the-art CNNs. Like other prevalent CNNs used for image classification, these networks are originally trained on the ImageNet dataset [55], and their pre-trained weights are publicly available. To facilitate the training process and in consideration of the data scarcity, we have used the pre-trained weights. All these CNNs have a fully-connected layer as their final classifier layer. It is composed of 1000 units, which corresponds to the 1000 image classes in the ImageNet. It takes the output of the previous layer and predicts a score for each class. When the CNN is used for another application, the number of the units for this layer is modified to match the number of the classes in the new application. To attach the 2D CNN to the transformer, this fully-connected classification layer is removed, and the output of the layer beneath it, which comprises the spatial features extracted from the frame, is fed into the transformer. To select a 2D CNN from the ones mentioned above, the regular approach is considering the performance of the whole TrCNN, i.e. the CNN together with the transformer, for different CNN architectures. To the advantage of time and data consumption and computational cost, we have instead optimized the CNN module separately from the rest of the model. To this end, we have 43 compared the performance of the above CNNs on an auxiliary classification task. This technique will be a good approximate of the regular approach if the auxiliary task is designed properly, so that it can well assess the discriminative power of the features extracted by the CNN. Hence, we have designed the auxiliary task in accordance with the original fall detection task. It has been defined as action recognition, with special focus on the actions associated with falling. The CNN is supposed to classify each frame into one of these six classes: walking, sitting, falling, fallen, recovering, and recovered. We have trained all the previously mentioned 2D CNNs for the auxiliary task and have evaluated their performance. The procedure details are explained in Section 4.2. Table 4.1 includes the validation accuracy in addition to the number of parameters for each CNN. The number of parameters is an indicator of the network computational complexity, which is especially important in edge computing where the model is implemented on an embedded or single-board processing system. This can be the case where cloud computing or local servers are not either available or preferred, and cameras with built-in processors are used. Therefore, we have taken the computational complexity into consideration when choosing the 2D CNN. According to Table 4.1, Table 4.1 Validation results of different CNNs for the auxiliary action classification task CNN Name # Parameters Validation Accuracy VGG16 138.4M 78.00% ResNet50 25.6M 84.20% ResNet101 44.7M 84.21% ResNet151 60.4M 85.04% DenseNet121 8.1M 84.44% DenseNet169 14.3M 84.64% DenseNet201 20.2M 84.70% 44 ResNet151 has the highest validation accuracy, followed by the DenseNet variations, while it also has the largest number of parameters after VGG16. Considering the trade-off between performance and computational complexity, we have chosen DenseNet121 for our model. Its accuracy is only (a) (b) Figure 4.2 DenseNet121. (a) Network architecture. (b) A 5-layer dense block [57]. 45 0.6% less than that ReseNet151, while it has the fewest number of parameters, which is only 13.41% of that of ReseNet151. The architecture of DenseNet121 is depicted in Figure 4.2 [57]. 4.1.2 Transformer Module The output of the 2D CNN represents the spatial content of the individual frames very well. However, actions are fully described by the temporal properties in addition to the spatial features. Hence, a mechanism is required to handle the temporal aspect of the videos. By applying the 2D CNN, the video becomes a sequence of one-dimensional feature vectors, and the methods used for processing sequences can be employed to take care of the temporal aspect of the video. For our model, we have proposed transformers. Transformers are state-of-the-art, and their strength and superiority in processing sequential data have been proven. Although transformers can technically process sequences of arbitrary lengths, the computational cost substantially increases with the length of the sequence [58]. Thus, giving the whole video to the transformer is not computationally feasible. Moreover, there is a continuous video stream in practice, and the concept of a whole video does not exist there. Considering these practical points, we have decided to apply a sliding window to the video. At each time step, a window of frames, or more precisely, a window of feature vectors that the 2D CNN generates, is fed to the transformer, and the transformer determines whether a fall is taking place in that window. The size of the window is a hyperparameter that should be selected through experiment. Transformers used in natural language processing are composed of an encoder and a decoder. The encoder gives a compressed representation of the input sequence, and the decoder generates the output sequence. As our purpose of including a transformer in our proposed model is to produce a representation for the windows of frames, we do not need the decoder and use only the encoder. 46 The general structure of the encoder in a vanilla transformer is demonstrated in Figure 4.3. The 2D CNN module in our proposed model does the input embedding by generating a compressed vector representation of the frames. The positional encoding helps the transformer determine the location of each component in the input sequence and the distance between them. It adds a small magnitude vector to each component. The values of the vector vary depending on the location of the component in the sequence. The encoder is a stack of 𝑁 layers. Each layer consists of a self-attention sublayer and a feed-forward neural network, with residual connections [46] and layer normalization [59] for each. The Figure 4.3 Structure of the encoder in a vanilla transformer Encoder Input SequenceFC NNFC NNFC NNMulti-head AttentionNorm+Norm+Norm+Norm+Norm+Norm+Input EmbeddingInput EmbeddingInput EmbeddingPositional Encoding+ + +OutputEncoder: N 47 feed-forward neural network comprises two fully-connected layers with an activation function in between, for which we have employed the Gaussian error linear unit (GELU) [60]. The number of the units in the second layer is the same as the dimension of the input to the first layer to match the residual connection, while the number of the units in the first layer is a hyperparameter and is optimized by experiment. The self-attention sublayer can be single or multi-headed. Different heads give different amount of attention to the elements of the sequence. The number of the heads is also a hyperparameter to be determined through experiment. Keeping the model complexity low and considering the great discriminative power of the features extracted by DenseNet121, proved by its high accuracy in the auxiliary classification task, we have used only one layer for the encoder. The output of the last layer of the encoder is the output of the transformer when there is no decoder. It is composed of a sequence of feature vectors with the same length as the input sequence. The dimension of the output feature vectors is typically the same as the number of the input features count to the first layer of the encoder. In our model, where the transformer does not have feature embedding, it this equals the number of the output features from the 2D CNN. The output of the transformer is given to a fully-connected layer, and Softmax is applied on the output scores to generate the probabilities of each class. Since the transformer is compatible with any sequence length, a pooling layer is required before the classification module. We inspected maximum and average pooling and obtained higher validation accuracy from TrCNN with the latter. Hence, we have opted for average pooling. 48 4.1.3 Transformer Convolutional Neural Network (TrCNN) Model To build the full TrCNN model, the transformer has been placed on top of the 2D CNN. DenseNet121 extracts 1024 features from the input image in the form a vector. For image classification tasks, these features are given to a fully-connected layer consisting of 𝐶 units, where 𝐶 is the number of the classes, to obtain the score of each class. To attach DenseNet121 to the transformer, the fully-connected classification layer has been removed to feed the feature vector to the transformer. The computational complexity of the transformer is polynomial with respect to the dimension of the feature vector received by its first layer [58]. Consequently, reducing the dimension of the feature vectors before giving them to the transformer can significantly decrease the computational cost of our model. Therefore, we have considered feature reduction by adding a fully-connected layer of less than 1024 units between DenseNet121 and the transformer. The number of the reduced features is the same as the number of the units in the added layer and is a hyperparameter to select. We have optimized this number based on the performance of the whole TrCNN. However, to get an estimate of the range to perform the grid search on, we have first assessed the discriminative capability of the reduced features for different number of features, which equals the number of units in the added layer. To this end, we have trained the module Table 4.2 Validation results of the reduced features for the auxiliary action classification task Number of Features Validation Accuracy 64 84.38% 128 84.77% 256 84.64% 512 84.73% 1024 (No feature reduction) 84.44% 49 consisting of DenseNet121 and the added layer on the auxiliary classification task and have evaluated its performance. We have considered layers of 64, 128, 256, and 512 units, in comparison to the module without any added layer. When training the module, we have kept the pre-trained DenseNet121 frozen and have only changed the weights of the added layer. Table 4.2 contains the validation results. Except for the case of 64 features, all numbers of reduced features result in a slightly better performance compared with the original 1024 features. Hence, we have discarded the case of 64 features when optimizing the number of features based on the performance of the TrCNN. 4.2 Performance Evaluation and Discussion We have run comprehensive experiments to figure out the best architecture and optimal hyperparameters for our model. We have evaluated the performance of our final model and compared it to that of the LRCN model we proposed in the previous chapter to examine if the substantial reduction in the model complexity has had any negative effects on the model performance. This section explains the details of the experiments, including the training, model selection, and evaluation procedures. We have taken a modular training approach to be more efficient regarding the computational cost, and time and data consumption. We have trained the modules sequentially in bottom-to-top order. Each module has been trained with the already trained modules beneath while keeping the lower modules frozen. The CNN module has been trained on the auxiliary task, as explained in Section 4.1.1, while the main fall detection task has been used to train the transformer module. We have employed cross entropy loss, Adam optimizer [56], L2 regularization, and batch size of 128 in training both the main and the auxiliary tasks. The initial learning rate for the auxiliary and 50 main tasks has been set at 0.0001 and 0.001, respectively. If the validation accuracy has not improved within respectively 10 and 7 epochs, the learning rate has been divided by two. Like the experiments on the LRCN, which were described in Section 3.2, our UBC-DML Fall Dataset, explicated in Chapter 6, has been used in all the experiments. We have employed the same train, validation and test subsets as the ones used in the last step of the LRCN experiments. The ratios of the three data subsets are respectively 60%, 20%, and 20%. The same subsets have been used for both the main and auxiliary tasks. Data normalization has been applied on the inputs of the model, which are the frames given to the CNN module. Similar to the experiments on the LRCN, the labeling of the data has been done at the frame and window levels for the auxiliary and main tasks, respectively. At the first step of the training procedure, i.e. when training the 2D CNN module on the auxiliary task, we obtained high training and low validation accuracies that implied overfitting. Applying data augmentation on the training data, including random cropping of the frames from 256 pixels × 256 pixels to 224 pixels × 224 pixels and random horizontal flipping of the frames, solved the problem. Data augmentation has been employed in all the trainings. The validation and test data have also been cropped to match the model, but theirs has been center-cropping rather than random-cropping. The hyperparameters to be optimized through experiment are the size of the sliding window and the numbers of the reduced 2D CNN features, the transformer self-attention heads, and the units in the first layer of the transformer feed-forward neural network. For simplicity, we call the last parameter the “feed-forward hidden dimension”. For the transformer self-attention sublayer, we have inspected 1 to 16 heads. The single-head self-attention sublayer has demonstrated the best 51 performance in the preliminary experiments, and we have continued the experiments with it. The size of the sliding window should not be too small or too large. A window that is too short loses the temporal depth required for recognizing the actions, whereas a window that is too long results in a lag in the action recognition, including the fall detection. Thus, we have experimented with windows of 8, 16, 30, and 60 frames, which are equivalent to roughly 0.25, 0.5, 1, and 2 seconds of video. We have considered 2n with n ∈ [2,9] for the feed-forward hidden dimension and, as explained in Section 4.1.3, 128, 256, and 512 for the number of the reduced features in addition to the 1024 original features. Table 4.3 contains the validation results for the combinations of the hyperparameters values at which the TrCNN has performed better. Windows of 8 and 60 frames and feed-forward hidden dimensions less than 16 or more than 256 resulted in lower performance of the TrCNN, so we have not presented them. Table 4.3.(a) depicts the optimal precision-recall pair over the whole threshold span of [0,1] with the other metrics measured at the same optimal threshold. Table 4.3.(b) does the same for the accuracy over the threshold span. The precision-recall pair has been optimized by maximizing each of the two metrics while keeping their difference minimal. It has been approximated by maximizing 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑟𝑒𝑐𝑎𝑙𝑙𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑟𝑒𝑐𝑎𝑙𝑙, which is equivalent to maximizing the F1-score. ROC_AUC and PR_AUC indicate the area under the curve for the ROC and precision-recall curves, respectively. According to Table 4.3, the TrCNN achieves its best performance with 128 reduced features, 16-frame window, and feed-forward hidden dimension of 32. For all numbers of the features and 52 Table 4.3 TrCNN validation results for different hyperparameters values (a) Optimal precision and recall # Features FF Hidden Dimension Window Size Maximum F1-score Optimal Precision Optimal Recall Accuracy ROC_AUC PR_AUC 128 16 16 0.8973 89.84 89.63 93.41 0.9765 0.9592 128 16 30 0.8956 90.1 88.98 92.74 0.9729 0.9598 128 32 16 0.9037 91.85 88.94 93.91 0.9779 0.9589 128 32 30 0.8937 88.95 89.8 92.53 0.9748 0.9623 256 32 16 0.9007 89.86 90.28 93.61 0.9672 0.9386 256 32 30 0.8721 87.7 86.73 91.1 0.9325 0.8865 256 64 16 0.9011 90.89 89.35 93.7 0.9761 0.9583 256 64 30 0.8799 88.65 87.33 91.65 0.9686 0.952 512 64 16 0.8864 88.7 88.53 92.71 0.9612 0.9078 512 64 30 0.8817 88.39 87.95 91.74 0.9711 0.9550 512 128 16 0.8870 90.79 86.7 92.9 0.9645 0.9449 512 128 30 0.8849 89.59 87.42 92.04 0.9721 0.9571 1024 128 16 0.9002 90.98 89.08 93.66 0.9520 0.9051 1024 128 30 0.8455 90.99 78.96 89.9 0.8892 0.8869 1024 256 16 0.8965 89.44 89.86 93.33 0.9765 0.9566 1024 256 30 0.8708 83.2 91.34 90.51 0.9471 0.8657 (b) Maximum accuracy # Features FF Hidden Dimension Window Size Maximum Accuracy F1-score Precision Recall ROC_AUC PR_AUC 128 16 16 93.42 0.8971 90.06 89.37 0.976 0.9592 128 16 30 92.81 0.8938 92.44 86.53 0.9729 0.9598 128 32 16 93.93 0.9030 92.66 88.07 0.9779 0.9589 128 32 30 92.73 0.8935 91.65 87.17 0.9748 0.9623 256 32 16 93.63 0.9006 90.31 89.81 0.9672 0.9386 256 32 30 91.1 0.8721 87.72 86.71 0.9325 0.8865 256 64 16 93.76 0.9004 92.31 87.88 0.9761 0.9583 256 64 30 91.69 0.8779 90.28 85.44 0.9686 0.9521 512 64 16 92.78 0.8862 89.78 87.49 0.9612 0.9078 512 64 30 91.77 0.8815 88.7 87.54 0.9711 0.9550 512 128 16 92.92 0.8858 91.92 85.48 0.9645 0.9449 512 128 30 92.07 0.8844 90.25 86.7 0.9721 0.9571 1024 128 16 93.66 0.9001 91.12 88.92 0.9520 0.9051 1024 128 30 89.91 0.8445 91.67 78.28 0.8892 0.8869 1024 256 16 93.35 0.8962 89.88 89.37 0.9765 0.9566 1024 256 30 90.56 0.8699 84.01 90.19 0.9471 0.8657 53 hidden dimensions, the 16-frame window provides better performance. The optimal hidden dimension doubles as the number of the features does, except for the case of original 1024 features, where the optimal hidden dimension is the same as that of the 512-feature case. The computational complexities of the transformer module and the whole TrCNN model are demonstrated in Table 4.4 for the original 1024 features and the reduced numbers of features. Without the feature reduction module, the complexity of TrCNN substantially increases, while its performance stays the same and even drops a little. Remarkably, the best performing configuration has also the least computational complexity. To assess the performance of our proposed model on unseen data, we have evaluated the TrCNN on the test subset. The model has been tested at all numbers of the reduced features and the original 1024 features with the feed-forward hidden dimension and window size set to their optimal values for the corresponding number of features. We have compared the performance of the TrCNN to that of the last version of the LRCN proposed in Chapter 3. All the data subsets, including the test subset, has been the same for both to have a fair comparison. The test results are provided in Table 4.5. Table 4.4 Number of parameters for the transformer module and the overall TrCNN model # Features FF Hidden Dimension # Transformer Parameters # TrCNN Parameters 128 32 77,218 8.177 M 256 64 301,890 8.401 M 512 128 1,193,602 9.293 M 1024 128 4,746,498 12.846 M 54 In accordance with the validation results, the TrCNN with 128 reduced features demonstrates the best performance on the unseen data, with 95% accuracy, 0.92 F1-score, 0.92 precision, 0.92 recall, 0.99 area under ROC curve, and 0.98 area under precision-recall curve. It has the feed-forward hidden dimension of 32 and the window size of 16 frames. The test results prove that the TrCNN outperforms the LRCN by a considerable margin, though it has substantially fewer parameters and therefore, significantly less computational complexity. 4.3 Conclusions In this chapter, we proposed TrCNN, a novel deep-learning-based model for human fall detection using RGB videos. It is composed of a transformer on top of a lightweight 2D CNN, enabling it to perform in real time on cameras built-in processors that have low computational power. In this way, it addresses privacy and security concerns associated with RGB video cameras. The video stream goes through the CNN to extract the spatial features frame by frame. The transformer takes windows of the features to process the temporal information. It generates a score for each window, which implies the probability of a fall incident taking place in the corresponding window of frames. According to the experiments conducted on the UBC-DML Fall Dataset, DenseNet 121 with 128 reduced features and a single layer single head transformer with a window size of 16 and a feed-Table 4.5 Comparison of TrCNN and LRCN performance on test subset Model # Features FF Hidden Dimension Window Size Maximum F1-score Optimal Precision Optimal Recall Accuracy ROC_AUC PR_AUC # Total Parameters TrCNN (Our Proposed Model) 128 32 16 0.9237 92.38 92.36 95.08 0.9871 0.9759 8.177 M 256 64 16 0.9237 92.9 91.8 94.59 0.98514 0.9726 8.401 M 512 128 16 0.9111 91.94 90.31 94.32 0.980 0.9607 9.293 M 1024 128 16 0.9144 91.18 91.71 94.47 0.9852 0.9695 12.846 M Our LRCN 128 N/A 16 0.8585 87.67 84.1 91.06 0.9518 0.9153 57.31 M 55 forward neural network of 32 hidden units make up the best combination. With 95% accuracy, 0.92 F1-score, 0.92 precision, 0.92 recall, 0.99 area under ROC curve, and 0.98 area under precision-recall curve, it has significantly improved performance and beats our LRCN model at a considerably less computational cost that helps the real-time performance of the TrCNN on cameras built-in boards. 56 Chapter 5: An Embedded System Prototype Development for Real-Time Fall Detection via RGB Camera In response to privacy concerns associated with RGB cameras, we proposed a light-weight fall detection model in Chapter 4 that works with RGB video and we expect can perform in real time on cameras built-in boards. In this chapter, we present an embedded system we have prototyped for proof of concept to evaluate the performance of this model on boards with low computational power and compare it to that of the model proposed in Chapter 3. Our prototype consists of a Raspberry Pi Camera V2 and an NVIDIA Jetson Nano Developer Kit. We overview the available hardware options in Section 5.1 and describe the details of the designed prototype in Section 5.2. Section 5.3 contains the results of the live test on the two models, and Section 5.4 concludes the chapter. 5.1 Hardware Options There are various systems on chip (SoC) with machine learning capabilities to be used with an RGB video sensor for our fall detection application. To find an appropriate computing board for our application, which should be portable, nonintrusive, and affordable, we conducted a thorough investigation on the existing boards in the market. Here, we review some of the ones that meet the computational requirements at a reasonable price. 5.1.1 NVIDIA Jetson Series Developer Kits NVIDIA Jetson series developer kits are small powerful single-board computers (SBCs) that can run multiple neural networks in parallel at as little as five watts. They are commonly used in embedded computer vision and speech processing applications. They can run models developed 57 Table 5.1 Specifications of NVIDIA Jetson series developer kits Jetson Nano Jetson TX2 Jetson Xavier NX Jetson AGX Xavier CPU Quad-core ARM Cortex A57 ▪ Dual-Core NVIDIA Denver 2 64-Bit ▪ Quad-Core ARM Cortex A57 6-core NVIDIA Carmel ARM v8.2 64-bit 8-Core NVIDIA Carmel ARM v8.2 64-Bit GPU 128-core NVIDIA Maxwell GPU 256-core NVIDIA Pascal GPU 384-core NVIDIA Volta GPU with 48 Tensor Cores 512-core NVIDIA Volta GPU with 64 Tensor Cores Memory 4 GB 64-bit LPDDR4 @ 25.6 GB/s 8GB 128-bit LPDDR4 @ 59.7 GB/s 8 GB 128-bit LPDDR4x @ 51.2GB/s 32 GB 256-Bit LPDDR4x @ 136.5 GB/s Storage MicroSD card slot ▪ 32GB eMMC 5.1 flash ▪ SD card slot ▪ SATA ▪ MicroSD card slot ▪ M.2 Key M for SSD ▪ 32 GB eMMC 5.1 flash ▪ Micro SD/UFS card socket ▪ M.2 Key M for SSD ▪ eSATAp Camera Interface 2 × MIPI CSI-2 6 × MIPI CSI-2 or 3 × MIPI CSI-4 (5 MP Fixed Focus MIPI CSI Camera included in the kit) 2 × MIPI CSI-2 16 × MIPI CSI-2 or 8 × MIPI CSI-4 USB ▪ 4 × USB 3.0 ▪ 1 × Micro USB 2.0 ▪ 1 × USB 3.0 ▪ 1 × Micro USB 2.0 ▪ 4 × USB 3.1 ▪ 1 × Micro USB 2.0 2 × USB 3.1 Type C Power Supply ▪ 5V/4A via power barrel ▪ 5V/2A via micro USB ▪ Optional PoE support 5.5V to 19.6V via power barrel 9 to 19V via power barrel 9 to 20V via power barrel Power Consumption 5-10W 7.5-15W 10-15W 10-30W Cooling Heatsink Fan Fan Fan Dimensions 100 × 80 ×26 mm 170 × 170 × 50 mm 103 × 90.5 × 31 mm 105 × 105 × ~55 mm Price 130 CAD 780 CAD (EOL) 910 CAD 520 CAD 58 by Keras in addition to Tensorflow models. Using TensorRT, an open-source library designed by NVIDIA for GPU inference acceleration and highly optimized for NVIDIA GPUs, NVIDIA Jetson series kits allow the full leverage of the GPU resources. However, their GPUs tend to throttle when running heavy inferences, and an additional heat sink may be needed. This series include Jetson Nano, Jetson TX2, Jetson Xavier NX, and Jetson AGX Xavier, in the ascending order with respect to computational power, which makes it easy to scale up the implemented solutions in computational power. Table 5.1 illustrates the specifications of each model of this series. As NVIDIA does not produce its own cameras, these boards have to work with third party cameras. This interface might not be completely seamless. 5.1.2 Amazon Web Services (AWS) DeepLens AWS DeepLens is a deep learning-enabled video camera that is integrated with the AWS machine learning services and can run models deployed from the AWS Cloud. Table 5.2 shows the specification of AWS DeepLens. Compared to NVIDIA Jetson series developer kits, it has less powerful GPU, no external general-purpose input/outputs (GPIOs), and no choice for the camera except for the built-in one. We can use an Intel Neural Compute Stick 2 to make its GPU functionality on par with the least powerful Jetson developer kit, namely Jetson Nano Developer Kit at an additional cost of 150 CAD. Although the integration with AWS services provides convenience, its substantial dependence on AWS may impose limitations in some aspects. 5.1.3 Raspberry Pi 4 B combined with Intel Neural Compute Stick 2 or Google’s Coral USB Accelerator Raspberry Pi 4 B is the standard general-use model in the fourth generation of Raspberry Pi SBCs, which are famous for their small sizes and low prices. This model, the specifications of which are 59 listed in Table 5.3, can be purchased at 75 CAD with 4GB of RAM or 105 CAD with 8 GB of RAM. As Raspberry Pi SBCs are not designed for deep learning inferences, they tend to throttle under processing peaks and do not have very powerful GPUs. However, they are compatible with various AI ASICs, such as Intel Neural Compute Stick 2 and Google’s Coral USB accelerator, though connecting an AI ASIC increases the size of the whole embedded system as well as blocking one of the USB ports that can be used for other general purposes. It should be noted that Google’s Edge TPU, which is the AI module of Coral USB, only supports the Lite version of Tensorflow. Table 5.2 Specifications of AWS DeepLens CPU Quad-core Intel Atom X5 GPU Intel Gen9 Memory 8 GB Storage ▪ 16GB eMMC flash (built-in) ▪ MicroSD card slot Camera 4MP (1080p) camera using MJPEG, H.264 encoding USB 2 × USB 3.0 Power Supply 5V/4A Power Consumption 20W Cooling Heatsink Dimensions 168 × 94 ×47 mm Price 340 CAD 60 5.2 Prototype Design Based on our study on the available SoCs, we selected the NVIDIA Jetson Nano Developer Kit, which is compact, compatible with various cameras, and affordable. For the camera, because of technical considerations explained in Sub-Section 5.2.2, we chose the Raspberry Pi Camera V2 that is compatible with the Jetson Nano. Table 5.4 contains the specifications of this camera. To prevent thermal throttling, we added a cooling fan to ensure the temperature of the system stays below 50 °C even under peak processing. Figure 5.1 displays the hardware components of our prototype. Table 5.3 Specifications of Raspberry Pi 4 B CPU Broadcom BCM2711 Quad-core Cortex-A72 (ARM v8) 64-bit @ 1.5GHz GPU Broadcom VideoCore VI Memory 1GB, 2GB, 4GB, or 8 GB LPDDR4 Storage MicroSD card slot Camera Interface MIPI CSI-2 USB ▪ 2 × USB 3.0 ▪ 2 × USB 2.0 Power Supply 5V/2.5A Power Consumption 4-6W Cooling Heatsink Dimensions 56.8 × 85.6 ×11 mm Price 50 CAD, 60 CAD, 75 CAD, or 105 CAD 61 5.2.1 NVIDIA Jetson Nano Developer Kit Setup The very first step of the setup was to install the NVIDIA Jetpack, which is the NVIDIA SDK for AI-at-the-edge development on Jetson modules and developer kits. It includes the operating system (OS) images for the processor, along with libraries, application programming interfaces (APIs), and developer tools. The SDK should be downloaded onto a microSD memory card to be used for Figure 5.1 Hardware components of our prototype Raspberry Pi Camera V2NVIDIA Jetson NanoCooling Fan32GB SD Card5V/4A Power SupplyTP-Link Wireless USB AdapterNecessary Add-onsTable 5.4 Specifications of Raspberry Pi Camera V2 Image Resolution 8 MP Video Modes 1080p30, 720p60 and 640 × 480p60/90 Sensor Sony IMX219 Weight 3g Dimensions 25 × 24 × 9 mm Price 25 CAD 62 booting the Jetson Nano. The next step was installing all the libraries and dependencies the deep learning model requires to run. Depending on the framework used to develop the model, we installed either Pytorch, for TrCNN, or Tensorflow and Keras, for our LRCN. The installed versions have to be compatible with the ones used to develop the models on the desktop. The last step was configuring the Jetson Nano to read video stream from the camera and output it to its screen display. 5.2.2 Hardware and Software Optimization After the above steps, the Jetson Nano processor could support our deep learning model. The on-board memory of the Jetson Nano Developer Kit is 4 GB, with 3.5 GB usable space. Part of this 3.5 GB is occupied by the OS and background processes. To prevent any crash during the inference, we added a 4 GB virtual RAM, commonly known as swap file, on the Jetson Nano. It starts shifting processes to the hard disk when the available physical RAM is critically low. In this memory setup, 1.5 GB of the physical RAM is dedicated to the maintenance of the Virtual RAM in addition to the OS and background processes, and the remaining 2 GB of the physical RAM along with the whole 4 GB virtual RAM is available for running the model. Using this memory allocation, the models could run on the Jetson Nano without crashing. However, the system took 9 and 17 minutes to load the TrCNN and LRCN, respectively, and start the inference. Furthermore, there was a significant lag in the inference. The reason was that the models had been developed on a desktop computer and not optimized for the Jetson Nano. To address this issue, we employed NVIDIA TensorRT, a C++ SDK that facilitates high-performance machine learning inference on NVIDIA GPUs and deep learning accelerators. It focuses on running already-trained neural networks quickly and efficiently on NVIDIA hardware, 63 while preserving the accuracy, and is highly optimized for NVIDIA hardware. Utilizing the TensorRT, the start-up time substantially decreased. It became 70 seconds and 2 minutes for the TrCNN and LRCN, respectively. Although the algorithm traversed through the model in less than a second, there was still a 3-second lag in the inference that violated the requirement for the real-time performance. Maximizing the clock speed, which regulates the CPU and GPU, effectively reduced the latency, but did not completely solve the problem. The root cause was the fact that all the video preprocessing was performed at a software level using OpenCV, which put significant strain on the system resources. Up to this stage, we were using a Logitech C920 webcam, connected to the Jetson Nano via USB. To tackle the lag problem, we switched to a Raspberry Pi Camera V2, connected to the Jetson Nano through MIPI CSI-2 interface, and employed a GPU accelerated GStreamer pipeline to shift the preprocessing towards hardware image processors. As the Raspberry Pi Camera V2 supports multiple video resolutions and frame rates, we examined different resolutions and frame rates. Thorough experiments proved that 1080p resolution and 30 fps provided a perfect balance between the speed and accuracy, with the Jetson Nano performing the inference in real-time, namely processing 30 frames in less than one second for both models. The final step we took to optimize the on-board implementation was modifying the codes to delete unneeded variables for the better utilization of the extremely limited available memory. Each input frame to the CNN is deleted immediately after its spatial features are extracted by the CNN, and all the variables containing values associated to a frame are deleted as soon as the frame gets out of the sliding window scope. In addition to freeing up memory space, this practice increases users’ 64 privacy and helps the non-intrusiveness of our solution. Garbage collection is also applied on variables when they are no longer needed. We built a 3D-printed mount for the Jetson Nano board and Raspberry Pi camera for our temporary use during the design and bought a metal case and mount for the final product. Figure 5.2 depicts the temporary and final setups. 5.3 Live Tests and Results To evaluate the performance of our models, namely TrCNN and LRCN, on the prototyped embedded system, we have led a series of live tests. Various camera arrangements, i.e. height, vertical angle, and distance from the scene, and several falling scenarios as well as ADLs have been tested. Each case has been performed a few times by different subjects. More than 100 trials have been conducted for each model. The TrCNN has been successful in 92% of the live tests, while the LRCN has successfully passed 88% of the live tests, which are close to the results (a) (b) Figure 5.2 Our prototype. (a) Temporary 3D-printed mount. (b) Metal box and mount. 65 obtained for the original models on the desktop with offline test data. Although our prototype achieves real-time performance for both models, the Jetson Nano runs on the edge of using up all the available memory for the LRCN, which is not the case for the TrCNN. This make it more difficult in the case of the LRCN to add more features such as person identification to the system. 5.4 Conclusions In this chapter we explained the embedded system prototype we have designed and built to assess the performance of our proposed RGB video-based models on a board with limited computational resources. Our prototype is comprised of an NVIDIA Jetson Nano Developer Kit as the single-board computer and a Raspberry Pi Camera V2 as the RGB camera. Adding a virtual RAM to the board and optimizing the models by TensorRT before loading them on the system mitigate the problem of the highly limited memory, while employing GStreamer pipeline reduces the latency. The frames and any associated variables are deleted immediately after they are processed to increase the security and privacy. Live tests proved the real-time performance of both models and confirmed their accuracy aligned with that obtained in offline tests. It shows that our models can perform reliably and in real time on cameras built-in boards to prevent the need for any data transmission that may put the occupants’ privacy at risk. However, the LRCN uses up all the available memory because of its large number of parameters, which may cause problems when more features are added to the system. 66 Chapter 6: UBC-DML Fall Dataset All video-based fall detection systems that use machine learning, i.e., classic machine learning or deep learning, need data to be studied and validated and to be trained before getting into application. Since falling is an uncommon and unpredictable event, natural fall incidents are usually captured by surveillance or monitoring cameras. Most of such cameras are not intended for public use or research and their recordings cannot be released. Those arranged for research purposes are usually installed in private spaces, and because of privacy constraints, they cannot be made public, either. Therefore, videos of real-life fall incidents that are publicly available are very scarce. Consequently, the common practice for researchers is to capture videos of staged falls in laboratory environments, which they may later release to the research community. Among the released datasets, there are only two that are large enough to be used for deep learning approaches [37],[50]. Despite being widely used, these two datasets have some limitations that constrain their usage. They do not cover various falling scenarios, and they consist of very short videos. To address these issues, we, the Digital Multimedia Lab (DML) at the University of British Columbia (UBC), have captured a comprehensive and representative video dataset, named UBC-DML Fall Dataset, which is publicly available and is introduced in this chapter. Section 6.1 reviews the existing public human fall video datasets collected in laboratory environments and discusses their shortcomings. Capturing procedure and setup for UBC-DML Fall Dataset are explained in Section 6.2, along with the attributes of the dataset, and Section 6.3 concludes the chapter. 67 6.1 Existing Public Video Datasets on Human Fall There are several video datasets on human fall that have been captured in laboratory environments for research purposes and have been publicly released. We go over the most prevalent ones, here. 6.1.1 Le2i Fall Detection Dataset (FDD) The FDD dataset [34] consists of 192 fall and 57 ADL videos. It has been captured in four different environments, including home, coffee room, lecture room and office, to portray the elderly home environment and simple office room settings. It has been recorded a single RGB camera placed 2m high from the floor. The videos have a frame rate of 25 fps with an image resolution of 320×240 pixels. Some of videos present occlusions, cluttered or textured background, illumination variations. 6.1.2 UR Fall Detection (URFD) Dataset The UR Fall Detection [25] is composed of front and overhead videos collected by two cameras at 30 fps. One camera is parallel to the floor at a height of 1m and another one is mounted on the 3m ceiling. The dataset consists of 40 ADL and 30 fall videos, performed by five participants. There are 10 fall-like activities, such as quickly lying on the floor and lying on the bed, among the ADL videos. Falling occurs from either standing position or sitting on a chair and includes and include forward, backward and sideways falls. 6.1.3 Multiple Cameras (Multicam) Fall Dataset Multicam dataset [24] includes 24 scenarios captured simultaneously by eight IP video cameras. 22 scenarios contain falling and abrupt actions, while the other two has only abrupt actions. 68 6.1.4 ACT42 Dataset The ACT42 dataset [26] mainly focuses on the ADLs, including 14 actions such as collapsing, drinking, making phone calls, mopping the floor, picking up, putting clothes on, reading books, sitting down, sit-ups, stumbling, taking clothes off, throwing away, twist opening and wipe cleaning. Two categories of falls are considered, namely collapsing and stumbling, which are falling because of internal and external factors, respectively. 6.1.5 TST Fall Detection Dataset v2 TST Fall Detection Dataset v2 [61] has been captured in a laboratory environment, using Microsoft Kinect v2 and two accelerometers placed on the wrist and waist of the participants. Each subject performs four ADLs and four types of falling, each repeated three times, generating a total of 264 videos. The ADLs are comprised of sitting on a chair, walking and picking up an object from the floor, walking back and forth, and lying down on a mattress. The four types of falling include front, backward, and sideward falling, ending up lying, as well as falling backward, ending up sitting. 6.1.6 NTU RGB+D Action Recognition Dataset This dataset consists of 56,880 videos, 948 of which are videos of falling and the rest contain one of the 59 ADLs. Out of these 60 categories of actions, 11 involve two persons and the rest are single-person actions. There are 40 subjects, aging between 10 and 35, who perform each action three times. The scenes have been captured by three Microsoft Kinect v2 cameras, located at the same height with horizontal angles of −45°, 0°, and +45°. The videos have been recorded in three modalities, RGB, depth, and infrared (IR), at 30 fps and the resolutions of 1920×1080, 512×424, and 512×424, respectively. The human skeletons have been extracted on the fly using the 69 Microsoft Kinect SDK. Each skeleton is composed of the 25 major body joints presented in Figure 2.2. The 2D and 3D coordinates of the joints are provided for the RGB and depth modalities respectively. As our proposed method requires only the 3D coordinates of the body joints, only the skeletons have been used in our experiments. Some samples have missing skeletons, and some skeletons have missing joints. These samples have been left out of our experiments. The application of a fall detection system is for the scenarios where the person is alone, so that the incident may not be discovered. Therefore, we have excluded the 11 actions involving two persons. This dataset pruning resulted in 44372 samples, out of which 890 are fall samples. 6.1.7 SDUFall Dataset This dataset is comprised of 1200 videos. 1000 videos belong to one the five categories of single-person activities of daily life, and 200 are videos of falling. Each action is performed ten times by 20 subjects. The recording is done by one Kinect v1 camera in RGB and depth modalities at about 30 fps and the resolutions of 640×480 and 320×240, respectively. Similar to the NTU dataset, the human skeletons have been extracted by the Kinect SDK on the fly. Each skeleton has 20 body joints, which includes all the joints displayed in Figure 2.2 except for joints 3 and 22 to 25. 6.1.8 Shortcomings of Existing Datasets All the public datasets overviewed above, except for the NTU RGB+D and SDUFall datasets, have a small number of fall videos. They may be sufficient to train models that are based on classic machine learning approach and to evaluate the performance of all types of models, though more fall samples may result in more optimal models and can provide assorted and more accurate 70 performance assessments. However, training deep learning-based models requires a large number of samples, and datasets not containing plenty of fall videos are not suitable for this purpose. To the best of our knowledge, the NTU RGB+D and the SDU Fall datasets are the only large datasets of fall videos that are publicly available. The shortcomings of these two datasets are twofold. First, they contain only one falling scenario, where the person enters the scene and immediately and abruptly falls. This does not well represent real-life fall incidents, as they occur in various ways, depending on the cause of the fall and other factors. Their second limitation is the very short duration of the videos in these two datasets. The average video length is three and two seconds for the NTU RGB+D and SDU Fall datasets, respectively. The videos consist of only one action, which constrains the applications of these datasets. To address the need of the research community for a large dataset of fall videos with diverse scenarios and longer videos, we have captured the UBC-DML Fall Dataset. It consists of videos with an average length of 15 seconds and includes a large number of fall videos with various real-life scenarios. 6.2 Capturing Procedure We captured UBC-DML Fall Dataset in three rounds at UBC Vancouver campus. The first and second rounds were arranged in the lounge area of the DML in the ICICS Building, and the third round was set up in a room in the McLeod Building. Figure 6.1 displays the camera setup for each round. We used three or four Microsoft Kinect v2 cameras, synchronously recording the scene from different views. The additional views increase the variety of content and background and add to the total number of videos, creating a more comprehensive and larger dataset. Multi-view applications may also benefit from multiple views. Although some cameras were facing each other, 71 there was no problem, since the distance between them was large enough for their emitted lights not to interfere with each other. We covered a small area, where the subjects were supposed to fall, with shock-absorbent rubber tiles and put a soft mat on it to prevent any injuries. (a) (b) (c) Figure 6.1 Camera setup for capturing the UBC-DML Fall Dataset. (a) Round 1 setup. (b) Round 2 setup. (c) Round 3 setup. KitchenetteSubjects’ Waiting AreaFurnitureDoorDoorDoorMatCam1Cam2Cam3Cam4KitchenetteSubjects’ Waiting AreaFurnitureDoorDoorDoorMatCam1Cam2Cam3DoorMatCam1Cam2Cam3Cam472 6.2.1 First Round of Capturing The first round of the data capturing took place in the DML lounge. There were four cameras on the corners of an 8’ 15’ rectangle, with the mat in the center. There have been 17 participants, mostly in their twenties. Each subject performs 10 to 15 scenarios of daily life activities, 15 to 30 scenarios of falling, and five to 10 scenarios of falling and recovering from it. The ADLs include walking in various directions, housekeeping activities like mopping and wiping the floor, sitting on a chair, and doing exercises. We also incorporated cases into ADL scenarios where the subject is about to fall, but manages to regain balance, and activities similar to falling, such as sitting on the floor, crouching down, and lying down on the floor. The fall scenarios contain forward, backward, and sideward falls, caused by loss of balance, tripping, stroke, heart attack, fainting, pelvic fracture, and dizziness. The recovery from falling is to either a sitting or a standing position. There are also occluded falls, where the subject is partially occluded by a chair. All the videos have been labeled at frame level, with labels presented by Table 6.1. The description of the first group of labels is as follows: Falling: the body moving towards the ground. Fallen: the upper part of the body having reached the ground, and the subject lying on the ground. Recovery: the body moving from the lying position to an upright or sitting position. Table 6.1 Frame labels for the first round of video capturing Group Labels 1 Falling, Fallen, Recovering, Recovered 2 Upright, Sitting, Lying down on the floor, Sitting on the floor, Other 73 Recovered: the subject standing with both feet on the ground or sitting on the ground. Table 6.2 illustrates the video categories with the total number of captured videos in each one. Videos containing “falling” and “fallen” frames without any “recovering” and “recovered” ones go to the “fall” category, while “fall with recovery” videos contain all these four types of frames. Any video that does not belong to either of these two categories is considered an “ADL” video. Sample frames from the first and the second group of labels are shown by Figure 6.2 and Figure 6.3, respectively. 6.2.2 Second Round of Capturing The second round of data was also captured in the DML lounge. The mat was placed in the middle of the capturing stage, and three cameras were arranged asymmetrically around the mat. The arrangement details of the cameras are depicted in Table 6.3. We covered the mat and floor by builder’s paper during this round to prevent any potential bias to the mat and the patterned carpet when using the recordings to train models. There have been 24 participants, mostly in their twenties. Each subject performs three to six scenarios, including activities of daily life, falling, and falling with recovery. The ADLs include walking in various directions, mopping the floor, sitting on a chair, and doing standing exercises. Table 6.2 Number of videos in each category for the first round of video capturing Category Number of Videos ADL 475 Fall 1145 Fall with Recovery 317 74 (a) (b) (c) (d) (e) Figure 6.2 Sample frames of the first group of labels in round 1 video capturing. (a) Falling. (b) Fallen. (c) Recovering. (d) Recovered to sitting. (e) Recovered to standing. 75 (a) (b) (c) (d) (e) (f) Figure 6.3 Sample frames of the second group of labels in round 1 video capturing. (a) Upright. (b) Sitting. (c) Lying down on the floor. (d) Sitting on the floor. (e), (f) Other. 76 Similar to the previous round, we also incorporated into ADLs cases where the subject is about to fall, but manages to regain balance, and activities similar to falling, such as sitting on the floor, lying down on the floor, bending down to pick up something from the floor, and doing exercises that involve lying or sitting on the floor. The fall scenarios are the same as those of the first round. Similarly, the recovery from falling is to either a sitting or a standing position, and there are fall cases partially occluded by a chair. Again, all the videos have been labeled at frame level. Table 3.2.(a) depicts the labels with the total number of frames for each label. Few sample frames from this round of capturing are displayed by Figure 6.4. Table 6.3 Cameras arrangement in the second round of video capturing Camera # Distance from Mat Center Distance from Height Angle towards Floor Camera 1 Camera 2 Camera 3 1 13’8” N/A 11’4” 17’1” 4’10” 30° 2 12’8” 11’4” N/A 21’0” 4’8” 25° 3 9’3” 17’1” 21’0” N/A 5’2” 40° (a) (b) Figure 6.4 Sample frames of the round 2 video capturing. (a) Standing. (b) Falling. 77 6.2.3 Third Round of Capturing In the third round of capturing, performed in the McLeod Building, the cameras were placed symmetrically at the corners of a 15’ 11’ rectangle, looking down at angles of approximately 30 degrees. Similar to the first round, the mat was not covered, but it was rotated and moved around in the rectangular region formed by the cameras. This prevents any potential bias, when training models, to the location of the fall incident in the scene, the distance of the fall incident from the camera, and the angle of the camera regarding the subject. Figure 6.5 shows few samples of frames from the third round of capturing. 6.3 Conclusions In this chapter we introduced a large video dataset, named the UBC-DML Fall Dataset, that we have captured at the University of British Columbia and is publicly available. This dataset is particularly designed to address the limitations of the existing public video datasets for indoor human fall detection. Consisting of indoor ADL and fall videos, it solves these weaknesses that negatively affect the performance of the trained models, especially the ones based on deep (a) (b) Figure 6.5 Sample frames of the round 3 video capturing. (a) Falling. (b) Standing. 78 learning. To meet the requirements of the video-based fall detection research community for a comprehensive and representative dataset, the UBC-DML Fall Dataset is composed of relatively long videos and encompasses various falling scenarios. 79 Chapter 7: Conclusions 7.1 Summary of the Contributions In this thesis, we proposed three novel methods using deep learning approach to detect indoor human fall incidents, all of which preserve occupants’ privacy. While one of the methods uses depth information and preserves identities, the other two use RGB video input to identity and track the fallen person, thus providing better service through personalized health support. We also introduced a new, large and comprehensive video dataset that we have captured for human fall detection, addressing one of the major bottlenecks in training video-based models for human monitoring. In Chapter 2, we proposed a method that was one of the first video-based fall detection solutions exerting deep learning approach. It was the first deep learning-based method working with depth cameras and requiring only the 3D locations of major body joints as the input, and thus, fully preserving the identity of the occupants as well as their privacy. The body joints are extracted on the fly by either the camera’s SDK or an off-the-shelf software. Since they encode the part of the spatial information of videos that describes human actions, further spatial processing is not needed, and they are directly fed to an LSTM neural network for temporal analysis. The output of the LSTM network is positive if the input sequence contains a fall. In contrast to the existing approaches, our proposed model is developed to process videos of varying lengths. Short videos are presented to the network in their entirety, without the need to be cut or zero-padded to a fixed length. Long or streaming videos are fed to the network in segments, where the length of the segments is arbitrarily selected. According to the experiments we conducted on the existing large 80 public fall datasets, our model outperforms the state-of-the-art video-based methods that use body joints for fall detection. Furthermore, it generalizes well to datasets it has not seen in the training. In Chapter 3, we proposed a deep learning-based method that works with RGB cameras to be used for applications where the identification of the fallen person is needed to provide better-quality assistance and personalized medical help. Similar to the method proposed in Chapter 2, it employs an LSTM network to analyze the temporal information of the video. The input to the LSTM is produced by a 2D CNN that processes each frame to extract discriminative and representative spatial features in the form of a vector. The LSTM takes in a sequence of these vectors, corresponding to a window of consecutive frames, and determines if the window of frames is part of a fall event. We improved this model in Chapter 4 to make it very lightweight, so that can more reliably perform in real time on cameras built-in boards, which is necessary to address the privacy concerns associated with employing RGB cameras for fall detection. We have employed DenseNet121, a lightweight yet high-performing 2D CNN, to process the spatial information of the videos. The temporal aspect is handled by a transformer, the sate-of-the-art deep neural network for sequence analysis. It is more parallelizable than LSTM network and therefor, better supports the real-time performance of the model. For lower computational complexity of the model, the dimension of the extracted spatial features is reduced passing through a fully-connected layer, before going to the transformer. The model takes in videos in segments and flags the segments as positive if they embody a fall event. In Chapter 5, we presented the hardware prototype we have built for the live test of the models proposed in Chapter 3 and Chapter 4. It is to verify that our proposed models can perform in real 81 time on the built-in boards of RGB cameras, and no video transmission, triggering privacy concerns, is required. The prototype consists of a Raspberry Pi Camera V2 working with an NVIDIA Jetson Nano Developer Kit. We have employed various technics for the optimal implementation of the models, so that they can utilize the limited available computational resources more efficiently. They include applying TensorRT on the models and utilizing a GStreamer pipeline. The live tests proved edge computing to be feasible for both models, though the model proposed in Chapter 3 is on the margin. In Chapter 6, we introduced UBC-DML Fall Dataset, the large video dataset we have captured to address the need of the fall detection research community for a comprehensive and representative public video dataset. It has been recorded in a laboratory environment and consists of both ADL and fall videos. This publicly available dataset consists of videos that are longer than those of the existing public datasets and encompasses various falling scenarios, which gives an advantage to our dataset over the existing ones. We used our dataset to train and evaluate the models we proposed in Chapter 3 and Chapter 4 and to conduct offline tests on them, which confirmed their high accuracies. 7.2 Significance and Potential Applications of the Research The research presented in this thesis aims at improving the automatic detection of indoor fall incidents by incorporating the state-of-the-art deep neural networks, while keeping privacy and real-time performance as the top priority. In addition to achieving high accuracy on data similar to that they are trained on, our proposed models have acceptable performance on totally different data, which can be enhanced with minimal retraining, i.e. through finetuning. It makes them 82 compatible with various space setups and conditions without making much overhead and results in higher performance in practice. In collaboration with TELUS Communications Inc. and the department of emergency medicine at the University of British Columbia, we had a demo at Vancouver General Hospital (VGH), where we presented a live test of the model proposed in Chapter 4, using the prototype described in Chapter 5. Hospitals, nursing homes, and the providers of digital personal health solutions, like TELUS Health, can benefit from the models we have developed. With the wide variety in its falling scenarios and the long videos it contains, using the UBC-DML Fall Dataset for training fall detection models equips them with more generalization capacity, which leads to better compatibility with different spaces and higher performance. It also enables more comprehensive evaluation of models in the design stage and before prototyping, which prevents wasting time and money on the failing ones. 7.3 Future Work Although our UBC-DML Fall Dataset contains occluded fall scenarios, we have not particularly examined the performance of our proposed models in this regard. As part of our future work, we will assess how well our models can handle occlusion, and based on the results, we may modify the models to improve their performance in occluded fall detection. Another situation which has to be investigated is multi-person scenario. Although automatic human fall detection systems are mainly designed for situations where the person is alone, they are still expected to accurately detect fall events when more people are present in the scene. The performance of our proposed models for multi-person scenarios can be evaluated through live 83 testing. We expect a performance drop, which should be addressed. A potential solution is to add a subject detection and tracking module to the pipeline. It is more straightforward for the model proposed in Chapter 2. This functionality is intrinsic to the Microsoft Kinect SDK and several skeleton extraction algorithms, and they extract the skeletons of all the subjects present in the scene, individually. We will only need to feed each set of joints to a separate LSTM network to detect any fall incident occurring to the corresponding subject. For the models proposed in Chapter 3 and Chapter 4, merging subject detection and tracking with fall detection is more challenging. A potential approach is to remove all the detected subjects, except for one, from each frame, feed the obtained single-person frames to the fall detection module, and do this for all the subjects simultaneously as long as they are present in the scene and tracked by the tracking module. Regarding the multi-person issue, an improvement can also be made to our UBC-DML Fall Dataset. We can add multi-person scenario to our dataset by capturing fall and non-fall videos with multiple main subjects present in the scene. Considering the fact that the identification of the fallen person can increase the quality of service in some applications, incorporating this functionality into the models proposed in Chapter 3 and Chapter 4 can be beneficial to such applications. We may realize this through face recognition, the prevalent approach for vision-based person identification, by employing the subject detection and tracking module explained above. When subjects are detected in a frame who have not been tracked throughout the previous frames, the face recognition is applied on them to find them matches from the small database containing the faces of the regular residents or frequent visitors of the place. 84 In addition to automatic detection of fall incidents, which enables quick assistance, predicting falls before they occur can reduce injuries by making the person alert, so that they can hold on to an object or wall for more support. To get a fall prediction system, our proposed models can be used without modification to their structures. However, they have to be re-trained by new data, designed for fall prediction. To adjust the data from fall detection datasets for this purpose, we may cut the falling and recovering parts of the videos and label all such videos as positive and the rest as negative. Window-level labeling, for the models proposed in Chapter 3 and Chapter 4, is a little more difficult. From the time the subject starts moving in an unstable way till the beginning of falling, i.e. the total loss of balance and heading towards the ground, all the windows can be labeled as positive, meaning the subject is about to fall. Additionally, in near-fall cases, where the subject is about to fall, but manages to keep balance and avoid falling, windows in the span of unstable movement start to the full regain of balance can be labeled as positive. Knowing the cause of the fall can provide better assistance to the fallen person. Hence, it is a potential enhancement to our fall detection systems, though it requires data that might be difficult to obtain. A slight modification to our proposed models is expected to be sufficient to handle this task. It is achieved by replacing the binary classification with multi-class classification through changing the number of units in the fully-connected layer from two to 𝐶, where 𝐶 is the number of causes to cover. 85 Bibliography [1] L. P. Malasinghe, N. Ramzan, and K. Dahal, “Remote Patient Monitoring: A Comprehensive Study,” Journal of Ambient Intelligence and Humanized Computing, vol. 10, no. 1, 2019, [2] N. Mozaffari, J. Rezazadeh, R. Farahbakhsh, S. Yazdani, and K. Sandrasegaran, “Practical Fall Detection Based on IoT Technologies: A Survey,” Internet of Things (Netherlands), vol. 8. 2019. [3] O. Aziz et al., “Validation of Accuracy of SVM-Based Fall Detection System Using Real-World Fall and Non-Fall Datasets,” PLoS ONE, vol. 12, no. 7, 2017, [4] H. Li, A. Shrestha, H. Heidari, J. Le Kernec, and F. Fioranelli, “A Multisensory Approach for Remote Health Monitoring of Older People,” in IEEE Journal of Electromagnetics, RF and Microwaves in Medicine and Biology, 2018, vol. 2, no. 2. [5] A. Singh, S. U. Rehman, S. Yongchareon, and P. H. J. Chong, “Sensor Technologies for Fall Detection Systems: A Review,” IEEE Sensors Journal, vol. 20, no. 13. 2020. [6] M. Mubashir, L. Shao, and L. Seed, “A Survey on Fall Detection: Principles and Approaches,” Neurocomputing, vol. 100, pp. 144–152, 2013, [7] J. Gutiérrez, V. Rodríguez, and S. Martin, “Comprehensive Review of Vision-Based Fall Detection Systems,” Sensors (Switzerland), vol. 21, no. 3. 2021. [8] L. Chen and I. Khalil, “Activity Recognition: Approaches, Practices and Trends,” in Activity Recognition in Pervasive Intelligent Environments, Atlantis Press, 2011, pp. 1–31. 86 [9] M. J. Al Nahian, M. H. Raju, Z. Tasnim, M. Mahmud, M. A. R. Ahad, and M. S. Kaiser, “Contactless Fall Detection for the Elderly,” in Intelligent Systems Reference Library, vol. 200, 2021. [10] E. Cippitelli, F. Fioranelli, E. Gambi, and S. Spinsante, “Radar and RGB-Depth Sensors for Fall Detection: A Review,” IEEE Sensors Journal, vol. 17, no. 12. 2017. [11] C. Rougier, J. Meunier, A. St-Arnaud, and J. Rousseau, “Fall Detection from Human Shape and Motion History Using Video Surveillance,” in Proceedings - 21st International Conference on Advanced Information Networking and Applications Workshops/Symposia, AINAW’07, 2007, vol. 1, pp. 875–880. [12] Z. P. Bian, J. Hou, L. P. Chau, and N. Magnenat-Thalmann, “Fall Detection Based on Body Part Tracking Using a Depth Camera,” IEEE Journal of Biomedical and Health Informatics, vol. 19, no. 2, pp. 430–439, 2015, [13] M. D. Solbach and J. K. Tsotsos, “Vision-Based Fallen Person Detection for the Elderly,” in Proceedings - 2017 IEEE International Conference on Computer Vision Workshops, ICCVW 2017, 2017, vol. 2018-Janua, pp. 1433–1442. [14] Z. A. Mundher and Z. Jiaofei, “A Real-Time Fall Detection System in Elderly Care Using Mobile Robot and Kinect Sensor,” International Journal of Materials, Mechanics and Manufacturing, vol. 2, no. 2, pp. 133–138, 2014, [15] N. Thome and S. Miguet, “A HHMM-Based Approach for Robust Fall Detection,” in 9th International Conference on Control, Automation, Robotics and Vision, 2006, pp. 1–8. 87 [16] G. Mastorakis and D. Makris, “Fall Detection System Using Kinect’s Infrared Sensor,” Journal of Real-Time Image Processing, vol. 9, no. 4, pp. 635–646, 2014, [17] M. M. Flores-Barranco, M. A. Ibarra-Mazano, and I. Cheng, “Accidental Fall Detection Based on Skeleton Joint Correlation and Activity Boundary,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015, vol. 9475, pp. 489–498. [18] E. E. Stone and M. Skubic, “Fall Detection in Homes of Older Adults Using the Microsoft Kinect,” IEEE Journal of Biomedical and Health Informatics, vol. 19, no. 1, pp. 290–301, 2015, [19] H. Nait-Charif and S. J. McKenna, “Activity Summarisation and Fall Detection in a Supportive Home Environment,” in Proceedings - International Conference on Pattern Recognition, 2004, vol. 4, pp. 323–326. [20] B. Jansen and R. Deklerck, “Context Aware Inactivity Recognition for Visual Fall Detection,” in 2006 Pervasive Health Conference and Workshops, 2006, pp. 1–4. [21] S. Wang, S. Zabir, and B. Leibe, “Lying Pose Recognition for Elderly Fall Detection,” Robotics: Science and Systems, vol. 7, pp. 345–353, 2012, [22] M. Volkhardt, F. Schneemann, and H. M. Gross, “Fallen Person Detection for Mobile Robots Using 3D Depth Data,” in Proceedings - 2013 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2013, 2013, pp. 3573–3578. [23] Y. Yun and I. Y. H. Gu, “Human Fall Detection in Videos by Fusing Statistical Features of 88 Shape and Motion Dynamics on Riemannian Manifolds,” Neurocomputing, vol. 207, 2016, [24] E. Auvinet, C. Rougier, J. Meunier, A. St-arnaud, and J. Rousseau, “Multiple Cameras Fall Dataset,” 2010. [25] B. Kwolek and M. Kepski, “Human Fall Detection on Embedded Platform Using Depth Maps and Wireless Accelerometer,” Computer Methods and Programs in Biomedicine, vol. 117, no. 3, pp. 489–501, 2014, [26] Z. Cheng, L. Qin, Y. Ye, Q. Huang, and Q. Tian, “Human Daily Action Analysis with Multi-View and Color-Depth Data,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2012, vol. 7584 LNCS, no. PART 2. [27] S. W. Yang and S. K. Lin, “Fall Detection for Multiple Pedestrians Using Depth Image Processing Technique,” Computer Methods and Programs in Biomedicine, vol. 114, no. 2, 2014, [28] C. Marzahl, P. Penndorf, I. Bruder, and M. Staemmler, “Unobtrusive Fall Detection Using 3D Images of a Gaming Console: Concept and First Results,” 2012. [29] “OpenNI 2 Downloads and Documentation | The Structure Sensor.” https://structure.io/openni [30] C. Rougier, E. Auvinet, J. Rousseau, M. Mignotte, and J. Meunier, “Fall Detection from Depth Map Video Sequences,” in International conference on smart homes and health telematics, 2011, pp. 121–128. 89 [31] R. Planinc and M. Kampel, “Introducing the Use of Depth Data for Fall Detection,” Personal and Ubiquitous Computing, vol. 17, no. 6, 2013, [32] H. Ramirez, S. A. Velastin, I. Meza, E. Fabregas, D. Makris, and G. Farias, “Fall Detection and Activity Recognition Using Human Skeleton Features,” IEEE Access, vol. 9, 2021, [33] A. Núñez-Marcos, G. Azkune, and I. Arganda-Carreras, “Vision-Based Fall Detection with Convolutional Neural Networks,” Wireless Communications and Mobile Computing, vol. 2017, 2017, [34] A. Trapet, “Fall Detection Dataset,” Last Visited, 2020. [35] J. Nogas, S. S. Khan, and A. Mihailidis, “DeepFall: Non-Invasive Fall Detection with Deep Spatio-Temporal Convolutional Autoencoders,” Journal of Healthcare Informatics Research, vol. 4, no. 1, 2020, [36] S. Vadivelu, S. Ganesan, O. V. R. Murthy, and A. Dhall, “Thermal Imaging Based Elderly Fall Detection,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017, vol. 10118 LNCS. [37] X. Ma, H. Wang, B. Xue, M. Zhou, B. Ji, and Y. Li, “Depth-Based Human Fall Detection via Shape Features and Improved Extreme Learning Machine,” IEEE Journal of Biomedical and Health Informatics, vol. 18, no. 6, pp. 1915–1922, 2014, [38] C. Ge, I. Y. H. Gu, and J. Yang, “Human Fall Detection Using Segment-Level Cnn Features and Sparse Dictionary Learning,” in IEEE International Workshop on Machine Learning for Signal Processing, MLSP, 2017, vol. 2017-Septe, pp. 1–6. 90 [39] Y. Liu, Y. Deng, C. Jia, Y. Yang, R. Wang, and C. Li, “Two-Stream Graph Convolutional Networks for 2D Skeleton-Based Fall Detection,” Isciia2020.Bit.Edu.Cn, [Online]. Available: https://isciia2020.bit.edu.cn/docs/20201114082452107377.pdf [40] M. Rahnemoonfar and H. Alkittawi, “Spatio-Temporal Convolutional Neural Network for Elderly Fall Detection in Depth Video Cameras,” in Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, 2019, pp. 2868–2873. [41] Z. Chen, Y. Wang, and W. Yang, “Video Based Fall Detection Using Human Poses,” in CCF Conference on Big Data, 2022, pp. 283–296. [42] T. H. Tsai and C. W. Hsu, “Implementation of Fall Detection System Based on 3D Skeleton for Deep Learning Technique,” IEEE Access, vol. 7, pp. 153049–153059, 2019, [43] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017, [44] N. Lu, Y. Wu, L. Feng, and J. Song, “Deep Learning for Fall Detection: Three-Dimensional CNN Combined with LSTM on Video Kinematic Data,” IEEE Journal of Biomedical and Health Informatics, vol. 23, no. 1, 2019, [45] Y. Kong, J. Huang, S. Huang, Z. Wei, and S. Wang, “Learning Spatiotemporal Representations for Human Fall Detection in Surveillance Video,” Journal of Visual Communication and Image Representation, vol. 59, 2019, [46] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 91 Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, vol. 2016-December. [47] Q. Ding et al., “RT-ADI: Fast Real-Time Video Representation for Multi-View Human Fall Detection,” 2019. [48] J. Redmon and A. Farhadi, “Yolov3: An Incremental Improvement,” arXiv preprint arXiv:1804.02767, 2018. [49] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv preprint arXiv:1409.1556, 2014. [50] A. Shahroudy, J. Liu, T. T. Ng, and G. Wang, “NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, vol. 2016-Decem, pp. 1010–1019. [51] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning Representations by Back-Propagating Errors,” Nature, vol. 323, no. 6088, 1986, [52] C. B. Lin, Z. Dong, W. K. Kuan, and Y. F. Huang, “A Framework for Fall Detection Based on Openpose Skeleton and Lstm/Gru Models,” Applied Sciences (Switzerland), vol. 11, no. 1, 2021, [53] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826. [54] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted 92 Residuals and Linear Bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510–4520. [55] Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” 2009, pp. 248–255. [56] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980, 2014. [57] N. Radwan, “Leveraging Sparse and Dense Features for Reliable State Estimation in Urban Environments,” University of Freiburg, Freiburg im Breisgau, Germany, 2019. [Online]. Available: http://ais.informatik.uni-freiburg.de/publications/papers/radwan19phd.pdf [58] A. Vaswani et al., “Attention Is All You Need,” 2017. [59] J. L. Ba, J. R. Kiros, and G. E. Hinton, “(LN) Layer Norm,” arXiv:1607.06450v1, 2015. [60] D. Hendrycks and K. Gimpel, “Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units,” arXiv, 2016. [61] S. Gasparrini et al., “Proposal and Experimental Evaluation of Fall Detection Solution Based on Wearable and Depth Data Fusion,” in Advances in Intelligent Systems and Computing, 2016, vol. 399.
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Nasiopoulos, Panos
Little, James Joseph, 1951-
Video-based human fall detection in indoor spaces for health monitoring
Text