Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Jiao, Yutong
Master of Applied Science - MASc
University of British Columbia
Closed-circuit television (CCTV) is being widely adopted in water pipeline inspection. The inspector needs to spend a long time to watch the recorded video during the office-based survey and can get fatigue easily. An automated process can release the inspector's workload and ensure the consistent quality of the survey. However, a fully automated survey of varied structural discontinuities still remains as a challenge. This study aims to first identify the key frames of the CCTV video, which contain the major anomalies captured from the internal surface of the pipe. Thus, the inspector can focus more on these key frames. First, the key-frame identification task is performed by using an unsupervised learning-based framework with state-of-the-art autoencoders. Three popular autoencoders were implemented in this framework, and results were collected accordingly. The experimental results illustrate that this framework achieves up to 0.940 accuracy with a relatively lower precision metric of 0.901. Secondly, a key-frame identification framework based on the steerable pyramid autoencoder (SPAE) is proposed in order to improve the accuracy of this task. Both the parameter optimization and comparative studies for the proposed SPAE were carried out in this research. Explicitly, the SPAE develops good capability of representation learning and extraction. The experimental results demonstrate that the SPAE-based approach can achieve 0.984 accuracy, which outperforms the other experimental methods selected for comparison. Thirdly, a log-Gabor autoencoder (LGAE) based framework is proposed to further enhance the performance on key-frame identification in terms of representation learning. It can be regarded as an improved version of the SPAE-based approach. Comparative studies were conducted, and the results show that this novel LGAE takes advantage of the feature detection over the other methods with the accuracy of 0.988 and the recall metric of 0.996. Additionally, the LGAE has a superior performance for the generalization on varied datasets. Hence, LGAE-based framework will significantly assist the office-based survey, which will highly facilitate the pipeline condition assessment through the CCTV inspection.
AUTOENCODER-BASEDKEY-FRAME IDENTIFICATION FORWATER PIPELINE CCTVINSPECTIONbyYutong JiaoB.Eng., Beijing University of Chemical Technology, 2018A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Electrical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)February 2021c© Yutong Jiao, 2021The following individuals certify that they have read, and recommend to the College ofGraduate Studies for acceptance, a thesis entitled:AUTOENCODER-BASED KEY-FRAME IDENTIFICATION FOR WATER PIPELINE CCTVINSPECTIONsubmitted by Yutong Jiao in partial fulfilment of the requirements of the degree of Master ofApplied ScienceDr. Zheng Liu, School of EngineeringSupervisorDr. Kenneth Chau, School of EngineeringSupervisory Committee MemberDr. Yang Cao, School of EngineeringSupervisory Committee MemberDr. Joshua Brinkerhoff, School of EngineeringUniversity ExamineriiAbstractClosed-circuit television (CCTV) is being widely adopted in water pipeline inspection.The inspector needs to spend a long time to watch the recorded video during the office-basedsurvey and can get fatigue easily. An automated process can release the inspector’s workloadand ensure the consistent quality of the survey. However, a fully automated survey of variedstructural discontinuities still remains as a challenge. This study aims to first identify the keyframes of the CCTV video, which contain the major anomalies captured from the internalsurface of the pipe. Thus, the inspector can focus more on these key frames.First, the key-frame identification task is performed by using an unsupervised learning-based framework with state-of-the-art autoencoders. Three popular autoencoders were imple-mented in this framework, and results were collected accordingly. The experimental resultsillustrate that this framework achieves up to 0.940 accuracy with a relatively lower precisionmetric of 0.901.Secondly, a key-frame identification framework based on the steerable pyramid autoen-coder (SPAE) is proposed in order to improve the accuracy of this task. Both the parameteroptimization and comparative studies for the proposed SPAE were carried out in this research.Explicitly, the SPAE develops good capability of representation learning and extraction. Theexperimental results demonstrate that the SPAE-based approach can achieve 0.984 accuracy,which outperforms the other experimental methods selected for comparison.Thirdly, a log-Gabor autoencoder (LGAE) based framework is proposed to further enhancethe performance on key-frame identification in terms of representation learning. It can beregarded as an improved version of the SPAE-based approach. Comparative studies were con-ducted, and the results show that this novel LGAE takes advantage of the feature detection overthe other methods with the accuracy of 0.988 and the recall metric of 0.996. Additionally, theLGAE has a superior performance for the generalization on varied datasets. Hence, LGAE-based framework will significantly assist the office-based survey, which will highly facilitatethe pipeline condition assessment through the CCTV inspection.iiiLay SummaryClosed-circuit television (CCTV) is being widely adopted in water pipeline inspection. Todate, companies still assign inspectors to perform office-based survey on the recorded videos,which show the internal surface of the water pipeline. The inspectors need to spend a long timechecking the videos manually, which is not only time-consuming but also strenuous. Also,the inspectors get fatigue easily. An automated process can release the inspectors’ workloadand ensure the consistent quality of the survey. However, a fully automated survey of variedstructural deficiencies still remains as a challenge. In this thesis, two key-frame identificationframeworks are carried out based on the proposed steerable pyramid autoencoder (SPAE) andlog-Gabor autoencoder (LGAE) to address this challenge. Thus, the inspector can focus moreon these key frames. This will highly facilitate the pipeline condition assessment through theCCTV inspection. Experimental results demonstrate the effectiveness of the proposed meth-ods.ivPrefaceThis thesis is based on the research work conducted in the School of Engineering at TheUniversity of British Columbia, Okanagan Campus, under the supervision of Prof. Zheng Liu.Published work is contained in this thesis.The following published paper is contained in the thesis and used with permission of Else-vier.• Yutong Jiao, Rakiba Rayhana, Junchi Bin, Zheng Liu, Angie Wu, and Xiangjie Kong.A Steerable Pyramid Autoencoder based Framework for Anomaly Frame Detection ofWater Pipeline CCTV Inspection. Measurement, vol. 174, p. 109020 (1-11), Jan. 2021.The author has contributed to the following published paper, which is used with permissionof IEEE.• Rakiba Rayhana, Yutong Jiao, Amirhossein Zaji, and Zheng Liu. Automated visionsystems for condition assessment of sewer and water pipelines. IEEE Transactions onAutomation Science and Engineering, p. 1-18, 2020.The author has contributed to the following published paper, which is used with permissionof SPIE.• Rakiba Rayhana, Yutong Jiao, Zheng Liu, Angie Wu, and Xiangjie Kong. Water pipevalve detection by using deep neural networks. In Smart Structures and NDE for Industry4.0, Smart Cities, and Energy Systems, SPIE, 2020, vol. 11382, p. 20-27.I am the principal contributor for the work in this thesis. Prof. Zheng Liu provided me withsome suggestions to improve my research work. Besides, Mrs. Angie Wu, the senior managerat Pure Technologies, helped with the acquisition of the data used in the thesis.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2.1 CCTV-based Condition Assessment of Water Pipeline . . . . . . . . . . 31.2.2 CCTV Video Analytics Methodologies . . . . . . . . . . . . . . . . . . 41.3 Research Challenges and Objectives . . . . . . . . . . . . . . . . . . . . . . . 51.4 Thesis Outline and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . 6Chapter 2: Autoencoder Techniques: The State of the Art . . . . . . . . . . . . . . 82.1 Autoencoder Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Convolutional Autoencoder Network . . . . . . . . . . . . . . . . . . . . . . . 102.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Chapter 3: Autoencoder-based Key-Frame Identification Frameworks and Im-plementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14viTABLE OF CONTENTS3.2 The Unsupervised Learning-based Framework . . . . . . . . . . . . . . . . . . 153.2.1 Framework Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2.2 Computational Methodologies . . . . . . . . . . . . . . . . . . . . . . 153.3 The Steerable Pyramid Autoencoder based Framework . . . . . . . . . . . . . 193.3.1 Framework Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.3.2 Computational Methodologies . . . . . . . . . . . . . . . . . . . . . . 203.4 The Log-Gabor Autoencoder based Framework . . . . . . . . . . . . . . . . . 293.4.1 Framework Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.4.2 Computational Methodologies . . . . . . . . . . . . . . . . . . . . . . 303.5 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Chapter 4: Experimental Results and Discussions . . . . . . . . . . . . . . . . . . 394.1 Data Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.3 Results and Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.3.1 The Unsupervised Learning-based Framework . . . . . . . . . . . . . . 404.3.2 The Steerable Pyramid Autoencoder based Framework . . . . . . . . . 454.3.3 The Log-Gabor Autoencoder based Framework . . . . . . . . . . . . . 574.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Chapter 5: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66viiList of TablesTable 3.1 The contingency table . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Table 4.1 Evaluation of three autoencoders . . . . . . . . . . . . . . . . . . . . . 44Table 4.2 Parameter optimization of the hidden layers . . . . . . . . . . . . . . . 46Table 4.3 Parameter optimization of the steerable orientations . . . . . . . . . . . 46Table 4.4 The comparison of the accuracy and recall from five-fold cross-validation 48Table 4.5 The comparison of the precision and F1-score from five-fold cross-validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49Table 4.6 Evaluation of key-frame identification on all the experimental methods . 51Table 4.7 The evaluation metrics from five-fold cross-validation on SPAE andLGAE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58Table 4.8 Computational costs of three frameworks on a five-minute video . . . . 61Table 4.9 Evaluation of key-frame identification between SPAE and LGAE ontwo varied datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62viiiList of FiguresFigure 1.1 The free-swimming robotic platform (PipeDiver R©). . . . . . . . . . . 1Figure 1.2 Some illustrations of the data obtained from CCTV inspection videos. . 2Figure 2.1 The schematic diagram of an autoencoder network. . . . . . . . . . . . 9Figure 2.2 The schematic structure of a convolutional neural network. . . . . . . . 11Figure 2.3 The illustration of video analysis using convolutional autoencoder. . . . 12Figure 3.1 The research overview of the water pipeline CCTV inspection. . . . . . 15Figure 3.2 The unsupervised learning-based framework with state-of-the-art au-toencoders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Figure 3.3 The schematic structure of an autoencoder. . . . . . . . . . . . . . . . 16Figure 3.4 The schematic structure of a convolutional autoencoder. . . . . . . . . 17Figure 3.5 The schematic structure of a variational autoencoder. . . . . . . . . . . 18Figure 3.6 The SPAE-based framework for key-frame identification. . . . . . . . 19Figure 3.7 The structure of a basic autoencoder network. . . . . . . . . . . . . . . 21Figure 3.8 System diagram of the steerable pyramid, illustrating the filtering andsampling operations, and the recursive subsystem in the dotted box.”2D” and ”2U” correspond to downsampling and upsampling by a fac-tor of 2. The circles correspond to the transform coefficients. . . . . . . 22Figure 3.9 The steerable pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . 23Figure 3.10 The overall structure of the steerable pyramid autoencoder based frame-work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Figure 3.11 The building block illustrating the lateral connection for the fusion ofsteerable pyramid in the SPAE. . . . . . . . . . . . . . . . . . . . . . 26Figure 3.12 The schematic structure of a nonlinear support vector machine. . . . . 27Figure 3.13 The LGAE-based framework for key-frame identification. . . . . . . . 29Figure 3.14 Sample video frame and the golden pyramid of log-Gabor filter on thesample frame: (a) sample video frame; (b) the golden pyramid of thelog-Gabor filter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Figure 3.15 The overall structure of the log-Gabor autoencoder based framework. . 33ixLIST OF FIGURESFigure 3.16 The model construction of the proposed LGAE. . . . . . . . . . . . . . 34Figure 3.17 The diagram for k-fold cross-validation. . . . . . . . . . . . . . . . . . 37Figure 4.1 The training curves for the loss of the experimental models. . . . . . . 41Figure 4.2 The reconstruction loss of the testing samples. . . . . . . . . . . . . . 42Figure 4.3 Loss distribution of the autoencoder. . . . . . . . . . . . . . . . . . . . 42Figure 4.4 Loss distribution of the convolutional autoencoder. . . . . . . . . . . . 43Figure 4.5 Loss distribution of the variational autoencoder. . . . . . . . . . . . . . 43Figure 4.6 The bar chart of the accuracy in the cross-validation of SPAE-basedframework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Figure 4.7 The training history of SPAE. . . . . . . . . . . . . . . . . . . . . . . 47Figure 4.8 The comparison of the extracted features between the selected experi-mental methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51Figure 4.9 Illustration of the comparative results. . . . . . . . . . . . . . . . . . . 52Figure 4.10 ROC curves of the deep learning methods in the study. . . . . . . . . . 53Figure 4.11 Precision-recall curves of the deep learning methods in the study. . . . 54Figure 4.12 3-dimensional t-SNE feature distribution based on: (a) SPAE; (b) AE;(c) CAE; (d) VAE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Figure 4.13 Temporal regularity of the CCTV inspection video. . . . . . . . . . . . 57Figure 4.14 The comparison of the evaluation metrics between SPAE and LGAE. . 58Figure 4.15 The hidden representations from SPAE and LGAE compared with rawframes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Figure 4.16 ROC curves of the deep learning methods in the study. . . . . . . . . . 60Figure 4.17 Precision-recall curves of the deep learning methods in the study. . . . 61xGlossaryCCTV Closed-Circuit Television. 1DT Destructive Testing. 3NDT Nondestructive Testing. 3DL Deep Learning. 4SED Sobel Edge Detection. 4CED Canny Edge Detection. 4HOG Histogram of Oriented Gradients. 4R-CNN Region based Convolutional Neural Networks. 4CNN Convolutional Neural Network. 8AE Autoencoder. 5CAE Convolutional Autoencoder. 5VAE Variational Autoencoder. 5SP Steerable Pyramid. 21ML Machine Learning. 8ANN Artificial Neural Network. 8NN Neural Network. 8MLP Multi-Layer Perception. 9NLP Natural Language Processing. 10SVM Support Vector Machine. 19RBF Radial Basis Function. 27ROC Receiver Operating Characteristic. 36MSE Mean Squared Error. 23TP True Positive. 35FP False Positive. 35TN True Negative. 35FN False Negative. 35TPR True Positive Rate. 36FPR False Positive Rate. 36xiAcknowledgementsFirst of all, I would like to present immense gratitude to my supervisor, Professor ZhengLiu, whose guidance and encouragement throughout my graduate study have been indispens-able. I have attained valuable research skills and technical writing skills from his guidance.Additionally, I would also like to thank him for helping with the opportunity to work withindustries, which has brought me precious working experience. I have developed professionalattitude and skills in my graduate study with the help of my supervisor.I would also like to thank Dr. Kenneth Chau and Dr. Yang Cao for their willingness to serveon the advisory committee. I really appreciate their valuable time and constructive commentson my thesis.In addition, I would like to give my appreciation to all of my colleagues at the IntelligentSensing, Diagnostic and Prognostic Research Lab (ISDPRL), who have encouraged me andtried to offer help with my research. Especially, I would thank Miss. Rakiba Rayhana for hersupport and constant encouragement during the past two years.Finally, I owe my parents the greatest gratitude for their understanding, mental support,patience and love over all the past years. I could not possibly achieve the research work withouttheir continuous encouragement and support, especially their kindness and care to me. I wishthe happiest and healthiest life to my parents.xiiChapter 1Introduction1.1 Background and MotivationWater pipeline plays a crucial role in human life for providing a stable water supply. How-ever, various defects are observed in the water pipelines such as water leakage, blockage, airpocket, debris, fracture, and so on [1, 2]. As a critical urban infrastructure, any damage canlead to severe distress of both the water mains and water quality [3]. These disruptions anddamages caused by pipeline events are costly to repair [4]. For instance, one large-diameterfailure may cost approximately 1.5 million dollars. Hence, it is essential to perform in-timecondition assessment and maintenance of the water pipelines to avoid any massive disaster.Condition assessment commonly represents the analysis of the condition of the infrastructurewith regard to many aspects, e.g., materials, age, deficiencies, and construction methods [5–7].Specifically, condition assessment plays a vital role in surveillance and security applications,which allows robots or human operators to respond in time and take actions as needed to en-sure the regular function of the equipment or platforms. With condition assessment, essentialinformation can be provided to evaluate the condition of an infrastructure [8]. To date, multi-ple technologies have been performed for condition assessment of water mains. Closed-circuittelevision (CCTV) is being widely adopted to provide visual inspection without man-entry inwater pipeline condition assessment.Figure 1.1: The free-swimming robotic platform (PipeDiver R©).In this study, PipeDiver R© [9] was employed as a platform to collect the inspection data.PipeDiver R© is a water pipeline condition assessment tool with free-swimming capacity in a11.1. Background and Motivationlong distance, which can operate while the water transmission and distribution system re-mains in service. As a result, it provides utility owners with an easier and less costly alter-native to inspection methods that require shutdown or dewatering of the system. This roboticplatform is developed by Pure Technologies, whose snapshot is presented in Figure 1.1. Ascan be seen, it is featured with collapsible pedals, which help passing through sharp bends,diameter-reduction, tees, and butterfly valves in the water pipelines. Besides, the robotic plat-form is equipped with a lighting array and multiple cameras to conduct inspection through thepipelines. It has high flexibility due to its deployment of the appurtenances, which allow it totravel through varied configurations of the pipes during the inspection. The benefits of this toolinclude providing actionable pipe wall data, no human disruption during the inspection, andlong inspection distances. Specifically, several cameras are attached to this tool from differentangles to conduct CCTV inspection and collect video data. Some illustrations of the data ob-tained from the CCTV videos are presented in Fig. 1.2. The first row and second row presentthe video frames obtained from the inspection videos of two different pipelines. The data pre-sented on the first row shows a worse illumination setting, which results in the more difficultvisual observation of the defects in the water pipes. As circled on the key frames, there arecertain longitudinal cracking, offtake, air pocket, and debris observed on the internal surfaceof the water pipes.Figure 1.2: Some illustrations of the data obtained from CCTV inspection videos.The inspection by PipeDiver R© through traveling along the pipelines creates a large numberof inspection videos. To date, companies still assign inspectors to check each video frameto identify anomalies by watching the whole recorded videos during the office-based survey,which is tedious, strenuous, inefficient, and error-prone. Sometimes, the manual inspection isnot accurate due to the operators’ fatigue. Above all, it is said that the inspectors can spendeight times as long as the video length to finish the manual labeling work, which means itis hard to realize water pipeline maintenance in time. Hence, it is necessary and critical to21.2. Literature Reviewperform automated key-frame identification to first identify the key frames of the videos forcondition assessment instead of spending hours observing the videos. An automated processcan release the inspectors’ workload and ensure the consistent quality of the survey. However,a fully automated survey of varied structural discontinuities still remains as a challenge toincrease the accuracy and efficiency of water pipeline inspection on CCTV videos. Advanceddata analytic capabilities are needed for inspection and interpretation of the system operationto empower the PipeDiver R© platform.1.2 Literature ReviewBased on the background of the study, water pipeline condition assessment and the existingtechniques are reviewed. Primarily, CCTV-based inspection of the water pipe is investigatedin this chapter. Besides, CCTV video analytics methodologies are introduced with potentialsolutions to enhance the water pipeline inspection.1.2.1 CCTV-based Condition Assessment of Water PipelineCondition assessment is necessary for water distribution and water transmission. Specifi-cally, it is essential in surveillance and security applications to ensure the inspection and inter-pretation of the system operation. This enables human inspectors to respond and take actionsin time to keep the consistent function of the equipment or platforms. Multiple technologieshave been carried out for condition assessment of the water pipeline. To date, destructive test-ing (DT) and non-destructive testing (NDT) are two types of techniques conducted in the waterpipeline inspection [10]. Destructive testing entails removal of a sample from a pipe to performanalysis. However, NDT offers a safe and reliable way of inspection, which is cost-effectiveand requires little disruption to the water system. NDT covers direct visual observation ofdefects as well as techniques that provide signals or signatures interpreted into distress indica-tors [10]. For example, the pressure in different nodes and the flow in different pipes have beenmeasured to estimate leakage and its severity in the water distribution system[1]. Pit depthmeasurement is one of the NDT techniques carried out when the pipe is exposed, which is in-convenient to some extent. Other technologies include ultrasonic testing, radiographic testing,and thermographic testing [10, 11]. In addition, advances in sensor technologies give rise tothe growing use of sensor networks in pipe inspection. Till-date, studies implemented differentanalysis and prediction techniques based on sensory data. Conditional monitoring across dif-ferent fields has been widely investigated through multi-sensor data analysis [12]. For instance,wireless sensor network is applied for condition assessment of water-flood [13]. Leakage or31.2. Literature Reviewblockage detection was performed through wireless sensor networks for water distribution sys-tem [14, 15]. Computational intelligent methods have also been applied to process sensory datafor water transmission and distribution system [16]. Furthermore, visual inspection, electro-magnetic inspection, and acoustic inspection are three other NDT techniques for water pipelinecondition assessment [10].As for visual inspection, the condition of the internal surfaces of the pipe can be assessedby a variety of vision aids. CCTV is one of the widely-adopted equipment to visualize theinternal conditions of the water or sewer pipelines recently [10, 17]. CCTV inspection providesa direct visual representation of defects in the pipe, which highly facilitates the inspectorsto understand and assess the pipe condition accurately [8]. This shows an advantage overother techniques; hence it is gaining attention from researchers. To date, the majority of thestudies for CCTV inspection are based on the sewer pipeline. As a daily infrastructure, sewerpipe commonly has a low level of sewer water, which means the pipe is not full of sewerwater. On the contrary, the water pipeline is usually in the condition of water transmission,namely, full of the water being transmitted. As introduced above, the video data acquired fromthe water pipeline is underwater data in this study, since the PipeDiver R© for data collectiondoes not require dewatering of the system in service. However, the data for sewer and waterpipeline have similarities on the in-pipe condition to some extent. Hence, related techniquesfor the sewer pipeline are also reviewed and referred to in the study. On the basis of CCTVinspection videos, studies implemented image processing methods on sewer pipeline, suchas Sobel edge detection (SED), canny edge detection (CED) [18] and histogram of orientedgradients (HOG) [19]. Moreover, a recent study by Yin et al. shows a deep learning-basedframework for inspection on sewer pipes by using faster region based convolutional neuralnetworks (R-CNN) [20]. However, few studies conduct water pipeline inspection using state-of-the-art deep learning (DL) methods, especially on CCTV videos. The objective of the thesisis to conduct key-frame identification, which is to detect key frames containing pipe anomalies(also known as anomaly frames) in the inspection videos. For instance, a study presents akey-frame extraction method for greenhouse monitoring video [21].1.2.2 CCTV Video Analytics MethodologiesThe last few years have seen an increased interest in CCTV video analytics, which hasreceived a lot of research attention. CCTV video analytics can be addressed through imageprocessing methodologies since each video frame can be regarded as an image. Conventionaltechniques include multiple detection methodologies such as Sobel edge detection (SED) andcanny edge detection (CED), which have been implemented on sewer pipe video data [18]. In41.3. Research Challenges and Objectivesthe past few years, deep learning techniques have risen up, especially in the domain of imageand video analytics. Deep learning methods have been well-adopted for various automatedtasks, which were proved to be efficient and accurate. For instance, CNN is a well-knownsupervised deep learning model for image classification and detection, which can be imple-mented on video frames [22, 23]. R-CNN is a type of deep learning model derived from CNN,which is normally applied for object detection on video data [24]. Faster R-CNN is then animproved version of R-CNN [20, 25]. Moreover, autoencoder (AE) is one of the state-of-the-art unsupervised learning methods in the pattern recognition domain. The encoder structureof an autoencoder is widely applied for image representation learning and extraction, whichcan then be used to various prediction algorithms. Autoencoder was first proposed in thepaper by Kramer [22]. Other autoencoder-based models include convolutional autoencoder(CAE) [26, 27], variational autoencoder (VAE) [28, 29], multimodal autoencoder [30], adver-sarial autoencoder [31], deep laplacian autoencoder [32] and so on. Another deep learningmethod called appearance and motion DeepNet was proposed for feature detection on surveil-lance videos in 2015 [33]. Besides, Chong and Tay carried out a spatio-temporal autoencoderfor event detection of videos [34]. These methods were proved to have good performance,especially for anomaly detection on surveillance videos. However, few researchers have im-plemented these methods for water pipeline CCTV inspection so far.1.3 Research Challenges and ObjectivesFrom the perspective of water pipeline inspection, a fully automated office-based survey ofvaried structural deficiencies still remains as a challenge. Also, the majority of the studies arebased on sensor technologies for the condition assessment of the water pipes. Few researcheshave carried out CCTV-based water pipeline inspection with a direct visual representation ofdefects in the pipe. For the office-based survey, it is important to first identify the probablekey frames of the CCTV videos, which contain the major anomalies captured from the internalsurface of the pipe. Thus, the inspector can focus more on these key frames to perform multipletasks. This will significantly facilitate the office-based survey to help save time and labor.On the other hand, few researchers have implemented deep learning methods for waterpipeline CCTV inspection till date. Hard mining on the underwater video data needs to beempowered through advanced deep learning techniques. The methodologies for performingthe key-frame identification can be further developed to adapt to the water pipeline datasetand achieve better performance in this study. To be more specific, accurate feature detection,namely, representation learning, is challenging, which has been extensively researched. Theinput image must be transformed into a feature representation, which is easier for an algorithm51.4. Thesis Outline and Contributionsto perform classification or detection tasks. Hence, it is crucial to develop a better algorithm toobtain outstanding capacity of feature extraction, especially on the specific in-pipe video datain the underwater condition in this study. As a result, a good set of representative features canbe produced, which makes it easier for the classifier to perform the detection task on waterpipeline video frames in the thesis.Therefore, the objectives of the thesis include the application of a methodology to addresskey-frame identification for water pipeline CCTV inspection. This is to identify the key framesof the CCTV video, which contain the major anomalies captured from the internal surface ofthe water pipe. In addition, the methodology needs to be enhanced to adapt to the specificin-pipe data in the underwater condition. The hard mining capability of the methodology is tobe developed to deeply exploit information on the video data.1.4 Thesis Outline and ContributionsThe thesis is organized into five chapters.Chapter 1 presents the background of water pipeline condition assessment and the currentsolutions to enhance the inspection. In addition, this chapter provides a literature review forthe water pipeline condition assessment and CCTV video analytics. Furthermore, the researchchallenges and objectives are also stated in this chapter.Chapter 2 investigates the state-of-the-art autoencoder-based techniques relevant to the the-sis. First, an overview of machine learning is presented. Secondly, the development historyof autoencoder-based techniques is introduced. Finally, the state-of-the-art autoencoder-basedtechnique, namely, convolutional autoencoder, is investigated, which is one of the most essen-tial techniques in this thesis.Chapter 3 introduces three autoencoder-based frameworks and their implementations on thetask of key-frame identification. First, an unsupervised learning-based framework is carriedout with state-of-the-art autoencoders. Secondly, a steerable pyramid autoencoder (SPAE)based framework is proposed for the purpose of enhancing accuracy. Thirdly, a log-Gaborautoencoder (LGAE) based framework is proposed, which is a further improved version of theSPAE-based framework.In Chapter 4, the experimental results and discussions are demonstrated regarding threeautoencoder-based frameworks conducted on key-frame identification. First, the unsupervisedlearning-based framework achieves up to 0.940 accuracy on the task with a lower precisionmetric of 0.901. Secondly, the SPAE-based framework shows improved performance withan averaged accuracy of 0.984 on the water pipeline video data in specific underwater con-dition. To the authors’ best knowledge, this is the first study to integrate steerable pyramid61.4. Thesis Outline and Contributionswith an autoencoder-based network, resulting in outperforming capability of representationlearning and extraction. Finally, the LGAE-based framework achieves the accuracy of 0.988and the recall metric of 0.996, which outperforms the other selected experimental methods.Explicitly, the superiority of LGAE comes from its compelling performance on feature gener-ation and generalization capacity. Hence, LGAE-based framework will significantly assist theoffice-based survey on water pipeline CCTV inspection. The inspectors can focus on the mostimportant information and conduct pipe assessment as well as maintenance effectively to avoidsevere mistakes.Chapter 5 concludes the thesis. Future work is also suggested.7Chapter 2Autoencoder Techniques: The State of theArtAutoencoder-based networks have found their applications on many complicated problems,especially for image or video analysis. They are proved to have good performance on featuredetection, image classification, and anomaly detection on surveillance videos [33, 34]. Hence,autoencoder techniques are selected as the research methodologies to be further developed andimplemented on the key-frame identification. The autoencoder techniques are investigated inthis chapter in terms of their development history and fundamentals. Additionally, one subsetof autoencoder networks, i.e., convolutional autoencoder network, is investigated since it is thebasic approach based on which the proposed methodology is developed.Machine learning (ML) technology evolves rapidly and powers many applications in life.For decades, it has been widely applied in a variety of domains such as computer vision, mar-keting, economics, and so on. Machine learning is a field of study on computer science, whichis to apply computer algorithms on the data to discover patterns of interest. It is seen as a subsetof artificial intelligence. Specifically, ML algorithms build a mathematical model based on aset of data in order to make predictions or decisions. It is achieved without the model being ex-plicitly programmed to do so. The data is also known as “training data”. The model improvesautomatically through experience on the training data. Artificial neural network (ANN), whichis commonly called neural network (NN), is a group of computing systems inspired by the bio-logical neural networks that constitute animal brains [35]. It is extensively adopted in machinelearning, which models the data through graphs of artificial neurons. Each neuron in an ANNworks approximately in the way how a neuron works in a biological brain. To be more specific,each neuron can receive, process and transmit information in the ANN. The output of a neu-ron is computed by a specified function on the sum of its inputs. Explicitly, an ANN containsmultiple layers with a certain number of neurons in each layer. Different layers could performvaried transformations on their inputs. Supervised learning, unsupervised learning, and rein-forcement learning are three major types of learning paradigms of ANN. Convolutional neuralnetwork (CNN) is a typical supervised learning model, while autoencoder is a typical unsuper-vised learning model. An autoencoder is a type of artificial neural network, which is performed82.1. Autoencoder Networkto learn efficient data codings in an unsupervised manner [22]. It is widely adopted in manydomains and tasks, including dimensionality reduction, anomaly detection, image processing,etc. With the development of deep learning family in the machine learning domain, autoen-coder can be increasingly deeper with more hidden layers in the feature learning stage of themodel. Deep autoencoder networks achieve good performance on multiple tasks.2.1 Autoencoder NetworkAn autoencoder is a type of artificial neural network, which is performed to learn efficientdata codings in an unsupervised manner [22]. The aim of an autoencoder is to learn a represen-tation for a set of data for the purpose of performing specific tasks. An autoencoder consists ofan encoder and a decoder, where the encoder generates a reduced encoding, and the decoderreconstructs an output from the encoding as close as possible to the raw input. The schematicdiagram of an autoencoder is illustrated in Fig. 2.1. More specifically, an autoencoder is trainedto encode the input data into a certain representation in the encoder, which is then decoded backto the original format of the input data. Essentially, the autoencoder attempts to optimize itshyper-parameters to improve the reconstruction outcome in the training process. Thus, the au-toencoder is able to learn and extract useful representations of the input continuously, as wellas tries to discard the useless information during the propagation. It is the major strength ofan autoencoder network. In addition, the efficiency of the training procedure for an autoen-coder can be relatively higher since a high dimensional input vector can be converted into alower-dimensional representation in the encoder.Figure 2.1: The schematic diagram of an autoencoder network.During the past few decades, the autoencoder has been rapidly developed among researchesregarding the ANN. In 1988, it is found out by Bourlard and Kamp [36] that a multi-layerperception (MLP) in auto-association mode could achieve dimensionality reduction and data92.2. Convolutional Autoencoder Networkcompression in the domains like information processing [37]. To date, different types of clas-sical autoencoders have been proposed for varied purposes. Sparse autoencoder was carriedout with the aim of acquiring simple interpretation of the raw input in 2006 [38]. Vincent cameup with denoising autoencoder with enhanced robustness in 2008 [39, 40]. Contractive au-toencoder was then proposed by Rifai et al. for the purpose of reducing the feature sensitivitywith regard to the raw input in 2011 [41]. Recently, some more variations of the autoencoderevolve from the basic model for the purpose of enhancing the representations of the raw inputin the intermediate layers, known as latent space of the autoencoder. VAE is one of the state-of-the-art autoencoders derived from the basic autoencoder [42, 43]. VAE shows superiority inrepresenting data in a compact probabilistic latent space, which has been widely developed inthe domains, including language modeling and image generation [44].2.2 Convolutional Autoencoder NetworkDeep learning is a subset of machine learning based on artificial neural network with rep-resentation learning. For decades, both shallow and deep learning of artificial neural networkshave been researched in many fields [45, 46]. More precisely, deep learning algorithms applymultiple layers to learn and extract useful features progressively from the raw input [47]. Forinstance, the early layers can learn and extract lower semantic features from the input, whilethe deeper layers develop complex features of the input. The word “deep” indicates the largenumber of layers used in the deep learning model to conduct the task. An autoencoder can alsobe deeper with more hidden layers in the feature learning stage of the model. Deep autoen-coder networks show higher potential for achieving good performance in multiple tasks. Withthe development of deep learning technique, convolutional neural network evolves rapidly indifferent domains. Convolutional neural network is a subset of deep neural networks in deeplearning, which is most commonly conducted to explore pattern information. CNN is prefer-ably adopted in the domains such as image processing, image classification, natural languageprocessing (NLP), and so on. Meanwhile, the layered structure of CNN is extensively com-bined with other structures to form new models with the aim of performing different tasks. Theschematic structure of a convolutional neural network is displayed in Fig. 2.2. A convolutionalneural network is composed of an input layer, multiple hidden layers, as well as an output layer.As shown in Fig. 2.2, the hidden layers in a CNN typically consist of convolutional layers, sub-sampling layers, and fully-connected layers. Feature maps can be obtained after each hiddenlayer, which is presented in the schematic structure.Specifically, convolutional layers are regularized versions of multi-layer perceptions, whichare also known as fully-connected layers. For fully connected networks, each neuron in one102.2.ConvolutionalAutoencoderNetworkFigure 2.2: The schematic structure of a convolutional neural network.112.2.ConvolutionalAutoencoderNetworkFigure 2.3: The illustration of video analysis using convolutional autoencoder.122.3. Summaryspecific layer is connected to all the neurons in the next layer. It means that these networksare prone to have overfitting problem on certain datasets. For the purpose of regularization,convolutional neural network takes advantage of the hierarchical patterns of the data to assem-ble complex patterns. Thus, CNN is less prone to be extremely constrained on the data. Dueto the superiority of convolutional structure, it is widely adopted in other networks to achievebetter performance on certain tasks. Convolutional autoencoder was proposed then, which isto implement convolutional layers in the form of an autoencoder. CAE takes advantage of thestrength of convolutional layers, thus showing potential in multiple tasks. Also, a convolutionalautoencoder can have multiple hidden layers with a deep structure as needed for performing aspecific task on the dataset.To be more precise, the illustration of video analysis using convolutional autoencoder ispresented in the Fig. 2.3. As illustrated in Fig. 2.3, a convolutional autoencoder is composed ofconvolutional (Conv) layers, pooling layers, and upsampling layers. Each convolutional layeris followed by a pooling layer in the encoding part of the CAE. On the other side, each con-volutional layer is constructed with an upsampling layer in the decoder. However, the poolinglayer can also be replaced by the stride function of a convolutional layer. For instance, if thestride parameter of a convolutional layer is defined as 2, the features output from this layerwill be downsampled by a factor of 2. Explicitly, the stride in the convolutional layer is ableto change the types of features extracted from the layer, whereas the pooling layer is simplychanging how much the data is downsampled. Above all, the convolutional autoencoder net-work has advantages for image or video analysis because it is able to learn the pattern of dataaccurately through the encoder-decoder structure. Explicitly, the model can extract represen-tative features from the images, which can then be applied to an algorithm for prediction. Therepresentative features detected are compressed and more separable, which will benefit variedapplication tasks.2.3 SummaryIn this chapter, the basic principles and the state-of-the-art techniques regarding the au-toencoder network are introduced. Due to the superiority of the autoencoder-based network infeature learning and extraction for image or video analysis, it is adopted in this research to en-hance the key-frame identification of water pipeline CCTV videos. Since the research majorlyinvolves the development based on state-of-the-art CAE, the origins and principles of CAE areexplicitly introduced in this chapter.13Chapter 3Autoencoder-based Key-FrameIdentification Frameworks andImplementations3.1 IntroductionAs introduced in the Section 1, the aim of the study is to perform key-frame identifica-tion on the water pipeline CCTV inspection data. Fig. 3.1 displays the research overview ofthe water pipeline CCTV inspection. First, CCTV data are acquired from the free-swimmingrobotic platform, i.e., PipeDiver R© through travelling along the pipelines. The office-based sur-vey is then conducted by the inspectors with the aim of performing pipe condition assessment,as demonstrated in Fig. 3.1. In the office-based survey, key-frame identification is performedwith an autoencoder-based framework, after which the key frames with pipe anomalies can becollected. Then, the inspectors can focus more on these key frames to conduct the assessmentreport for further pipe assessment and maintenance by the utility manager. Three frameworksare introduced and implemented in the thesis. To be more specific, an unsupervised learning-based framework is first conducted to perform the task, which is built based on state-of-the-artautoencoder networks. Secondly, SPAE-based framework is carried out with the proposedautoencoder-based network, which is incorporated with steerable pyramid method. Precisely,this framework is developed with the combination of an unsupervised deep learning model anda supervised classifier to improve the performance. Furthermore, another framework, namely,LGAE-based framework, is then performed to improve the accuracy of key-frame identifica-tion on the dataset. This framework has a similar schematic structure to that of the SPAE-basedframework, but with an integration of log-Gabor filter to enhance the capability of identifyingkey frames. Three frameworks and related evaluation metrics for the frameworks are intro-duced in detail in this chapter.143.2. The Unsupervised Learning-based FrameworkFigure 3.1: The research overview of the water pipeline CCTV inspection.3.2 The Unsupervised Learning-based FrameworkAs stated above, key-frame identification is to detect the key frames containing pipe anoma-lies (also known as anomaly frames) in the CCTV inspection videos. The unsupervised learning-based framework is first performed for this task. It is composed of a state-of-the-art autoen-coder and a threshold determinator for identifying the key frames out of the video frames.3.2.1 Framework StructureIn this section, the structure of the unsupervised learning-based framework is shown inFig. 3.2. Specifically, the framework is formed by an autoencoder-based network and a thresh-old determinator. Normal video frames are used for training the model and obtaining thethreshold. The top of the flow diagram shows the training procedure, and the bottom of thatpresents the testing process. The pre-trained autoencoder model and the pre-defined thresholdare obtained from the training process and applied to identify the key frames of the CCTVinspection videos. To be more explicit, the threshold is determined on the training result of theautoencoder-based network. After the training process of the network, the loss values of thenormal frames from the network will fall onto a certain level, which is relatively lower sincethe network has learnt the patterns of the normal frames accurately. However, the loss valuesof the key frames from the network will fall onto a much higher level because the network hasnot learnt the patterns of the key frames in the training process. A threshold can be obtainedfrom the averaged estimation for the loss values of normal frames. As a result, the thresholdcan be applied to detect the key frames out of the video frames.3.2.2 Computational MethodologiesUnsupervised learning is conducted by using three autoencoder-based models, which areautoencoder, variational autoencoder, and convolutional autoencoder. Autoencoder-based mod-153.2. The Unsupervised Learning-based FrameworkFigure 3.2: The unsupervised learning-based framework with state-of-the-art autoencoders.els are capable of reconstructing images. Reconstruction loss can be comprehended as thedifference between the input and output data, which is also known as reconstruction error. Inthe proposed framework, the model is only trained on normal frames of the CCTV videosto achieve a minimum reconstruction error. Hence, the model can reconstruct normal videoframes accurately but not the frames with pipe anomalies. If a threshold is appropriately de-fined, key frames can be detected effectively based on the reconstruction loss.AutoencoderAn autoencoder is a type of neural network, which is performed to learn efficient datacodings in an unsupervised manner [22]. The aim of an autoencoder is to learn a representation(encoding) for a set of raw data. Autoencoder consists of an input layer, an encoder network,and a decoder network [48]. Specifically, it contains an input layer, multiple hidden layers,and an output layer. The raw data are fed into the model through the input layer. The encoderlayer is for encoding the input data, learning the representative features. While the decoderlayer is the reverse of the encoder layer, using the features to reconstruct the raw data. A basicautoencoder is formed by fully-connected layers in the model. Fig. 3.3 shows the schematicdiagram of a basic autoencoder network, where the gray hidden layers represent the fully-connected layers.Figure 3.3: The schematic structure of an autoencoder.163.2. The Unsupervised Learning-based FrameworkAssuming the autoencoder has L layers, the input data is defined as N-dimensional vectorx and the corresponding values of the output layer is xL. Based on the parameter optimization,an autoencoder network can learn representative features in the latent space and reconstruct theoriginal data. The output data of an autoencoder has the same dimensions as that of the inputdata, which is fed into the network.The loss function of an autoencoder can be described as:loss =1KK∑k=1(xLk − xk)2 (3.1)where K is the number of training samples. xk is the kth training sample, and xLk is the outputvalue of the kth training sample. loss is the reconstruction loss of the autoencoder network(also known as mean squared error).Convolutional AutoencoderConvolutional autoencoder is developed from AE by adding convolutional neural network(CNN) structures [26] [27]. CAE takes advantage of CNN because it applies convolutionalstructure in the form of an autoencoder. Commonly, a convolutional layer is applied to ex-tract the features from the input data, followed by a pooling layer to compress the features ofthe input. The representation in the latent space is then converted back to the original dimen-sions in reverse through convolutional and upsampling layers. Fig. 3.4 indicates the schematicstructure of a convolutional autoencoder. To be more explicit, the first gray block of Fig. 3.4represents the hidden convolutional layers with pooling layers for feature compression. In con-trast, the second gray block shows the hidden convolutional layers and upsampling layers inthe decoder part of the model. Obviously, the orange block demonstrates the representationextracted through the encoding process in the latent space.Figure 3.4: The schematic structure of a convolutional autoencoder.173.2. The Unsupervised Learning-based FrameworkVariational AutoencoderVariational Autoencoder (VAE) is a popular generative model, which allows representingdata in a compact probabilistic latent space [49, 50]. Specifically, it produces an encoding ofthe raw input x to a latent vector v = E(x) ∼ e(v|x), after which the latent vector is convertedback to the origin dimensions of the raw input, i.e., x′ = D(v)∼ d(x|v). The x and x′ representthe input data and output data of the VAE. The marginal log-likelihood of each pixel in theinput sample is maximized for the purpose of image reconstruction. Explicitly, the reconstruc-tion loss of the VAE lr is expected to be the negative log-likelihood of pixels in the raw input.Besides, VAE has a key component, that is, the capability of controlling the distribution of thelatent vector. This is conducted through Gaussian random variable, i.e., v ∼ N(0, I). Further-more, the distribution of a Gaussian distribution is also named KL divergence. The differencebetween the distribution of e(v|x) and the KL divergence can be quantified and minimized bygradient descent algorithm [49]. Hence, the training process of VAE involves the optimizationof the lr and KL divergence loss lkl . Fig. 3.5 illustrates the schematic structure of a variationalautoencoder. Specifically, the gray blocks of Fig. 3.5 demonstrate the hidden fully-connectedlayers in the encoder and decoder of the model. Moreover, the green block shows the distribu-tion of the latent vector through the controlling of the encoding layers. Related equations arealso displayed on the Fig. 3.5.Figure 3.5: The schematic structure of a variational autoencoder.lr =−Ee(v|x)[log(d(x|v))] (3.2)lkl = Dkl(e(v|x)||d(v)) (3.3)lvae = lr + lkl (3.4)To sum up, an unsupervised learning-based framework is proposed to conduct key-frame183.3. The Steerable Pyramid Autoencoder based Frameworkidentification for water pipeline CCTV inspection. Specifically, three state-of-the-art autoen-coder networks are performed in this framework to achieve the objective.3.3 The Steerable Pyramid Autoencoder based FrameworkThe steerable pyramid autoencoder (SPAE) based framework is on the basis of represen-tation learning and extraction. It is an integration of unsupervised learning method and super-vised learning method. Specifically, this framework consists of a feature extractor for extract-ing representative features and a classifier for performing the classification task. The featureextractor SPAE is trained in an unsupervised manner, while the classifier support vector ma-chine (SVM) is trained in a supervised learning manner with labels.3.3.1 Framework StructureAs introduced, key-frame identification is to detect frames containing pipe anomalies inthe CCTV inspection videos. The proposed key-frame identification framework is formed bysteerable pyramid autoencoder and support vector machine [51]. Steerable pyramid autoen-coder is performed for learning as well as extracting representative features and support vectormachine is used for classification. The structure of the framework for key-frame identificationis shown in Fig. 3.6. Specifically, the top of the flow diagram shows the training procedure andthe bottom of that presents the testing process. First, the video frames pre-extracted from theCCTV inspection videos are pre-processed to train the feature extractor and classifier. The pre-trained feature extractor and classifier are then obtained and applied to conduct the detectiontask, as shown at the bottom of Fig. 3.6. In this process, the videos are pre-processed and inputinto the model frame by frame, after which the frames predicted as key frames are collected.Figure 3.6: The SPAE-based framework for key-frame identification.In deep learning, feature learning, also known as representation learning is a variety oftechniques that enables a system to automatically exploit the representations needed for feature193.3. The Steerable Pyramid Autoencoder based Frameworkclassification or detection from raw data. Autoencoder is one of the state-of-the-art methodsfor this work. Significantly, this allows a machine to both learn the features automaticallyand make use of them to perform a specific task. However, autoencoder is learning the fea-tures layer by layer in depth, not in multi-scale or multi-orientation. Multi-scale representationlearning has drawn attention of researchers due to the different structures of objects at differentscales. Image pyramid is an implementation of learning image features at scales with sub-sampling operations. Hence, it is considerable to integrate the image pyramid features withthe automatically extracted features of a neural network to improve the performance of featurelearning. In order to improve the representation learning of the autoencoder network, steerablepyramid is integrated into the encoder of the model to perform multi-orientation feature learn-ing. In comparison with conventional image pyramids and feature pyramid networks, steerablepyramid is not only scale-selective, but also orientation-selective, which allows higher flexi-bility and deeper feature extraction for image analysis. In addition, steerable pyramid is alsosuperior in edge feature enhancement of the objects on images thanks to its recursive transfor-mation. Hence, we developed a steerable pyramid autoencoder, which is to fuse an autoencodernetwork with steerable pyramid, seeking to enhance representation learning and extraction.The SPAE takes advantage of both pre-defined and automatically extracted features, which isthe strength of SPAE.3.3.2 Computational MethodologiesThe SPAE-based framework is developed on the basis of the steerable pyramid techniqueand a customized autoencoder-based network. The computational methodologies are intro-duced in succession below.Autoencoder NetworkAs introduced in Section 3.2.2, an autoencoder is composed of an input layer, multiplehidden layers, and an output layer. The schematic structure of an autoencoder network isdisplayed in Fig. 3.7 with the aim of showing the training process of the model. The outputs ofthe layers denoted with the same color in Fig. 3.7 have the same dimensions of data. Precisely,each circle represents a node (also known as unit) in the specific layer, where the number ofunits is pre-defined. Each layer in the neural network has its own hyper-parameters, whichwill be optimized and updated in the training procedure of the autoencoder. The detailedinterpretation of the model training is stated in the following paragraph.Assuming the autoencoder has L layers, where l is defined as the lth hidden layer. Theweights between the (l−1)th and lth layer of the network are defined as the matrix Wl . More-203.3. The Steerable Pyramid Autoencoder based FrameworkFigure 3.7: The structure of a basic autoencoder network.over, bl is the matrix of biases in the lth layer, and φ l is the activation function in the lth layer.Activation functions are used to perform element-wise transformation and enable autoencoderto learn nonlinear reduction of the feature dimensions. Assuming input data as N-dimensionalvector x, the values of the lth hidden layer xl(l = 2,3, ...,L−1) and output layer xL are calcu-lated using the following recursive formula [30]:xl = φ l (Wlx(l−1)+bl), l = 2,3, ...,L (3.5)Based on the recursive formula, an autoencoder network can learn representative featuresin the latent space and reconstruct the original data. The output data of an autoencoder has thesame dimensions as that of the input data, which is fed into the network.Steerable PyramidA steerable pyramid (SP), proposed by Simoncelli et al., is an implementation of a multi-orientation and multi-scale bandpass filter bank used for applications, including image com-pression, object recognition, texture synthesis, as well as image registration and fusion [52]. Itcan be considered as an orientation-selective method, in which a set of steerable filters are usedat each level of the pyramid instead of a single filter [53, 54].On the basis, the steerable pyramid has a set of directional derivative operators with thenumber of n, leading to n+1 orientations. As for the derivatives, they span a rotation-invariantsubspace, in which the derivatives are designed and sampled. Fig. 3.8 shows the system dia-gram of a steerable pyramid [52]. It illustrates the filtering and sampling operations, as well asthe recursive subsystem in the dotted box. The frequency axis is defined as θ ranging from−pito pi in the spectral decomposition of steerable pyramid. The symbols, i.e., “2D” and “2U,”correspond to downsampling and upsampling by a factor of 2, respectively. The circles re-fer to the transform coefficients at the specific level [52]. The raw image is represented withhighpass and lowpass subbands, using highpass filters H0 and lowpass filter L0, respectively.213.3. The Steerable Pyramid Autoencoder based FrameworkFurthermore, the lowpass subband is then divided into a group of oriented bandpass subbandsand a lowpass subband. This lowpass subband is subsampled by a factor of 2 in two direc-tions, namely, X and Y [55]. In the system diagram of the steerable pyramid illustrated inFig. 3.8, the radial function is represented as Bn(θ), which is constrained by decompositionand aliasing reduction during the subsampling procedure. The filters indicated as H0(θ) andL0(θ) are implemented for the preparation of the recursion. The raw signal is divided by thesubsystem of steerable pyramid into two parts, which are highpass signal and lowpass signal.Next, the lowpass signal is subsampled, and the recursion is performed by repeatedly applyingthe recursive transformation to the lowpass signal [55]. The initial shape of lowpass is formedthrough L0(θ) = L1(θ/2), same as that used in the recursive subsystem. Here, L1(θ) is strictlybandlimited, and Bn(θ) is power-complementary.Figure 3.8: System diagram of the steerable pyramid, illustrating the filtering and samplingoperations, and the recursive subsystem in the dotted box. ”2D” and ”2U” correspond to down-sampling and upsampling by a factor of 2. The circles correspond to the transform coefficients.The constraints performed on the filters in the system diagram are as follows:Bandlimiting:L1(θ) = 0 f or |θ |> pi/2 (3.6)Flat system response:|B(θ)|2 = |B0(θ)|2+ |B1(θ)|2+ ...+ |Bn(θ)|2 (3.7)|H0(θ)|2+ |L0(θ)|2 [|L1(θ)|2+ |B(θ)|2] = 1 (3.8)Recursion:|L1(θ/2)|2 = |L1(θ/2)|2 [|L1(θ)|2+ |B(θ)|2] (3.9)As introduced above, the steerable pyramid method has the capacity of extracting repre-223.3. The Steerable Pyramid Autoencoder based Frameworksentative features of an image, emphasizing the region of interest on an image from differentorientations in multiple scales. Fig. 3.9 illustrates that the features of a sample video frame areextracted into pyramid scale with four orientations horizontally and four levels vertically. To bemore precise, the feature maps for the first level, namely, level 0 have the same dimensions asthe original input. With the increase of the depth of the pyramid, the dimensions of the featuremaps are downsized by a factor of 2 with more discriminative features extracted from the rawinput. Moreover, each small pyramid in Fig. 3.9 indicates the feature extraction in depth fromone specific orientation on a sample.Figure 3.9: The steerable pyramid.Steerable Pyramid AutoencoderThe steerable pyramid autoencoder (SPAE) is developed based on the incorporation ofsteerable pyramid into a customized CAE-based network. In the steerable pyramid autoen-coder, mean squared error (MSE) is applied for training the model. To be more specific, MSEis the average squared difference between the predicted value and the actual value. Symbol Lrepresents the output layer of the model. Assuming input data as N-dimensional vector x, xiand xLi are the ith dimensional values of the input x and output xL of the model (i = 1,2, ...,N).MSE(x) is the mean square error (also known as reconstruction loss) of the input sample x.Accordingly, the function of MSE can be described as:MSE(x) =1NN∑i=1(xLi − xi)2 (3.10)The training procedure of the autoencoder network involves parameter optimization of Wland bl for l = 2, ...,L, which are stated in the subsection 3.3.2. Assuming the training datasetas x1, ...,xk (k = 1,2, ...,K), the optimization procedure is to minimize the mean of the MSEscore with this training set by looking for the suitable parameters:233.3. The Steerable Pyramid Autoencoder based FrameworkminWl , bl1K1NK∑k=1N∑i=1(xLk,i− xk,i)2, l = 2, ...,L (3.11)where xLk,i and xk,i are the ith dimensional value of xLk and xk. The algorithm implemented forsolving the (3.11) problem is adaptive moment estimation (also known as adam) [56].The proposed steerable pyramid autoencoder is developed based on convolutional autoen-coder network, where the encoding structure is used as feature extractor. The encoder of themodel applies convolutional (Conv) layers to extract the image features, and the decoder ap-plies convolutional layers with upsampling layers to reconstruct the original image. Fig. 3.10shows the overall structure of the proposed framework. In Fig. 3.10, the grey layers representconvolutional layers in the SPAE model. The strides parameter is defined as 2 in the con-volutional layer of the encoder to downsize the feature maps, while the size parameter is setas 2 in the upsampling layer (represented by the yellow layer in Fig. 3.10) to upsample thefeature maps. The orientation is set to be 4 in each level of the steerable pyramid displayedin Fig. 3.10. Hence, four feature maps can be obtained at each level of the pyramid. Onthe basis of the availability of the extracted pyramid features, the third level of the steerablepyramid is selected to be incorporated into the autoencoder-based network. Specifically, thisis to avoid the insufficient or excessive effect of the recursive transformation of the steerablepyramid. From the trial and error study, we found that the first two levels of the pyramid didnot achieve useful feature extraction for the dataset, while the fourth level achieved excessivefeatures with certain aliasing. Therefore, the third level of the pyramid is selected to integratewith the proposed network.The diagram in Fig. 3.10 presents an example of the integration of four oriented steerableinputs with the proposed network. The fusion of steerable pyramid can be considered as alateral connection, which originates from the input layers of multiple steerable features fromvaried orientations. As demonstrated, the layer with four purple nodes indicates the featurefusion layer of steerable features, which is orientation selective. One input layer with one con-volutional layer (represented by each purple node) is used for each oriented steerable feature,followed by an addition layer to fuse them all. The input steerable features are then combinedwith encoded features with the same dimensions from SPAE through an addition layer. Thus,the dimensions of the selected intermediate layer for merging must match those of the steerableinputs.Fig. 3.11 presents a detailed diagram of the model construction from the input layer to thelayer used for representation extraction. The building block illustrates the lateral connection forthe fusion of steerable pyramid in the SPAE. The input is a 1×512×512 matrix with the pixelinformation from the video frame. And the input for the lateral connection is a 1×128×128243.3.TheSteerablePyramidAutoencoderbasedFrameworkFigure 3.10: The overall structure of the steerable pyramid autoencoder based framework.253.3. The Steerable Pyramid Autoencoder based Frameworkmatrix. Furthermore, a small ResNet module is also adopted into the autoencoder-based net-work, which is within the backbone of the network. It consists of two Conv layers, one nor-malization layer between two Conv layers, and at last a skip connection to the first Conv layerin the module [57]. Precisely, the ResNet module is directly connected between two layers inthe proposed network, without performing dimensionality reduction. It is represented as thegreen block in the model structure of Fig. 3.10. As is known, ResNet has several compellingadvantages such as alleviating the vanishing-gradient problem, powerful representational abil-ity, and strengthening feature propagation with a ”shortcut connection” [58]. Many computervision applications have been boosted by taking advantage of the powerful representationalability of ResNet, such as object detection and face recognition. As for normalization, groupnormalization (GN) is used in this model since it proves to be better than batch normalizationwhen batch size is small, which equals to 4 in our case [57].Figure 3.11: The building block illustrating the lateral connection for the fusion of steerablepyramid in the SPAE.263.3. The Steerable Pyramid Autoencoder based FrameworkSupport Vector MachineSupport vector machine, also known as support vector network, is a type of supervisedlearning model, which aims to analyze data for classification or regression prediction [59]. Tobe more precise, a support vector machine constructs a hyper-plane or a set of hyper-planes inthe space, which are then used to perform various kinds of tasks, e.g., classification, regression,or outlier detection. Explicitly, a hyper-plane constructed will be considered as a good one ifit has the largest functional margin, which means the distance to the nearest training sampleof any class. As a result, the classifier would have better generalization performance, thatis, lower generalization error [60]. However, it often happens that the problem on the rawsamples cannot be solved simply in a low-dimensional space. Therefore, the idea rose up that amapping could be constructed from the original space to a higher-dimensional space, which istargeted to perform the task more easily in the new space. The mapping is designed by a kernelfunction to define the computation regarding the input data vectors [61]. Radial basis function(RBF) is a type of nonlinear kernel functions, which is widely applied to any distribution ofthe samples. The schematic structure of a nonlinear SVM is shown in Fig. 3.12. It takes abinary classification in two-dimensional space as an example, where the orange scatters andblue scatters belong to two different classes. The black curve represents the decision functionconstructed to separate the training data points of these two classes. Besides, the dotted curvesdemonstrate the tolerance of the decision boundary on the misclassifications as introducedabove.Figure 3.12: The schematic structure of a nonlinear support vector machine.The RBF kernel function can be expressed as:K(xm,xn) = exp(−γ ‖ xm− xn ‖)2 (3.12)where xm denotes the data base point, and xn represents the testing data point for the mapping.273.3. The Steerable Pyramid Autoencoder based Frameworkγ is a parameter defined for the kernel function, which will be introduced in detail in thefollowing.With the definition of the kernel function, the problem the support vector machine aims tosolve is the minimization problem denoted as follows:minαm12N∑m=1N∑n=1ymynαmαn exp(−γ ‖ xm− xn ‖)2−N∑m=1αm (3.13)Among s.t.N∑m=1ymαm = 0, 0 <= αm <=Cwhere each xm is a multi-dimensional vector and ym is the corresponding label of the data pointxm. Also, it is the same for xn and yn. N is the number of the training samples in the dataset.The minimum value of the problem (3.13) depends on the choice of the parameters (C, γ) [62].Intuitively, the C parameter can be regarded as a penalty parameter, which imposes a penaltyfor each misclassified sample. The purpose of defining C is to seek for a trade-off between thecorrect prediction of the training samples against the maximization of the hyper-plane margin.In other words, smaller value of C leads to lower penalty on the misclassified samples, and thusa larger margin of the decision hyper-plane is prone to be selected with certain prediction errorsaccepted. On the contrary, larger value of C results in higher penalty on the misclassifications,that is, the classifier tries to minimize the misclassification rate. Hence, the decision boundaryconstructed by the classifier will encourage a smaller margin. On the other side, the γ parameterdefines the distance of the influence of a single training sample. To be more specific, lower γindicates that the decision hyper-plane has a larger radius, which is able to group more trainingsamples. However, higher γ means the radius of the decision boundary is smaller. In this case,only the samples close to each other are prone to be grouped in the same class. Therefore, asupport vector machine with much higher γ tends to be over-constrained on the training data,which means it tends to be overfitting. As a result, this classifier will not achieve promisingaccuracy for the prediction on the testing data.As illustrated in Fig. 3.10, the representative features extracted from the feature extractorare then fed into SVM classifier. Radial Basis Function (RBF) kernel is applied in the SVMconducted in this experiment. The γ parameter of the classifier is selected according to differentexperimental methods. Specifically, it is adjusted on the basis of varied dimensions of thefeatures extracted from the feature extractors. In this study, the γ is set as 0.001 for SED andCED as well as 0.003 for HOG and AE. As for VAE, the γ of the classifier is selected as 0.1.Additionally, the γ is adapted as 6× 10−5 for CAE and SPAE. The penalty parameter C is283.4. The Log-Gabor Autoencoder based Frameworkdefined as 1.0.To sum up, SPAE-based framework is proposed to conduct key-frame identification for wa-ter pipeline CCTV inspection. Explicitly, a feature extraction model SPAE is carried out basedon the incorporation of steerable pyramid and customized autoencoder network to achieve out-performing performance of representative feature learning.3.4 The Log-Gabor Autoencoder based FrameworkThe proposed log-Gabor autoencoder (LGAE) based framework involves unsupervised fea-ture extraction and supervised classification, which is similar to the SPAE-based framework.Log-Gabor filter is integrated into the autoencoder-based network to enhance the hard miningcapability. The detailed structure is introduced in succession below.3.4.1 Framework StructureThe proposed log-Gabor autoencoder (LGAE) based framework is formed by log-Gaborautoencoder and support vector machine (SVM) [19], which is similar to the schematic struc-ture of SPAE-based framework. Log-Gabor autoencoder is performed for extracting features,which are then fed into the support vector machine for prediction. The schematic structure ofthe LGAE-based framework is shown in Fig. 3.13. The top flow of the diagram demonstratesthe training procedure, while the bottom flow illustrates the testing process of the pre-trainedLGAE and SVM classifier.Figure 3.13: The LGAE-based framework for key-frame identification.As introduced in Section 3.3.1, autoencoder is learning the features layer by layer in depth,not in multi-scale or multi-orientation. It is crucial to strengthen the feature learning and extrac-tion, namely, to obtain more representative features from the images. Hence, it can help assistthe establishment of the decision hyper-plane of the classifier. In this study, it is proved that en-hancing edge feature detection of the anomalies captured on the key frames can be helpful for293.4. The Log-Gabor Autoencoder based Frameworkthe identification work. To date, image feature detection has gained a lot of research attentionin many fields. It has been extensively researched in computer vision, pattern recognition, andcontent-based image retrieval. For image feature detection, multi-scale representation learn-ing is widely implemented for image analysis and feature detection. Hence, multi-scale imageanalysis can be considered to improve the accuracy of image edge detection. Log-Gabor filterhas led to a renewed interest in the detection of image edge features. It has outstanding perfor-mance in many fields such as pattern recognition and computer vision, because the response ofthe filter may form a good representation for the application purpose. Hence, it is considerableto integrate the log-Gabor filter with the neural network to gain higher accuracy than that ofthe SPAE proposed in Section 3.3. Similarly, log-Gabor filter is integrated into the encoderpart of the autoencoder to perform multi-scale feature learning. Hence, a log-Gabor autoen-coder is developed, which is to fuse an autoencoder network with log-Gabor filter, aiming toenhance the hard mining on video frames. The LGAE takes advantage of both pre-defined andautomatically extracted features, which is the superiority of LGAE.3.4.2 Computational MethodologiesThe LGAE-based framework is developed on the basis of the log-Gabor technique and acustomized autoencoder-based network. The computational methodologies are investigated insuccession below.Log-Gabor FilterIn signal processing, space and frequency are two characteristics of a signal. It is impossibleto locate the part of a signal, which produces a specific frequency based on Fourier transform.Although a short-time Fourier transform can be helpful on this, it limits the basis functions to besinusoidal. In this case, log-Gabor filter has been proposed to provide a space-frequency signaldecomposition with higher flexibility, which is an improvement upon the original Gabor filter.Compared with other alternative filters, it is suggested that log-Gabor filter can better encodenatural images because it can better fit the statistics of those images. In pattern recognition, arepresentation, also known as extracted features will be obtained from the input images, whichcan then be fed into a classifier for a classification task. Log-Gabor filter can be widely appliedin this field since the response of the filter may form a good representation, that is, a good set ofrepresentative features for the application purpose. For instance, the log-Gabor filter has beensuccessfully implemented in face expression classification [63].To be more specific, log-Gabor filter has two important features. The functions of log-Gabor filter have no DC component (the mean amplitude of the wavelet function). In other303.4. The Log-Gabor Autoencoder based Frameworkwords, the DC component of the log-Gabor filter equals to zero. On the other hand, there is anextended tail at the high-frequency end of log-Gabor transfer function. Precisely, the functionof the original Gabor filter includes Gaussian transfer when viewed on the linear frequencyscale, while that of log-Gabor filter contains Gaussian transfer when viewed on the logarithmicfrequency scale [64]. Thus, log-Gabor function can better reflect the frequency response ofa natural image compared with the original Gabor filter, which is demonstrated on the linearfrequency scale [65]. The transfer function of log-Gabor function has the form:G( f ) = e−(log( f/ f0))22(log(ω f / f0))2 (3.14)where f0 and ω f are the parameters of the log-Gabor filter. f0 shows the filter’s centre fre-quency and ω f represents the bandwidth of the filter. To maintain consistent shape, the ω f / f0should be kept constant across varied frequency parameter f0. Accordingly, G( f ) is the fre-quency response of a log-Gabor filter.The last few years have seen an increased interest in log-Gabor filter for image processing.Hence, the log-Gabor filter is considered for the 2-dimensional extension [66]. To be moreprecise, the extended filter is designed for a particular frequency as well as a particular orienta-tion together. The orientation component of the filter is defined as a Gaussian distance functionwith regard to the angle in polar coordinates. The radial component of the 2-dimensional filteris described the same as 3.14, while the orientation component has a response denoted by:G(θ) = e−(θ−θ0)22ω2θ (3.15)where θ0 represents the center orientation, and ωθ is the width parameter of the orientation.With 3.14 and 3.15, the response of the 2-dimensional log-Gabor filter can then be de-scribed as 3.16 in succession below. The filter can be divided into two components, namelyradial filter and angular filter (also known as orientation filter). Thus, the 2-dimensional log-Gabor filter results from the combination of the functions for two components.G( f ,θ) = G( f ) ·G(θ) (3.16)where ( f ,θ) indicates the polar coordinates.A golden image pyramid of a sample frame is represented in Fig. 3.14(b) with eleven suc-cessive levels, which affects the edges of the objects on the input frame at varied scales byusing oriented log-Gabor filter. Meanwhile, the raw frame is also shown in Fig. 3.14(a). Theaspect ratio, namely, golden section, is selected as φ = 5√2+1 and each level of the pyramidcorresponds to the orientation performed by log-Gabor filter. The down-scaling factor is de-313.4. The Log-Gabor Autoencoder based Frameworkfined as φ2. The deeper the pyramid level is, the more evident the effect of the log-Gabor filteris. In this study, the first level of the golden pyramid is selected to be incorporated into theautoencoder-based network in case of excessive effect of the filter.(a) (b)Figure 3.14: Sample video frame and the golden pyramid of log-Gabor filter on the sampleframe: (a) sample video frame; (b) the golden pyramid of the log-Gabor filter.From the formulas above, it can be seen that the log-Gabor filter is determined by fourparameters: f0, θ0, ω f and ωθ . Precisely, f0 and θ0 correspond to the local frequency andorientation of the specific dataset. Besides, ω f and ωθ are determined based on the experienceor filter fitting on the dataset [67]. Regarding the detailed definition, the ω f is set to be 1.8,while ωθ is defined as 0.1589 according to the filter fitting and empirical information on thedataset. Besides, f0 is 92.29 and θ0 is 0.4053 for the dataset of this study. Above all, the firstlevel of the golden image pyramid is selected to be incorporated into the autoencoder-basedneural network.Log-Gabor AutoencoderIn the log-Gabor autoencoder, mean squared error (MSE) is applied for training the model,which is the same as the SPAE introduced in Section 3.3.2.The proposed log-Gabor autoencoder is also developed based on a customized convolu-tional autoencoder network, where the encoding structure is used as feature extractor. Fig. 3.15illustrates the model structure of LGAE, where the layer with one purple node indicates thefusion layer of log-Gabor filtered features. As mentioned, the first level of the log-Gabor im-age pyramid is selected to be incorporated into the autoencoder-based network according tothe quality of the extracted pyramid features as well as in case of excessive effect of the filter.Hence, each input sample will come with one log-Gabor filtered feature map to be input intothe network. To be more precise, one convolutional layer and one batch normalization layer areapplied after the input layer of log-Gabor features in the feature fusion. The input log-Gaborfeatures are then combined with encoded features from the autoencoder network through anaddition layer. Also, LGAE consists of the small ResNet module represented by a green block323.4.TheLog-GaborAutoencoderbasedFrameworkFigure 3.15: The overall structure of the log-Gabor autoencoder based framework.333.4.TheLog-GaborAutoencoderbasedFrameworkFigure 3.16: The model construction of the proposed LGAE.343.5. Evaluation Metricsin Fig. 3.15 to enhance the feature propagation. Group normalization (GN) is used in the smallResNet module since it proves to be good during the feature propagation when batch size issmall. The yellow layer represents the upsampling layer with the size parameter defined as2. Fig. 3.16 demonstrates a detailed diagram of the model construction for the LGAE. Thediagram displays the backbone of LGAE, the lateral connection for the fusion of log-Gabor,and the structure of the small ResNet module.As for the classifier, the representation extracted from the LGAE is fed into the SVM clas-sifier with the Radial Basis Function (RBF) kernel. The penalty parameter C is defined as 1.0.As for the γ parameter, it is adapted as 6×10−5 for LGAE, which is the same as that of SPAE.To summarize, LGAE-based framework is proposed to conduct key-frame identificationfor water pipeline CCTV inspection. Precisely, a feature extraction model LGAE is carriedout based on the integration of log-Gabor filter with a customized autoencoder-based networkto enhance the hard mining on the video frames for the purpose of improving the accuracy ofkey-frame identification.3.5 Evaluation MetricsThe performance of the proposed key-frame identification framework is validated usingthe contingency table of the binary classifier. Precisely, the contingency table is shown inthe following Table 3.1. For the classification task in the thesis, there are two classes of thepipeline data, one of which is normal frame, and the other is key frame (also known as anomalyframe). The classifier will predict each input as belonging to one class, namely, predictedclass, while the input also has its own actual class. The contingency table aims to compare theprediction performance for each class of the data. Specifically, it provides a basic picture of theinterrelation between two variables and can help find interactions between them. Assumingthat one dataset is labeled into two categories, the categories can be denoted as variables,namely, ‘0’ and ‘1’. As is demonstrated, the first column shows the actual variable, and thefirst row presents the predicted variable. Thus, the interrelation between the actual variableand the predicted variable can be illustrated through the calculation. Normally, the variable 1is regarded as positive, while 0 is considered as negative. Accordingly, true positive is denotedas TP, and false positive is indicated as FP. Besides, true negative is represented as TN, whilefalse negative is denoted as FN.Accuracy, precision, recall, and F1-score are four important evaluation metrics calculatedfrom the contingency table. The equations of these four evaluation metrics are given below:Accuracy =T P+T NT P+FP+T N+FN(3.17)353.5. Evaluation MetricsTable 3.1: The contingency tableActual/Predicted 0 10 TN FP1 FN TPPrecision =T PT P+FP(3.18)Recall =T PT P+FN(3.19)F1-score = 2× Precision×RecallPrecision+Recall(3.20)where T P is the true positive (correctly detected positive ones). FP is the false positive (falselypredicted as positive). Similarly, T N is the true negative (correctly detected negative ones) andFN is the false negative (falsely predicted as negative).Explicitly, accuracy represents the ratio of the correctly predicted observations to the totalobservations. Precision shows the ratio of the correctly predicted anomaly observations tothe total predicted anomaly observations. Recall denotes the ratio of the correctly predictedanomaly observations to the all observations in actual class. F1-score indicates the weightedaverage of precision and recall. Hence, recall is one of the most essential metrics on accountof the purpose of the study, which is to identify the anomaly frames contained in the videos asmany as possible.Receiver operating characteristic (ROC) curve and precision-recall (P-R) curve are consid-ered as another two important evaluation plots of the prediction model skill. ROC curve is aplot of the false positive rate (x-axis) versus the true positive rate (y-axis) for a group of dif-ferent candidate threshold values between 0.0 and 1.0. AUC is the area under the ROC curve.The precision-recall curve is a plot of the recall (x-axis) and the precision (y-axis) for a set ofdifferent thresholds.True positive rate (TPR) and false positive rate (FPR) can be described as following equa-tions:TPR =T PT P+FN(3.21)FPR =FPFP+T N(3.22)where T P is the true positive value as mentioned above. FP is the false positive value and T N363.5. Evaluation Metricsis the true negative value. Besides, FN is the false negative value.For deep neural networks, there is a concern of overfitting problem, which is frequentlyencountered in the model training process. Therefore, k-fold cross-validation is conducted toevaluate the experimental methods. In k-fold cross-validation, the raw dataset is randomlysplit into k equal sized subsamples. Among the k subsamples, one subsample is retained as thetesting data of the model, and the remaining k− 1 subsamples are used as training data. Thecross-validation process is then repeated k times, where each subsample is used as the testingdata one by one, which is displayed in the diagram of Fig. 3.17. k results can be collected andthen be calculated to produce an average estimation. The goal of cross-validation is to test themodel’s ability to predict new data that was not used in estimating it, in order to prove againstproblems like overfitting. The k value can be larger if the raw data is a very small amountof dataset. For the dataset with 5,000 samples in our study, k value is selected as 5. Thus,five-fold cross-validation is implemented in this study to provide an accurate estimate of themodel performance.Figure 3.17: The diagram for k-fold cross-validation.Temporal regularity is another evaluation metric applied in the visualization of key-frameidentification. It is represented as the probability value from the classifier to detect each frameas key frame. For the classifier, the frame with a probability value larger than the threshold willbe classified as key frame, whereas the frame with a probability value less than the thresholdis identified as normal frame. Therefore, the pattern of the video is presented, where a peak inthe pattern probably indicates there are key frames identified.373.6. Summary3.6 SummaryIn this chapter, an unsupervised learning-based framework is first implemented for the key-frame identification on the water pipe dataset. Related state-of-the-art autoencoder networksare introduced, which are applied in the framework. For the purpose of further improving ac-curacy, SPAE-based framework is then proposed to conduct key-frame identification for waterpipeline CCTV inspection. Explicitly, a feature extraction model SPAE is carried out based onthe integration of steerable pyramid with a customized autoencoder network to achieve outper-forming performance of representation learning. Thirdly, LGAE-based framework is proposedto enhance the hard mining on video frames. Precisely, the LGAE-based framework is com-posed of a log-Gabor autoencoder and a classifier, which has a similar schematic structure tothat of the SPAE-based approach. The log-Gabor filtered features are incorporated into theautoencoder-based network, constituting the LGAE to achieve superior feature learning andgeneration. Finally, the crucial evaluation metrics are introduced for the estimation of the ex-perimental methods in the study.38Chapter 4Experimental Results and Discussions4.1 Data DescriptionIn this study, PipeDiver R© [9] was employed as a platform to collect the inspection data.The robotic platform is equipped with a lighting array and multiple cameras. The cameras areattached to this tool from different angles to conduct CCTV inspection and collect video data.Specifically, this study employed the video data collected from the PipeDiver R© through trav-eling along the pipelines. There are two datasets collected from two different water pipelines.The first dataset contains 5,000 video frames obtained from water pipeline CCTV videos in2018, which were labeled into two categories, normal frames, and key frames, respectively.The key frames include various kinds of pipe anomalies such as air pocket, longitudinal crack,and different types of outlets. This dataset is mainly applied for fitting and testing the selectedexperimental methods. The second dataset contains 880 labeled video frames obtained in 2015,among which the key frames include several types of pipe anomalies, i.e., debris, outlet, andcrack. This dataset is only used for estimating the generalization performance of the proposedmethods, which will be introduced in the corresponding section below.4.2 Experimental SetupThe experiments were conducted on the computer with i7−8750H processor and NVIDIAGeForce GTX 1070 Graphic Processing Unit (GPU). The training dataset contains 5,000 videoframes obtained from water pipeline CCTV videos in 2018, which were labeled into two cat-egories, normal frames (2,560) and key frames (2,440), respectively. The key frames includevarious kinds of pipe anomalies, i.e., air pocket and different types of outlets.For the unsupervised learning-based framework, the raw frames were pre-processed andresized to 512× 512 pixels with one channel for the purpose of controlling computationalmemory. The channel is selected appropriately with acceptable quality. Since only the normalframes were used for training the autoencoder-based networks, the normal frames were shuf-fled and split into training set (60%), validation set (20%), and testing set (20%). In addition,the key frames were used only for testing the models. Hence, the key frames were shuffled,394.3. Results and Discussionsamong which 512 frames were randomly selected for testing the model performance. The ini-tial epochs of training the neural networks were set to be 50 with batch size as 4. Furthermore,adam [56] was configured to be the optimizer of training the neural networks.For the SPAE and LGAE based frameworks, five-fold cross-validation was conducted toavoid overfitting problem in model selection. The raw frames were pre-processed and resizedto 512×512 pixels with one channel, which is the same as that of previous framework. In thefive-fold cross-validation, the dataset was shuffled and split into five subsamples, where eachsubsample is retained as the testing data one by one, and the remaining subsamples are usedfor training. Namely, 4,000 video frames were used for training the model, and 1,000 frameswere used for testing in each iteration of the cross-validation. To be more specific, there are2,048 normal frames and 1,952 key frames in the training set. The testing set consists of 512normal frames and 488 key frames. The initial epochs of training the neural networks wereset to be 50 with batch size as 4. Furthermore, adam was configured to be the optimizer oftraining the neural networks. To avoid overfitting problems, early-stopping was applied in thetraining procedure of the neural networks. In addition, dropout layer with the rate of 0.2 wasimplemented in the SPAE and LGAE. Furthermore, group normalization with the groups of4 is also able to help prevent overfitting in practice. As for CNN, weight regularization wasapplied for the layer’s kernel and bias. Regularizer with both L1 and L2 penalties was createdwith the factor of 0.001 for each penalty. Meanwhile, dropout layer with the rate of 0.3 wasconfigured in the model.4.3 Results and DiscussionsThe study involves three different frameworks for key-frame identification on water pipelineCCTV videos. For each framework, comparative studies were conducted, and the experimentalresults were collected.4.3.1 The Unsupervised Learning-based FrameworkAs introduced in the Section 3.2, this framework is based on unsupervised learning, wherethe state-of-the-art autoencoders are only trained with normal frames. Specifically, the au-toencoders will learn and extract the representative features of a normal frame, outputting areconstructed frame with corresponding reconstruction loss. In the training process, the au-toencoders try to reconstruct the raw frames as accurately as possible, namely, with lowerreconstruction loss. In other words, the autoencoders do not learn to extract the features of thekey frames. Thus, they are not able to reconstruct the key frames as accurately as the normal404.3. Results and Discussionsones, which means the loss values of the key frames are much higher than those of the nor-mal ones. In the study, normal video frames are fed into the autoencoders, and the trainingiterations are set to be 50.Comparative AnalysisThe training loss curves of three experimental methods are illustrated in Fig. 4.1. As canbe observed, the training loss of each method shows a stable decrease along with the trainingiterations. In Fig. 4.2, the models are tested on 1,024 samples, among which the first 512 sam-ples are normal frames, while the last 512 ones are anomaly frames. It can be observed that thenormal and anomaly frames are more prone to be separated by AE through applying an appro-priate threshold on the loss values. However, the losses of normal and anomaly frames fromCAE and VAE show certain overlapping, which results in the lower identification accuracy ofthe anomaly frames.Figure 4.1: The training curves for the loss of the experimental models.For better visualization, a loss distribution graph is plotted for each experimental method.On the x-axis of the graph, the first 512 samples are normal frames, while the last 512 onesare anomaly frames. To be more specific, blue scatters represent the reconstruction errors ofnormal video frames, while the orange ones are those of the anomaly frames. A threshold isdefined for each method to perform the detection task, which is indicated by the red line inthe loss distribution graph. Hence, it can be observed in Fig. 4.3 that the majority of the loss414.3. Results and DiscussionsFigure 4.2: The reconstruction loss of the testing samples.Figure 4.3: Loss distribution of the autoencoder.values of anomaly frames distribute approximately above the threshold line, while those of thenormal ones distribute under the threshold line for AE. Apparently, the distributions of the lossvalues of two classes have larger aliasing for CAE in Fig. 4.4. The loss distribution of VAE issimilar to that of AE, which is promising in Fig. 4.5.424.3. Results and DiscussionsFigure 4.4: Loss distribution of the convolutional autoencoder.Figure 4.5: Loss distribution of the variational autoencoder.The evaluation indices for the key-frame identification are shown in Table 4.1. Since recallrepresents the ability of predicting the anomaly frames correctly, it is one of the most importantmetrics for evaluating the experimental methods. Based on the comparison, the accuracy of the434.3. Results and DiscussionsTable 4.1: Evaluation of three autoencodersMethodEvaluation MetricsAccuracy Precision Recall F1-scoreCAE 0.833 0.772 0.943 0.849VAE 0.921 0.902 0.945 0.923AE 0.940 0.901 0.988 0.943AE is 0.940, higher than 0.833 of CAE and 0.921 of VAE. Regarding the F1-score, the AEalso holds 0.943, which is higher than 0.849 of the CAE and 0.923 of the VAE. The fact thatthe F1-score of CAE is obviously lower is due to the low precision of CAE with the value of0.772. As for the recall, AE shows better performance with the value of 0.988, however, theprecision of AE is relatively low with the value of 0.901. CAE does not perform well becauseit can reconstruct the anomaly frames well, which leads to lower loss values of the anomalyframes. Hence, the conclusion can be drawn that AE outperforms the other two models in thisframework on the detection task.DiscussionsIn this unsupervised framework, one factor is crucial in terms of affecting the model per-formance, which is the defined threshold. The threshold determines the trade-off between theratio of the correctly predicted normal frames and that of the correctly predicted key frames tothe total observations of either class. To be more specific, the reconstruction losses of normaland anomaly frames have certain difference. If a threshold is defined properly, the frames withthe losses below the threshold can be regarded as normal frames, whereas the frames with thelosses above the threshold can be identified as key frames. Certainly, there are sample pointsdistributed close to the threshold defined. Hence, it is not hard to understand that lower thresh-old leads to higher ratio of the correctly detected key frames to the total observations of thisclass. On the contrary, higher threshold results in an increase of the accuracy for identifyingnormal frames, but meanwhile, it leads to a lower recall metrics. Accordingly, the trade-off be-tween these two metrics, i.e., precision and recall, is hard to some extent, which is a drawbackof this framework. This is also the reason why the framework with higher recall above 0.940shows relatively lower precision, which is below 0.910. On the other side, this framework ison the basis of unsupervised learning, where the feature learning model only learns the patternof normal frames. Without the supervision of the data labels, it is difficult to have control overthe learning process of the model. Therefore, the unsupervised learning-based framework doesnot show competitive performance in this task.444.3. Results and DiscussionsTo summarize, comparative results regarding four evaluation metrics are analyzed and dis-cussed to estimate the model performance. Additionally, loss distribution plots are provided tohave a better visualization of the model performance on the key-frame identification. Accord-ing to the experimental results, the unsupervised learning-based framework does not achievepromising performance on this task.4.3.2 The Steerable Pyramid Autoencoder based FrameworkThe experimental results are illustrated regrading the experiments conducted on the CCTVinspection videos of the internal surface of pipe. Parameter optimization of the SPAE was alsoperformed. Furthermore, the SPAE framework was compared with three image processingmethods, and state-of-the-art autoencoder models. The proposed framework was validatedwith labeled video frames and the comparative results are presented. The SPAE frameworkwas also tested on a video clip of the internal surface of water pipe.Parameter OptimizationParameters of the proposed SPAE were optimized by comparing the model performanceregarding different numbers of hidden layers. On the other hand, varied numbers of orientationsregarding the steerable pyramid features were also considered to optimize the SPAE model.Hidden Layers in the SPAE As shown in Fig. 3.11, the hidden representation of 4×64×64in SPAE was adopted to train the classifier, which meant each sample was represented by4× 64× 64 features. Hence, the representations of samples were predicted by the classifier,and accuracy can be computed. There are hidden layers for down-sampling in the encoderof the model between the layer of 64× 64× 64 and the hidden representation of 4× 64× 64.Dropout layer and early-stopping are applied as well in the training process to prevent fromoverfitting. From the perspective of reducing parameters, convolutional layer (Conv) is adoptedin the latent space of SPAE.Results of model performance regarding different numbers of hidden layers are comparedin Table 4.2. It can be observed that SPAE performs best with two Convs in the latent space,achieving the highest accuracy of 0.975. For one Conv and three Convs, the former one winswith an accuracy of 0.970 compared with 0.955 of the latter one. The fact that they showworse results indicates that neither more layers or less layers achieve improvement for themodel performance. Therefore, two Convs are used for down-sampling to the representationof 4×64×64.454.3. Results and DiscussionsTable 4.2: Parameter optimization of the hidden layersParameterEvaluation MetricsAccuracy Precision Recall F1-score AUCOne Conv 0.970 1.000 0.936 0.967 0.968Two Convs 0.975 1.000 0.947 0.972 0.973Three Convs 0.955 1.000 0.904 0.950 0.952Orientations of Steerable Pyramid In a steerable pyramid, two types of filters are appliedfor extracting features; one of which is high-pass filter and the other is low-pass filter. Basically,features can be extracted from one to four orientations with corresponding filters at 0◦(180◦),45◦(225◦), 90◦(270◦) or 135◦(315◦), respectively. The filters are defined accordingly to extractfeatures from different orientations. Table 4.3 shows the evaluation of the model performancewith different steerable orientations. It is concluded that SPAE performs best on key-frameidentification with four steerable orientations, achieving the highest F1-score of 0.988 andAUC of 0.989.Table 4.3: Parameter optimization of the steerable orientationsParameterEvaluation MetricsAccuracy Precision Recall F1-score AUCOne orientation 0.975 1.000 0.947 0.972 0.973Two orientations 0.969 0.943 0.993 0.968 0.970Three orientations 0.980 0.963 0.995 0.979 0.980Four orientations 0.989 0.981 0.995 0.988 0.989Comparative AnalysisFive-fold cross-validation is conducted to avoid overfitting problems. The validation resultswere collected across five subsamples and the mean value as well as standard deviation of eachmethod were calculated. The results of accuracy and recall are displayed in the Table 4.4 as theyare two of the most important evaluation indices in the experiment. As can be observed, theCED and HOG achieve lower accuracy across five subsamples. Although SED performs higheraccuracy, it fails to detect key frames accurately, which results in the lower recall. Specifically,there should be a balance between precision and recall for the method selection. As for theneural networks. They perform well with a much higher accuracy over 0.950. Nevertheless, theperformance on recall should be considered as recall is one of the most important evaluation464.3. Results and Discussionsmetrics in the work. It can be observed that SPAE outperforms the other selected methodsacross five subsamples with the recall higher than 0.980. In terms of accuracy, CNN performsbetter than SPAE on the first subsample with the accuracy of 0.976, whereas SPAE wins on theother four subsamples.Figure 4.6: The bar chart of the accuracy in the cross-validation of SPAE-based framework.Figure 4.7: The training history of SPAE.The bar chart of the accuracy across five subsamples with the error bar of the mean accuracyis displayed in Fig. 4.6, where several neural networks show good performance. The black lineon top of each error bar denotes the standard deviation of the corresponding method in five-fold cross-validation. SPAE shows better performance than the other selected neural networks.In addition, the comparative results of the precision and F1-score is shown in Table 4.5 with474.3.ResultsandDiscussionsTable 4.4: The comparison of the accuracy and recall from five-fold cross-validationMethodAccuracy RecallFold Index Mean Fold Index Mean1 2 3 4 5 1 2 3 4 5SED [18] 0.948 0.948 0.946 0.946 0.949 0.947±0.001 0.893 0.893 0.889 0.889 0.895 0.892±0.002CED [18] 0.836 0.860 0.848 0.829 0.851 0.844±0.011 0.668 0.727 0.702 0.659 0.700 0.692±0.025HOG [19] 0.902 0.893 0.886 0.911 0.891 0.896±0.008 0.862 0.854 0.854 0.827 0.821 0.844±0.016CNN [22] 0.976 0.968 0.958 0.973 0.960 0.967±0.007 0.950 0.934 0.913 0.952 0.918 0.934±0.016AE [22] 0.975 0.940 0.960 0.963 0.964 0.960±0.011 0.948 0.877 0.918 0.924 0.926 0.919±0.023CAE [26] 0.961 0.945 0.948 0.949 0.970 0.954±0.009 0.920 0.887 0.893 0.895 0.938 0.907±0.019VAE [29] 0.953 0.945 0.950 0.951 0.949 0.949±0.002 0.903 0.889 0.897 0.905 0.903 0.900±0.006SPAE 0.975 0.987 0.968 0.995 0.995 0.984±0.010 0.995 0.981 0.991 0.989 0.989 0.990±0.005484.3.ResultsandDiscussionsTable 4.5: The comparison of the precision and F1-score from five-fold cross-validationMethodPrecision F1-scoreFold Index Mean Fold Index Mean1 2 3 4 5 1 2 3 4 5SED [18] 1.000 1.000 1.000 1.000 1.000 1.000±0.000 0.943 0.943 0.941 0.941 0.944 0.943±0.001CED [18] 0.993 0.980 0.980 0.984 0.991 0.986±0.005 0.799 0.835 0.818 0.790 0.821 0.813±0.016HOG [19] 0.931 0.920 0.906 0.987 0.947 0.938±0.028 0.895 0.886 0.879 0.900 0.880 0.889±0.008CNN [22] 1.000 1.000 1.000 0.991 1.000 0.998±0.003 0.974 0.966 0.955 0.971 0.957 0.965±0.008AE [22] 1.000 1.000 1.000 1.000 1.000 1.000±0.000 0.973 0.934 0.957 0.960 0.961 0.958±0.013CAE [26] 1.000 1.000 1.000 1.000 1.000 1.000±0.000 0.958 0.940 0.943 0.944 0.968 0.951±0.011VAE [29] 1.000 0.997 1.000 0.993 0.991 0.996±0.003 0.949 0.940 0.946 0.947 0.945 0.946±0.003SPAE 0.954 0.991 0.945 1.000 1.000 0.978±0.023 0.974 0.986 0.968 0.994 0.994 0.984±0.011494.3. Results and Discussionsregard to eight experimental methods. It can be observed that SED, AE, and CAE all achievethe best performance on the precision index with the value of 1.000 on five subsamples in thecross-validation. Also, CNN shows good precision of 1.000 on four subsamples. Explicitly,these methods is prone to detect the normal frames correctly, that is, less likely to misidentifynormal frames as key frames. However, they also have a larger probability of falsely predictingthe key frames as normal frames. Thus, based on the weighted average of precision and recall,the SPAE has outperforming performance on the F1-score compared with other methods. Thetraining history of SPAE is also illustrated in Fig. 4.7, which presents the training loss andvalidation loss in different training iteration. It can be seen that the training and validationloss are descending stably. The curves reach the optimal level, keeping flat after around 25iterations.The proposed method is compared with existing state-of-the-art methods in Table 4.6 withrespect to five important evaluation metrics. The results show the average performance of theexperimental methods with standard deviation based on five-fold cross validation. Since recallrepresents the ability of predicting the key frames correctly, it is one of the most importantmetrics for evaluating the experimental methods. First, SED, CED and HOG were conductedand classification was performed by SVM with the same kernel. From Table 4.6, it can befound that CED achieves the lowest accuracy of 0.844 in this work. Although, SED and HOGhave higher accuracy than CED with 0.947 and 0.896, respectively, the results regarding therecall of these two methods are not very promising. The poor performance indicates that thesethree image processing methods are not suitable on this dataset, which could result from thelow illumination problem of the dataset. For better visualization, the feature extractions bySED, CED and HOG are tested on one testing sample and the feature maps are illustrated inFig. 4.8(a). As observed, the testing sample shows a small and unclear anomaly (outlet) onthe internal surface of the pipe. However, the features extracted by SED are not clear due tothe illumination. The CED detect a few features of the anomaly but not necessarily enough forthe detection task. As for HOG, it fails to extract adequate features of the unclear anomaly onthe frame. On the contrary, the CNN performs much better than the previous methods with anaccuracy of 0.967, which implies that the deep learning model is preferable in this work. CNNperforms well especially on precision with a value of 0.998. Nevertheless, the recall of CNNis reported as 0.934 with the SD of 0.016, which is relatively lower compared with the othermetrics of it.In this study, three state-of-the-art autoencoders were implemented, which were AE, CAEand VAE, respectively. After that, the hidden representations of the models were adopted totrain the SVM and the prediction results were computed and illustrated in Table 4.6. Amongthree encoder-based models, the AE is better than CAE and VAE with the accuracy and AUC504.3. Results and DiscussionsTable 4.6: Evaluation of key-frame identification on all the experimental methodsMethod Evaluation MetricsAccuracy Precision Recall F1-score AUCSED [18] 0.947±0.001 1.000±0.000 0.892±0.002 0.943±0.001 0.946±0.001CED [18] 0.844±0.011 0.986±0.005 0.692±0.025 0.813±0.016 0.841±0.011HOG [19] 0.896±0.008 0.938±0.028 0.844±0.016 0.889±0.008 0.895±0.009CNN [22] 0.967±0.007 0.998±0.003 0.934±0.016 0.965±0.008 0.966±0.007AE [22] 0.960±0.011 1.000±0.000 0.919±0.023 0.958±0.013 0.959±0.012CAE [26] 0.954±0.009 1.000±0.000 0.907±0.019 0.951±0.011 0.953±0.010VAE [29] 0.949±0.002 0.996±0.003 0.900±0.006 0.946±0.003 0.948±0.003SPAE 0.984±0.010 0.978±0.023 0.990±0.005 0.984±0.011 0.984±0.011(a) Features extracted bySED, CED and HOG(b) Example hidden representations from CNN, CAE andSPAE compared with raw framesFigure 4.8: The comparison of the extracted features between the selected experimental meth-ods.of 0.960 and 0.959. However, these three autoencoders did not achieve improvement on recallcompared with that of CNN. From the perspective of further improving the recall, we developeda steerable pyramid autoencoder that sought to further enhance the representation learningby integrating steerable pyramid into the autoencoder network. As shown in Table 4.6, theproposed model achieves best results on accuracy, F1-score and AUC. The fact that SPAEoutperforms other methods proves the improvement by this fusing strategy, which strengthensthe representation learning and extraction. Our proposed model has a much higher recall of514.3. Results and Discussions(a) (b)(c)Figure 4.9: Illustration of the comparative results.0.990 and accuracy of 0.984, because it explicitly extract more accurate representation, thusimproves the prediction accuracy. It is worth noticing that the F1-score of SPAE is 0.984,which is much higher than all the other tested methods.For better visualization of Table 4.6, bar charts are reported on three evaluation metrics,which are accuracy, F1-score and AUC. From Fig. 4.9(a), it can be noticed that the SPAEremains the highest bar on accuracy, whereas the CED has the lowest accuracy under 0.850.As stated in Fig. 4.9(b), the highest F1-score of SPAE indicates that this model performs bestacross precision and recall compared with other methods. Moreover, Fig. 4.9(c) shows thatSPAE achieves the best result on AUC with 0.984, where CED performs worst with a lowAUC of 0.841. Based on the comparison among all the experimental methods, the three imageprocessing methods are not applicable for key-frame identification on our dataset. The poor524.3. Results and Discussionsperformance can be caused by the low illumination problem of the dataset. Additionally, thesemethods rely on more clear and useful information in data, which could be the reason why theirperformance is not good, especially in the aspect of recall. Furthermore, CNN shows promisingresults, however, it is not suitable in this work due to the relatively low recall. Among allthe tested encoder-based models, the proposed steerable pyramid autoencoder outperforms theother models especially in terms of recall. It has explicitly outstanding prediction results onthree essential metrics, namely, accuracy, F1-score and AUC.Figure 4.10: ROC curves of the deep learning methods in the study.As discussed above, the deep learning methods conducted in this study show good perfor-mance in this work. According to the ROC curves in Fig. 4.10, it can be found that the star line(SPAE) stays highest compared with those of the others, accounting for the largest area undercurve, which corresponds to the largest AUC value of 0.984. It proves that SPAE achieves thebest result on true positive rate, leading to the best performance on identifying the key frames.As for the other tested models, the curves imply that they achieve higher false positive ratescompared with SPAE, which means they are more prone to falsely predict the normal framesas key frames. In the precision-recall curves from Fig. 4.11, the curve from SPAE remainshighest, above all the curves of other models, namely AE, VAE, CAE and CNN. The fact thatSPAE shows the best performance in precision-recall curves implies that it has an advantageof maintaining good precision over other models, while achieving high recall close to 1.000.534.3. Results and DiscussionsFigure 4.11: Precision-recall curves of the deep learning methods in the study.The hidden representation of SPAE is compared with those of CNN and CAE in Fig. 4.8(b).It is worth noticing from the feature maps that CNN and CAE achieve good performance offeature learning from our raw data, but they do not generate sufficient representative featuresfor samples with small and unclear defects. However, it can be found that SPAE is able to ex-tract more edge features of the anomaly regions on the key frames. Thus, the proposed SPAEhas compelling capacity of representation learning and extraction. It can be beneficial for theclassifier to predict varied key frames out of the normal frames.As is known, the early layer of an autoencoder technically learns the lower semantic in-formation extracted from the input. Skip connections is able to compromise between shortand long networks. Specifically, complex features can be developed in the layers close to thecentre of the model with skip connections and lower semantic features can also be accessed atthe same time from the earlier layers. The purpose of applying skip connection is to enhancethe development of complex features in the latent space. Hence, the hidden representationsof the SPAE can capture the complexity of the raw input and be more representative. A ma-chine learning algorithm called t-distributed stochastic neighbor embedding (t-SNE) is alwaysused for better visualization of high-level representations learnt by an artificial neural net-work. Specifically, it is a technique for embedding high-dimensional data for visualization in alow-dimensional space of two or three dimensions. Hence, 3-dimensional t-SNE feature distri-544.3. Results and Discussions(a) (b)(c) (d)Figure 4.12: 3-dimensional t-SNE feature distribution based on: (a) SPAE; (b) AE; (c) CAE;(d) VAE.bution on the testing samples is presented as Fig. 4.12 to illustrate the feature generation of theaforementioned autoencoder-based models in the latent space. As can be observed in Fig. 4.12,the pink round scatters represent normal frames, whereas the blue star scatters represent the keyframes in the space. It can be seen in the Fig. 4.12(a) that the SPAE is prone to extract the fea-tures of the normal frames into close clusters, while the features of the key frames distributedoutside the clusters of normal frames. The scatters of key frames fall outside the relativelycentralized clusters of normal ones, which makes hyper-plane prone to be established to iden-tify the anomaly ones. In other words, the features extracted by SPAE from normal frames areaggregated to some extent, which helps assist the establishment of the decision hyper-plane of554.3. Results and Discussionsthe classifier.As for the other three models, they also generate separable features with certain degreeof aliasing. Specifically, the features of the normal frames are generated into separate clusterswith certain distance in the space through AE, which is displayed in Fig. 4.12(b). Nevertheless,the aggregation performance of AE is not very promising in terms of the feature extractionon normal frames, resulting in the decentralized feature distribution in the cluster of normalframes. As for the CAE, it achieves similar performance to that of the AE, as demonstrated inFig. 4.12(c). Moreover, VAE shows stronger aggregation capability on the feature generationof normal frames than AE and CAE. Explicitly, VAE generates two separate clusters for normalframes, but the clusters of key frames have certain overlapping with those of the normal onesin Fig. 4.12(d). The other reason of implementing SPAE rather than other methods is thatit outperforms the other selected methods on edge feature detection of the pipe anomaly onthe video frame, which was illustrated in Fig. 4.8(b) above. The superiority can be furtherdeveloped to perform soft proposal networks for weakly supervised anomaly localization inthe future work. Hence, SPAE has practical advantage to be implemented on this water pipedataset.Visualization of Key-Frame IdentificationThe proposed SPAE-based framework was also tested on one video clip of the internalsurface of water pipe. In water pipeline system, various types of pipe anomalies contain outlet,air pocket, debris, crack and so on. One type of pipe anomaly is included in the video clip,namely, outlet. The temporal regularity of the video frames is displayed through the probabilityvalue from the classifier. It is observed in Fig. 4.13 that the key frames highlighted by the pinkbackground can be detected as large probability values. This framework shows compellingperformance with most of the key frames correctly identified and only a few normal framesfalsely predicted as key frames. With this offline video processing, the inspectors are able toassess only the key frames automatically obtained, which cost less time than manually checkingthe whole inspection video. Consequently, it will save time and labor for the operators toconduct further evaluation and assessment of the water pipeline.To summarize, parameter optimization was conducted first to find the best structure of theproposed model SPAE. Five-fold cross-validation was then implemented, and results are illus-trated to estimate the model performance. Bar charts are provided to have a better visualizationof the comparative results. ROC curve and precision-recall curve are also applied to com-pare the models. Additionally, the feature extraction capacities of the selected experimentalmethods are discussed based on the feature maps and t-SNE distribution. Eventually, a visual-ization of the key-frame identification is presented. On the basis of the comparative results and564.3. Results and DiscussionsFigure 4.13: Temporal regularity of the CCTV inspection video.discussions, the proposed SPAE shows good performance in the study.4.3.3 The Log-Gabor Autoencoder based FrameworkThe LGAE-based framework was validated on the CCTV inspection data of the internalsurface of the pipe, and the experimental results are demonstrated accordingly. Since it isa further improved version of the proposed SPAE-based framework, the experimental resultsand discussions are focused on the comparison between the LGAE as well as SPAE basedframeworks. Also, the precision-recall curve and ROC curve are plotted to evaluate the newlyproposed LGAE-based framework.Comparative AnalysisFive-fold cross-validation is conducted to avoid overfitting problems. The validation resultswere collected across five subsamples, and the mean value was thus obtained accordingly.Standard deviation of each method was calculated as well. The results of five crucial evaluationmetrics are demonstrated in the Table 4.7 with standard deviation. It is known that SPAEoutperforms the other selected experimental methods in the Section 4.3.2. Also, LGAE-basedframework is the further improved version of SPAE-based framework. Hence, the comparisonis conducted only between SPAE and LGAE across five subsamples of the cross-validationin the table. As can be observed, LGAE outperforms SPAE across the first three subsampleswith the accuracy over 0.990. Obviously, LGAE reports the impressing results with the 1.000accuracy on the first subsample. As for the precision, LGAE does not perform very wellon the last two subsamples compared with 1.000 precision reported by SPAE. Additionally,LGAE presents outstanding performance on the metric of recall. To be more explicit, LGAE574.3. Results and Discussionsachieves 1.000 recall on three subsamples, which implies promising performance on the key-frame identification.Table 4.7: The evaluation metrics from five-fold cross-validation on SPAE and LGAEFold IndexAccuracy Precision Recall F1-score AUCSPAE LGAE SPAE LGAE SPAE LGAE SPAE LGAE SPAE LGAE1 0.975 1.000 0.954 1.000 0.995 1.000 0.974 1.000 0.975 1.0002 0.987 0.994 0.991 0.989 0.981 0.997 0.986 0.993 0.986 0.9943 0.968 0.992 0.945 1.000 0.991 0.983 0.968 0.991 0.968 0.9914 0.995 0.982 1.000 0.964 0.989 1.000 0.994 0.981 0.994 0.9825 0.995 0.975 1.000 0.951 0.989 1.000 0.994 0.975 0.994 0.975Mean 0.984 0.988 0.978 0.980 0.990 0.996 0.984 0.988 0.984 0.988SD 0.010 0.008 0.023 0.019 0.005 0.007 0.011 0.009 0.011 0.009Figure 4.14: The comparison of the evaluation metrics between SPAE and LGAE.For better visualization, the bar chart of the evaluation metrics averaged from five subsam-ples in the cross-validation is displayed in Fig. 4.14. The standard deviation is also shownon each bar with the black line on the top. As can be observed, LGAE achieves good resultson each metric, which slightly outperforms SPAE. The fact that SPAE outperforms the otherselected experimental methods implies the enhancement by this incorporation strategy, whichstrengthens the representation learning and extraction. This indicates that the SPAE explicitlyextracts more accurate representation, thus improves the prediction accuracy. The log-Gaborautoencoder (LGAE) was proposed to further improve the accuracy of key-frame identification584.3. Results and Discussionsand obtain better generalization on varied water pipeline datasets. LGAE outperforms SPAEwith the integration of log-Gabor filtered features into the autoencoder network. Precisely,the accuracy of LGAE is 0.988 higher than 0.984 of SPAE. In addition, the recall is reportedas 0.996, which is improved by 0.006 compared with that of SPAE. Furthermore, the hiddenrepresentation of LGAE is extracted from the intermediate hidden layer and compared withthat of SPAE in Fig. 4.15. It is worth noticing from the feature maps in Fig. 4.15 that SPAEachieves good performance of edge feature generation from our raw data, even though on thesamples with small and unclear defects. Nevertheless, it can be observed that the LGAE is ableto provide more frequency information of the frames with less aliasing. Hence, the proposedLGAE-based framework has compelling capacity of representation extraction, which is moreseparable for the classifier to predict varied key frames out of the normal frames.Figure 4.15: The hidden representations from SPAE and LGAE compared with raw frames.As mentioned in section 4.3.2, the deep learning methods show good performance in thekey-frame identification. According to the plot of ROC curves in Fig. 4.16, it can be foundthat the cross orange line (LGAE) remains highest, accounting for the largest area under thecurve, which corresponds to the largest AUC value of 0.988. The result proves that LGAEhas the best performance on TPR, which means that LGAE achieves the best performance onkey-frame identification. As for the other selected experimental methods, the curves imply thatthey achieve higher false positive rates than that of LGAE. Precisely, they are more prone tofalsely predict the normal frames as anomaly frames. From the plot of precision-recall curvesin Fig. 4.17, LGAE presents more competitive performance than the other models, namely AE,VAE, CAE, CNN, and SPAE. Especially, it is worth noticing that LGAE is slightly better thanthe proposed SPAE, which achieves the recall with a 0.006 higher value. The LGAE maintains594.3. Results and DiscussionsFigure 4.16: ROC curves of the deep learning methods in the study.good recall and, in the meantime, achieves higher precision of 0.980 than 0.978 of the SPAE.Thus, the proposed LGAE is more superior in representation learning and extraction to theproposed SPAE. The fact results from that the features extracted by LGAE are more separable.In other words, the features obtained from LGAE are prone to be classified by the predictionalgorithm.Computational CostThe computational costs of the methodologies were also investigated in the experiments.Table 4.8 presents the computational costs of three frameworks on a five-minute video. As canbe observed, the unsupervised learning-based framework takes 69s, which is around one-fifthof the video length, to accomplish key-frame identification. For the SPAE-based framework, ittakes longer time to process the video, which is 299s. Moreover, the LGAE-based frameworktakes 280s for the computational task on the video, which is close to the video length. Thereason why the first framework spends less time on the task is that the computational model isshallower compared with the SPAE and LGAE.604.3. Results and DiscussionsFigure 4.17: Precision-recall curves of the deep learning methods in the study.Table 4.8: Computational costs of three frameworks on a five-minute videoFramework Computational CostThe Unsupervised Learning-based Framework 69sThe SPAE-based Framework 299sThe LGAE-based Framework 280sDiscussionsFurthermore, the LGAE and SPAE based frameworks were tested on the second dataset,namely, the dataset from a different pipeline, for the purpose of discussing the generaliza-tion capability of the proposed models. To be more precise, the dataset contains 440 normalframes and 440 key frames, which were shuffled and split into training set (60%), validationset (20%), and testing set (20%). The second dataset was only used to fit a SVM classifier.Explicitly, the pre-trained feature extractor on the first dataset was used; however, a new SVMclassifier was trained and fit on the second dataset. As can be observed in the Table 4.9, boththe SPAE and LGAE based frameworks can achieve encouraging results on two datasets inthe study. Especially, they report 1.000 accuracy on the second dataset. It implies that thesetwo frameworks have good generalization capacity on these two datasets. Above all, due to614.4. Summarythe lack of quantity and variations of the samples, it is prone to encounter overfitting problemswhile using a small amount of dataset to fit a neural network. However, the proposed LGAEhas outstanding performance on the generalization capability. Specifically, there is no needto refit the existing feature extractor or fit a new feature extractor to address the task on thenew dataset from another pipeline. The pre-trained LGAE can be applied directly on the newdata, while a new SVM classifier needs to be fit. Generally, the LGAE-based framework canbe conducted to perform key-frame identification on another dataset with only fitting a newSVM classifier on the dataset. This would save energy and time for the inspectors to carry outkey-frame identification on varied pipeline datasets.Table 4.9: Evaluation of key-frame identification between SPAE and LGAE on two varieddatasetsMethodAccuracy Precision Recall F1-score AUCDataset 1 Dataset 2 Dataset 1 Dataset 2 Dataset 1 Dataset 2 Dataset 1 Dataset 2 Dataset 1 Dataset 2SPAE 0.984 1.000 0.978 1.000 0.990 1.000 0.984 1.000 0.984 1.000LGAE 0.988 1.000 0.980 1.000 0.996 1.000 0.988 1.000 0.988 1.000To summarize, five-fold cross-validation was implemented on LGAE-based framework,and corresponding results are illustrated to estimate the model performance. Besides, the barchart, ROC curves, and P-R curves are provided to have a better visualization of the compar-ative results. In addition, the feature extraction capabilities of the proposed LGAE and SPAEare compared based on the feature maps extracted from the intermediate layers. A discussionis also conducted on the generalization performance of two proposed methods. On the basisof the comparative results and discussions, the proposed LGAE-based framework achieves thebest performance in this study.4.4 SummaryIn this chapter, the experimental results of three frameworks are presented and discussedwith the proof of different evaluation plots. For the unsupervised learning-based framework,comparative results are analyzed regarding four essential evaluation metrics. Moreover, lossdistribution plots are discussed to estimate the model performance on the key-frame identifica-tion. This approach is easy to conduct with less dataset required. For the SPAE-based frame-work, parameter optimization was conducted first to find the best structure of the proposedmodel SPAE. Five-fold cross-validation was then implemented, and results are illustrated toestimate the model performance. Bar charts are provided to have a better visualization of the624.4. Summarycomparative results. ROC curve and precision-recall curve are also applied to compare themodels. Additionally, the feature extraction capabilities of the selected experimental methodsare discussed based on the feature maps and t-SNE distribution. Eventually, a visualization ofthe key-frame identification is presented. On the basis of the comparative results and discus-sions, the proposed SPAE shows good performance in the study. As for LGAE-based frame-work, the evaluation tables and related visualization plots are also illustrated. Discussions aremajorly included in the comparison of SPAE and LGAE based frameworks. This frameworkachieves the best results among the three frameworks in the study.63Chapter 5ConclusionIn this thesis, the autoencoder-based frameworks are proposed to address the challengesthat exist in the office-based survey of water pipeline CCTV inspection. To the authors’ bestknowledge, this is the first study to present autoencoder-based approaches to perform key-frame identification on water pipeline CCTV videos, which will facilitate the condition assess-ment of water pipeline. The main contributions can be listed as follows:• To tackle the challenge of water pipeline CCTV inspection, the thesis first carries out anunsupervised learning-based framework for key-frame identification. This framework iseasy to conduct with less dataset required, which achieves up to 0.940 accuracy with arelatively lower precision metric of 0.901.• To improve the accuracy of key-frame identification, the thesis proposes a steerable pyra-mid autoencoder (SPAE) based framework, which is the first to incorporate the steerablepyramid into the autoencoder network. In addition, ResNet module and group normal-ization are employed to further enhance powerful representation learning and strengthenfeature propagation of the SPAE framework. The SPAE enables representative and dis-criminative features to be generated and applied to the classifier. The experimental resultsshow SPAE outperforms the other state-of-the-art methods in terms of selected evalua-tion metrics, namely, 0.984 accuracy, 0.990 recall, and 0.984 F1-score. The superiorityof SPAE comes from its outstanding capability of representation learning and extractionby integrating features from multi-scale image pyramid. Thus, this framework will as-sure accuracy and greatly improve the efficiency for the survey of water pipeline CCTVinspection videos. The inspectors can focus on the information from the key frames toconduct water pipeline condition assessment.• To further empower the capability of hard mining on the video frames, a log-Gabor au-toencoder (LGAE) based framework is proposed to perform the identification task. Theschematic structure of this framework is similar to the proposed SPAE-based framework,with the integration of log-Gabor filtered features instead of steerable pyramid features.LGAE takes advantage of feature detection over SPAE on the anomalies of the videoframes. Comparative studies demonstrate that LGAE has an outperforming performance64Chapter 5. Conclusionwith the accuracy of 0.988, the recall of 0.996, and the F1-score of 0.988. The fact thatLGAE achieves better results than the other experimental methods implies that LGAEcan generate a better set of features, which will then be employed into the classifier. Be-sides, LGAE shows superiority in the deep exploitation of the information on the videoframes. Hence, LGAE-based framework will significantly assist the office-based surveyon water pipeline CCTV inspection.To conclude, in terms of water pipeline CCTV inspection, this study proposes an automatedoffice-based survey, which is key-frame identification to investigate varied pipe deficiencies.Hence, the inspection of PipeDiver R© platform is empowered for the preparation of furtheroperation in the water mains system. Additionally, the workload of the inspectors is reduced,namely, the inspectors are less prone to get fatigue on account of watching hours of recordedvideos. The manual inspection can not ensure the consistency and quality of the office-basedsurvey. However, with the proposed framework, they can focus more on the information of theidentified key frames for further pipe assessment. From the perspective of the methodologies,two autoencoder-based frameworks are proposed based on the fusing strategy of pre-definedfeatures and customized deep neural networks. They are both superior in the deep exploitationon the underwater video data, which has overcome the illumination limitation of the raw data.Besides, the proposed SPAE and LGAE have the encouraging capacity of generalization forfeature learning on different datasets.The thesis proposes promising approaches for water pipeline CCTV inspection, which isaccurate and robust. Nonetheless, the research is not intact in the thesis. There are many pos-sible future extensions for the proposed method. The potential future studies for the researchesare discussed and listed below:• In future studies, transfer learning can be used to make the framework more generalizedto the datasets from varied water pipelines.• Besides, the superiority of the proposed LGAE-based method can also be employed toperform weakly supervised anomaly localization using soft proposal networks on thedataset.• Furthermore, multi-class anomaly classification on the identified key frames can be in-vestigated to facilitate the automated CCTV data analysis for water pipeline inspection.65Bibliography[1] M. S. Islam, R. Sadiq, M. J. Rodriguez, A. Francisque, H. Najjaran, and M. Hoor-far, “Leakage detection and location in water distribution systems using a fuzzy-basedmethodology,” Urban Water Journal, vol. 8, no. 6, pp. 351–365, 2011. → pages 1, 3[2] Z. Liu and Y. Kleiner, “State of the art review of inspection technologies for conditionassessment of water pipes,” Measurement, vol. 46, no. 1, pp. 1–15, 2013. → pages 1[3] Z. Liu and Y. Kleiner, “State-of-the-art review of technologies for pipe structural healthmonitoring,” IEEE Sensors Journal, vol. 12, no. 6, pp. 1987–1992, Jun. 2012. → pages 1[4] T. T. T. Zan, H. B. Lim, K. Wong, A. J. Whittle, and B. Lee, “Event detection and local-ization in urban water distribution network,” IEEE Sensors Journal, vol. 14, no. 12, pp.4134–4142, Dec. 2014. → pages 1[5] K. Lillie, C. Reed, M. Rodgers, S. Daniels, and D. Smart, Workshop on Condition Assess-ment Inspection Devices for Water Transmission Mains. Denver, CO: Iwa Publishing,2004. → pages 1[6] D. Marlow, S. Heart, S. Burn, A. Urquhart, S. Gould, M. Anderson, S. Cook, M. Am-brose, B. Madin, and A. Fitzgerald, “Condition assessment strategies and protocols forwater and wastewater utility assets,” Water Environment Research Foundation, Alexan-dria, VA, USA, Tech. Rep., 2007. → pages[7] J. Thomson and L. Wang, “Condition assessment of ferrous water transmission and distri-bution systems state of technology review report,” U.S. Environmental Protection Agency,Washington, DC, Tech. Rep., 2009. → pages 1[8] R. Rayhana, Y. Jiao, A. Zaji, and Z. Liu, “Automated vision systems for condition as-sessment of sewer and water pipelines,” IEEE Transactions on Automation Science andEngineering, pp. 1–18, 2020. → pages 1, 4[9] X. Inc., “Confidently manage your pipes without the need to shutdown or dewater,” https://puretechltd.com/, 2020, accessed Jan. 08, 2020. → pages 1, 3966Bibliography[10] Z. Liu, Y. Kleiner, B. Rajani, L. Wang, and W. Condit, “Condition assessment tech-nologies for water transmission and distribution systems,” U.S. Environmental ProtectionAgency, Washington, DC, Tech. Rep., 2012. → pages 3, 4[11] K. S. Ong, W. H. Png, H. S. Lin, C. H. Pua, and F. A. Rahman, “Acoustic vibrationsensor based on macro-bend coated fiber for pipeline leakage detection,” in 2017 17thInternational Conference on Control, Automation and Systems (ICCAS), Oct. 2017, pp.167–171. → pages 3[12] T. Wang, G. Lu, and P. Yan, “Multi-sensors based condition monitoring of rotary ma-chines: An approach of multidimensional time-series analysis,” Measurement, vol. 134,pp. 326 – 335, 2019. → pages 3[13] S. Yoon, W. Ye, J. Heidemann, B. Littlefield, and C. Shahabi, “SWATS: Wireless sensornetworks for steamflood and waterflood pipeline monitoring,” IEEE Network, vol. 25,no. 1, pp. 50–56, 2011. → pages 3[14] S. Mounce, R. Mounce, and J. Boxall, “Novelty detection for time series data analysis inwater distribution systems using support vector machines,” Journal of Hydroinformatics,vol. 13, no. 4, pp. 672–686, 2011. → pages 4[15] A. Cataldo, G. Cannazza, E. De Benedetto, and N. Giaquinto, “A new method for de-tecting leaks in underground water pipelines,” IEEE Sensors Journal, vol. 12, no. 6, pp.1660–1667, Jun. 2012. → pages 4[16] Z. Liu and Y. Kleiner, “Computational intelligence for urban infrastructure conditionassessment: Water transmission and distribution systems,” IEEE Sensors Journal, vol. 14,no. 12, pp. 4122–4133, Dec. 2014. → pages 4[17] O. Duran, K. Althoefer, and L. D. Seneviratne, “Automated sewer pipe inspection throughimage processing,” in Proceedings 2002 IEEE International Conference on Robotics andAutomation (Cat. No.02CH37292), vol. 3, May 2002, pp. 2551–2556. → pages 4[18] M. Halfawy and J. Hengmeechai, “Efficient algorithm for crack detection in sewer imagesfrom closed-circuit television inspections,” Journal of Infrastructure Systems, vol. 20, p.04013014, Jun. 2014. → pages 4, 48, 49, 51[19] M. R. Halfawy and J. Hengmeechai, “Automated defect detection in sewer closed circuittelevision images using histograms of oriented gradients and support vector machine,”Automation in Construction, vol. 38, pp. 1–13, 2014. → pages 4, 29, 48, 49, 5167Bibliography[20] X. Yin, Y. Chen, A. Bouferguene, H. Zaman, M. Al-Hussein, and L. Kurach, “A deeplearning-based framework for an automated defect detection system for sewer pipes,”Automation in Construction, vol. 109, p. 102967, 2020. → pages 4, 5[21] J. Ma, X. Li, H. Wen, Z. Fu, and L. Zhang, “A key frame extraction method for process-ing greenhouse vegetables production monitoring video,” Computers and Electronics inAgriculture, vol. 111, pp. 92 – 102, 2015. → pages 4[22] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks,vol. 61, pp. 85–117, 2015. → pages 5, 9, 16, 48, 49, 51[23] S. Albawi, T. A. Mohammed, and S. Al-Zawi, “Understanding of a convolutional neu-ral network,” in 2017 International Conference on Engineering and Technology (ICET),2017, pp. 1–6. → pages 5[24] C. Chen, M.-Y. Liu, O. Tuzel, and J. Xiao, R-CNN for Small Object Detection. Cham:Springer International Publishing, 2017, pp. 214–230. → pages 5[25] R. Girshick, “Fast R-CNN,” in Proceedings of the IEEE International Conference onComputer Vision (ICCV), Dec. 2015. → pages 5[26] S. Park, M. Kim, and S. Lee, “Anomaly detection for http using convolutional autoen-coders,” IEEE Access, vol. 6, pp. 70 884–70 901, 2018. → pages 5, 17, 48, 49, 51[27] M. Ke, C. Lin, and Q. Huang, “Anomaly detection of logo images in the mobile phoneusing convolutional autoencoder,” in 2017 4th International Conference on Systems andInformatics (ICSAI), Nov. 2017, pp. 1163–1168. → pages 5, 17[28] M. Lopez-Martin, B. Carro, A. Sanchez-Esguevillas, and J. Lloret, “Conditional varia-tional autoencoder for prediction and feature recovery applied to intrusion detection iniot,” Sensors, vol. 17, no. 9, p. 1967, 2017. → pages 5[29] Y. Lu and P. Xu, “Anomaly detection for skin disease images using variational autoen-coder,” CoRR, vol. abs/1807.01349, Jul. 2018. → pages 5, 48, 49, 51[30] Y. Ikeda, K. Ishibashi, Y. Nakano, K. Watanabe, and R. Kawahara, “Anomaly detectionand interpretation using multimodal autoencoder and sparse optimization,” CoRR, vol.abs/1812.07136, 2018. → pages 5, 21[31] L. Beggel, M. Pfeiffer, and B. Bischl, “Robust anomaly detection in images using adver-sarial autoencoders,” CoRR, vol. abs/1901.06355, 2019. → pages 568Bibliography[32] X. Zhao, M. Jia, and M. Lin, “Deep laplacian auto-encoder and its application into im-balanced fault diagnosis of rotating machinery,” Measurement, pp. 1–21, Nov. 2019. →pages 5[33] D. Xu, E. Ricci, Y. Yan, J. Song, and N. Sebe, “Learning deep representations ofappearance and motion for anomalous event detection,” CoRR, vol. abs/1510.01553,2015. [Online]. Available: http://arxiv.org/abs/1510.01553 → pages 5, 8[34] Y. S. Chong and Y. H. Tay, “Abnormal event detection in videos using spatiotemporalautoencoder,” CoRR, pp. 189–196, Jan. 2017. → pages 5, 8[35] Y. Y. Chen, Y. H. Lin, C. C. Kung, M. H. Chung, and I. H. Yen, “Design and imple-mentation of cloud analytics-assisted smart power meters considering advanced artificialintelligence as edge analytics in demand-side management for smart homes,” Sensors(Basel, Switzerland), vol. 19, no. 9, p. 2047, 2019. → pages 8[36] H. Bourlard and Y. Kamp, “Auto-association by multilayer perceptrons and singular valuedecomposition,” Biological cybernetics, vol. 59, no. 4-5, pp. 291–294, 1988. → pages 9[37] W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, “A survey of deep neuralnetwork architectures and their applications,” Neurocomputing, vol. 234, pp. 11 – 26,2017. → pages 10[38] M. Ranzato, C. Poultney, S. Chopra, and Y. Lecun, “Efficient learning of sparse rep-resentations with an energy-based model,” in 19th International Conference on NeuralInformation Processing Systems, Jan. 2006, pp. 1137–1144. → pages 10[39] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing ro-bust features with denoising autoencoders,” in 25th international conference on Machinelearning. ACM, 2008, pp. 1096–1103. → pages 10[40] P. Vincent, “A connection between score matching and denoising autoencoders,” Neuralcomputation, vol. 23, no. 7, pp. 1661–1674, 2011. → pages 10[41] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio, “Contractive auto-encoders:Explicit invariance during feature extraction,” in Proceedings of the 28th InternationalConference on Machine Learning, ICML 2011, Jan. 2011, pp. 1–8. → pages 10[42] J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstructionprobability,” in Special Lecture on IE, 2015, pp. 1–18. → pages 1069Bibliography[43] S. Khobahi and M. Soltanalian, “Model-aware deep architectures for one-bit compressivevariational autoencoding,” CoRR, vol. abs/1911.12410, 2019. → pages 10[44] H. Partaourides and S. P. Chatzis, “Asymmetric deep generative models,” Neurocomput-ing (Amsterdam), vol. 241, pp. 90–96, 2017. → pages 10[45] T. Robinson, “A real-time recurrent error propagation network word recognition system,”in [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech,and Signal Processing, vol. 1, 1992, pp. 617–620. → pages 10[46] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, “Phoneme recognitionusing time-delay neural networks,” IEEE transactions on acoustics, speech, and signalprocessing, vol. 37, no. 3, pp. 328–339, 1989. → pages 10[47] L. Deng and D. Yu, “Deep learning: Methods and applications,” Foundations and Trendsin Signal Processing, vol. 7, no. 3-4, pp. 197–387, Jan. 2013. → pages 10[48] F. J. M. Gonza´lez and M. Balajewicz, “Deep convolutional recurrent autoen-coders for learning low-dimensional feature dynamics of fluid systems,” CoRR, vol.abs/1808.01346, 2018. → pages 16[49] D. Kingma and M. Welling, “Auto-encoding variational bayes,” in International Confer-ence on Learning Representations (ICLR), Dec. 2014, pp. 1–14. → pages 18[50] D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropagation and approxi-mate inference in deep generative models,” in Proceedings of the 31st International Con-ference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 32.Bejing, China: PMLR, Jun. 2014, pp. 1278–1286. → pages 18[51] S. M. Erfani, S. Rajasegarar, S. Karunasekera, and C. Leckie, “High-dimensional andlarge-scale anomaly detection using a linear one-class svm with deep learning,” PatternRecognition, vol. 58, pp. 121–134, 2016. → pages 19[52] E. Simoncelli and W. Freeman, “Steerable pyramid: a flexible architecture for multi-scalederivative computation,” in IEEE International Conference on Image Processing, vol. 3,1995, pp. 444–447. → pages 21[53] Z. Liu, K. Tsukada, K. Hanasaki, Y. K. Ho, and Y. P. Dai, “Image fusion by using steerablepyramid,” Pattern Recognition Letters, vol. 22, no. 9, pp. 928–937, Jun. 2001. → pages2170Bibliography[54] Z. Liu, Y. K. Ho, K. Tsukada, K. Hanasaki, Y. P. Dai, and L. Q. Li, “Using multipleorientational filters of steerable pyramid for image registration,” International Journal onMulti-Sensor, Multi-Source Information Fusion, vol. 3, no. 3, pp. 203–214, Sep. 2002. →pages 21[55] E. Simoncelli, “The steerable pyramid,” https://www.cns.nyu.edu/∼eero/steerpyr/, 2008,accessed Jan. 08, 2020. → pages 22[56] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Con-ference on Learning Representations, pp. 1–15, Dec. 2014. → pages 24, 40[57] A. Myronenko, “3D MRI brain tumor segmentation using autoencoder regularization,”in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Cham,2019, pp. 311–320. → pages 26[58] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2016,pp. 770–778. → pages 26[59] C. Cortes and V. Vapnik, “Support-vector networks,” Mach Learn 20, p. 273–297, 1995.→ pages 27[60] T. Hastie, R. Tibshirani, and J. H. Friedman, The elements of statistical learning: datamining, inference, and prediction, 2nd ed. New York, NY: Springer, 2009. → pages 27[61] W. H. Press, Numerical recipes: the art of scientific computing. New York: CambridgeUniversity Press, 1986. → pages 27[62] S. Han, C. Qubo, and H. Meng, “Parameter selection in svm with rbf kernel function,” inWorld Automation Congress 2012. IEEE, 2012, pp. 1–4. → pages 28[63] N. Rose, “Facial expression classification using Gabor and log-Gabor filters,” in 7th In-ternational Conference on Automatic Face and Gesture Recognition (FGR06), 2006, pp.346–350. → pages 30[64] Z. Xiao, C. Guo, M. Yu, and Q. Li, “Research on log Gabor wavelet and its applicationin image edge detection,” in 6th International Conference on Signal Processing, 2002.,vol. 1, 2002, pp. 592–595. → pages 31[65] D. J. Field, “Relations between the statistics of natural images and the response propertiesof cortical cells,” Journal of the Optical Society of America. A, Optics and image science,vol. 4, no. 12, pp. 2379–2394, 1987. → pages 3171Bibliography[66] S. Fischer, F. Sˇroubek, L. Perrinet, R. Redondo, and G. Cristo´bal, “Self-invertible 2D log-Gabor wavelets,” International journal of computer vision, vol. 75, no. 2, pp. 231–246,2007. → pages 31[67] W. Wang, J. Li, F. Huang, and H. Feng, “Design and implementation of log-Gabor filter infingerprint image enhancement,” Pattern recognition letters, vol. 29, no. 3, pp. 301–308,2008. → pages 3272
Thesis/Dissertation
eng
Electrical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Autoencoder-based key-frame identification for water pipeline CCTV inspection
Text